<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi Riccardo<br>
      <br>
      I guessed you mentioned about
      trilinos_residualbased_elimination_builder_and_solver.h. However
      it still does not handle all the dofs correctly. Some dofs are
      synchronized and some do not in my case. Looking in the
      SetUpDofSet, I was not clear about the meaning of the code segment
      below:<br>
      <br>
      <br>
              //now send our local ndofs to the other processors using
      coloring<br>
              for (unsigned int i_color = 0; i_color &lt;
      r_model_part.GetCommunicator().NeighbourIndices().size();
      i_color++)<br>
              {<br>
                  int destination =
      r_model_part.GetCommunicator().NeighbourIndices()[i_color];<br>
                  if (destination &gt;= 0)<br>
      //            if (destination &gt;= 0 &amp;&amp; destination !=
      rank)<br>
                  {<br>
                      MPI_Status status;<br>
                      int send_tag = i_color;<br>
                      int receive_tag = i_color;<br>
                      <br>
                      //first of all obtain the number of nodes we will
      need to get remotely<br>
                      int remote_gids_size;<br>
                      int local_gids_size =
      local_gids[destination].size();<br>
                      MPI_Sendrecv(&amp;local_gids_size, 1, MPI_INT,
      destination, send_tag, &amp;remote_gids_size, 1, MPI_INT,
      destination, receive_tag,MPI_COMM_WORLD, &amp;status);<br>
                      remote_gids[destination].resize(remote_gids_size);<br>
                     
      remote_ndofs[destination].resize(remote_gids_size);<br>
      <br>
                      //receive the remote GiDs<br>
                      MPI_Sendrecv(local_gids[destination].data(),
      local_gids[destination].size(), MPI_INT, destination, send_tag,
      remote_gids[destination].data(), remote_gids_size, MPI_INT,
      destination, receive_tag,MPI_COMM_WORLD, &amp;status);<br>
      <br>
                      //receive the remote ndofs (same size as the gids)<br>
                      MPI_Sendrecv(local_ndofs[destination].data(),
      local_ndofs[destination].size(), MPI_INT, destination, send_tag,
      remote_ndofs[destination].data(), remote_gids_size, MPI_INT,
      destination, receive_tag,MPI_COMM_WORLD, &amp;status);<br>
      <br>
                      //find the number of non local dofs to receive<br>
                      int remote_keys_size;<br>
                      int local_keys_size =
      local_keys[destination].size();<br>
                      MPI_Sendrecv(&amp;local_keys_size, 1, MPI_INT,
      destination, send_tag, &amp;remote_keys_size, 1, MPI_INT,
      destination, receive_tag,MPI_COMM_WORLD, &amp;status);<br>
                      remote_keys[destination].resize(remote_keys_size);<br>
      <br>
                      //receive the keys<br>
                      MPI_Sendrecv(local_keys[destination].data(),
      local_keys[destination].size(), MPI_INT, destination, send_tag,
      remote_keys[destination].data(), remote_keys_size, MPI_INT,
      destination, receive_tag,MPI_COMM_WORLD, &amp;status);<br>
                  }<br>
              }<br>
      <br>
      <br>
      Can you help to explain why we use same source and dest in
      MPI_Sendrecv and the outcome of this part ?<br>
      <br>
      <br>
      Ciao<br>
      Bui<br>
      <br>
      <br>
      <br>
      <br>
      On 06/13/13 13:02, Riccardo Rossi wrote:<br>
    </div>
    <blockquote
cite="mid:CAOVdALznrD9dguFTswyXLjB49y6-+mAa1_zqWukf5ZJ0VQ64Tg@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>
          <div>
            <div>
              <div>
                <div>
                  <div>Dear Bui,<br>
                  </div>
                               the issue you raise is quite hairy, but i
                  think i solved it some time ago in the builder and
                  solver. <br>
                  <br>
                </div>
                to begin, all of the nodes in the kratos shall have the
                same list of dofs, which means that some of the dofs may
                be uninitialized in some of the nodes (or even wors, may
                be utilized in one domain but not in the next)<br>
                <br>
              </div>
              in any case when dofs are syncronized the "owner" simply
              spawns its values to all of the others, assuming that each
              node has all of the dofs...<br>
              <br>
              <br>
            </div>
            the thing works as the builder and solver (with the
            modifications i committed some time ago) shall correctly
            handle one node having two different lists of active dofs on
            two different mpi domains. <br>
          </div>
          This implies that after the solution is finished the owner of
          the node shall have the correct values and hence the sync
          should work.<br>
          <br>
        </div>
        <div>to understand how this works look around line 540 of the
          trilinos_residualbased_builderandsolver....<br>
          <br>
        </div>
        <div>Riccardo<br>
        </div>
        <div><br>
        </div>
        <br>
      </div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On Thu, Jun 13, 2013 at 12:04 PM, Hoang
          Giang Bui <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:hgbk2008@gmail.com" target="_blank">hgbk2008@gmail.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
            Hi<br>
            <br>
            I want to understand more about the method Kratos use to
            synchronize<br>
            dofs between different process. I have a case like this:<br>
            <br>
            + Process 1 contains elements with multiple dofs
            (DISPLACEMENT(_XYZ) &amp;<br>
            WATER_PRESSURE)<br>
            <br>
            + Process 2 contains only conditions which involve only
            DISPLACEMENT(_XYZ)<br>
            <br>
            + node k belongs to process 2<br>
            <br>
            When dofs are enumerated in process 1, node k on the
            boundary of process<br>
            1 &amp; 2 is fully enumerated with DISPLACEMENT(_XYZ) and
            WATER_PRESSURE<br>
            When dofs are enumerated in process 2, node k only has
            DISPLACEMENT(_XYZ)<br>
            <br>
            When dofs are synchronized, the dof on process 2 supersede
            process 1.<br>
            Which means WATER_PRESSURE dof on node k is assigned with
            EquationId 0.<br>
            <br>
            In this case what should I do to enable dof enumeration
            correctly in<br>
            parallel?<br>
            <br>
            Ciao<br>
            Bui<br>
            <br>
            _______________________________________________<br>
            Kratos mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:Kratos@listas.cimne.upc.edu">Kratos@listas.cimne.upc.edu</a><br>
            <a moz-do-not-send="true"
              href="http://listas.cimne.upc.edu/cgi-bin/mailman/listinfo/kratos"
              target="_blank">http://listas.cimne.upc.edu/cgi-bin/mailman/listinfo/kratos</a><br>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <br>
        -- <br>
        <p><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;">Dr.

            Riccardo Rossi, Civil Engineer<br>
          </span></p>
        <p><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;">Member

            of Kratos Team<span style="color:#1f497d"> </span></span></p>
        <p><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d">International

            Center for Numerical Methods in Engineering - CIMNE<br>
            Campus Norte, Edificio C1</span><span style="color:#1f497d"> </span></p>
        <p style="text-align:justify"><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d"
            lang="ES">c/ Gran Capitán s/n</span></p>
        <p style="text-align:justify"><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d"
            lang="ES">08034 Barcelona, España</span></p>
        <p style="text-align:justify"><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d"
            lang="ES">Tel:        <a moz-do-not-send="true"
              value="+34934015696">(+34) 93 401 56 96</a></span></p>
        <p style="text-align:justify"><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d"
            lang="ES">Fax:       <a moz-do-not-send="true"
              value="+34934016517">(+34) 93.401.6517</a></span></p>
        <span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#1f497d"
          lang="ES">web:</span><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;"
          lang="ES">       </span><span lang="ES"><a
            moz-do-not-send="true" href="http://www.cimne.com/"
            target="_blank"><span
style="font-size:10.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;">www.cimne.com</span></a></span>
      </div>
    </blockquote>
    <br>
  </body>
</html>