From owner-mpi-pt2pt@CS.UTK.EDU Mon Nov 23 14:37:41 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18480; Mon, 23 Nov 92 14:37:41 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA23460; Mon, 23 Nov 92 14:28:46 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23454; Mon, 23 Nov 92 14:28:42 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA06042; Mon, 23 Nov 92 14:28:41 -0500 Message-Id: <9211231928.AA06042@thud.cs.utk.edu> To: mpi-pt2pt@cs.utk.edu Cc: wade@cs.utk.edu Subject: a test Date: Mon, 23 Nov 92 14:28:40 EST From: Reed Wade this is a test of the mpi-pt2pt mailing list. -reed Mon Nov 23 16:03:12 EST 1992 From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 24 23:08:30 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15646; Tue, 24 Nov 92 23:08:30 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA25775; Tue, 24 Nov 92 22:28:54 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25771; Tue, 24 Nov 92 22:28:51 -0500 Received: from Electryon.Think.COM by mail.think.com; Tue, 24 Nov 92 22:28:43 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA20255; Tue, 24 Nov 92 22:28:41 EST Date: Tue, 24 Nov 92 22:28:41 EST Message-Id: <9211250328.AA20255@electryon.think.com> To: mpi-pt2pt@cs.utk.edu Subject: Nonblocking functions and handlers. I would like to float a few questions about what we mean by the modifier nonblocking it is applied to the MPI specification. Nonblocking Send: This is the curious one. Does is mean: a) The user may alter the contents of the buffer once the send function returns control? or b) The user must check the status of the send operation to determine when it is safe to alter the contents of the buffer? Nonblocking receive: This seems pretty clear. The user must inquire of the system to determine if the user specified buffer contains the desired message. In either case, it may be desirable to specify an action that is to occur once the operation is complete. Consider adding two arguments to the send and receive functions: MPI__n( int , int tag, void * buffer, int len, void (* handler)(), void * arg ) Here, handler is a pointer to a function which accepts two arguments: handler( int msg_id, * arg ) Upon completion of the send or receive operation, the handler function is invoked and provided with the id of the message operation just completed and the argument address provided by the user. I raise the issue of handlers because we have ben asked repeatedly to provide such fuctionality. Thoughts? moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:38:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17471; Wed, 25 Nov 92 10:38:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06314; Wed, 25 Nov 92 10:13:00 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06310; Wed, 25 Nov 92 10:12:58 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA08435; Wed, 25 Nov 92 09:12:54 CST Received: by teak18.cray.com id AA05289; 4.1/CRI-5.6; Wed, 25 Nov 92 09:12:52 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251512.AA05289@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 9:12:49 CST Cc: par@teak.cray.com (Peter Rigsbee) In-Reply-To: <9211250328.AA20255@electryon.think.com>; from "Adam Greenberg" at Nov 24, 92 10:28 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Adam Greenberg writes: > In either case, it may be desirable to specify an action that is to occur once > the operation is complete. Consider adding two arguments to the send and > receive functions: > > MPI__n( int , int tag, void * buffer, int len, > void (* handler)(), void * arg ) > > Here, handler is a pointer to a function which accepts two arguments: > handler( int msg_id, * arg ) > > Upon completion of the send or receive operation, the handler function is > invoked and provided with the id of the message operation just completed and > the argument address provided by the user. I've never really liked this approach. I think it adds complexity without really adding much value. A couple of issues: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) - second, even if you have a signal handler, the mainline of the application must still poll something to see if the operation has completed; the difference is that the "something" if often a local variable rather than the messaging system. But the logic in the program is similar and it is likely that the number and decision of where to poll is driven by the algorithm. - third, do all architectures provide the user-level interrupts needed to implement this? If one doesn't, adding this functionality may simply not be possible. IMO, this seems like something that isn't desirable as a part of MPI1. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:40:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17537; Wed, 25 Nov 92 10:40:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06692; Wed, 25 Nov 92 10:30:22 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06686; Wed, 25 Nov 92 10:30:19 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03147; Wed, 25 Nov 92 09:28:46 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12985; Wed, 25 Nov 92 09:28:43 CST Date: Wed, 25 Nov 92 09:28:43 CST Message-Id: <9211251528.AA12985@godzilla.mcs.anl.gov> To: moose@think.com Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Adam Greenberg's message of Tue, 24 Nov 92 22:28:41 EST <9211250328.AA20255@electryon.think.com> Subject: Nonblocking functions and handlers. I'd love to see a standard implementation that invoked a routine on receipt, particularly if that routine was not restricted in any way (for example, restricted to not using any MPI calls). Unfortunately, doing this right makes demands on the underlying OS that may be more than some vendors are willing to commit to. We need to be careful here. Note that Unix does not provide this functionality to users, though VMS has for a long time. (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. Bill Gropp From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:07:46 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19010; Wed, 25 Nov 92 11:07:46 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07267; Wed, 25 Nov 92 11:02:46 -0500 Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07263; Wed, 25 Nov 92 11:02:43 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Wed, 25 Nov 1992 11:02:31 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA26484; Wed, 25 Nov 1992 11:02:30 -0500 Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: Nonblocking functions and handlers. Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Yale Computer Science Department From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:38:31 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19507; Wed, 25 Nov 92 11:38:31 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07714; Wed, 25 Nov 92 11:22:36 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07710; Wed, 25 Nov 92 11:22:33 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA13731; Wed, 25 Nov 92 10:22:31 CST Received: by teak18.cray.com id AA05599; 4.1/CRI-5.6; Wed, 25 Nov 92 10:22:30 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251622.AA05599@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 10:22:26 CST In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU>; from "Harry Berryman" at Nov 25, 92 11:02 am X-Mailer: ELM [version 2.3 PL11b-CRI] Scott Berryman writes: > - first, like most signal handling, this kind of capability tends > to be more oriented towards C programs and away from Fortran > programs (note that your prototype for "handler" is such that > it could not be programmed in Fortran) > > Although the prototype given by Greenberg precludes the use of > Fortran, the approach certainly does not preclude Fortran. > Obviously Fortran can pass pointers to functions as parameters. Point well taken. But there are two aspects of the interface to consider. Fortran passing pointers to functions as parameters is fine. But a Fortran programmer might also want to write the handler in Fortran, which would then require that the handler's arguments all be pointers to things. But this would bother C programmers and would be inconsistent with other "signal handlers". > In particular, the iPSC series (including the Delta) support > a similar mechanism in Fortran through the hrecv(), et al. My iPSC documentation (which may be out of date) shows that hrecv() can be called by Fortran but requires that the handler be written in C: proc is the procedure executed when the receive is complete. This procedure must be written by the user and must be a C procedure with four arguments. I'd argue that having to write the handler in C makes this less useful to Fortran programmers, especially if you are trying to write something portable. There is no real standard defining how C and Fortran interact. The C handler could call a subhandler written in Fortran, but this is getting kind of messy for something that should be simple. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:43:10 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21772; Wed, 25 Nov 92 13:43:10 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA10110; Wed, 25 Nov 92 13:30:11 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10106; Wed, 25 Nov 92 13:30:09 -0500 Received: from Electryon.Think.COM by mail.think.com; Wed, 25 Nov 92 13:30:05 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA22742; Wed, 25 Nov 92 13:30:04 EST Date: Wed, 25 Nov 92 13:30:04 EST Message-Id: <9211251830.AA22742@electryon.think.com> To: berryman-harry@CS.YALE.EDU Cc: mpi-pt2pt@cs.utk.edu, par@teak.cray.com In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> "berryman-harry@CS.YALE.EDU" Subject: Nonblocking functions and handlers. Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Mea culpa. The prototype was not meant to exclude Fortran (in fact we through a discussion of this issue here last week). The handler should viewed as taking two pointers: void handler( int * msg_id, * arg ) Sorry for any confusion: moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:44:47 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21797; Wed, 25 Nov 92 13:44:47 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA09803; Wed, 25 Nov 92 13:14:38 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09784; Wed, 25 Nov 92 13:14:29 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA22170; Wed, 25 Nov 92 13:14:31 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 131340.4011; Wed, 25 Nov 1992 13:13:40 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA12688; Wed, 25 Nov 1992 12:06:04 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08958; Wed, 25 Nov 92 12:06:02 -0600 Date: Wed, 25 Nov 92 12:06:02 -0600 Message-Id: <9211251806.AA08958@brisk.kai.com> To: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-formal@cs.utk.edu, mpi-ptop@cs.utk.edu Reply-To: William.Gropp's.message.of.Wed@kai.com, 25 Nov 92 09:28:43 CST <9211251528.AA12985@godzilla.mcs.anl.gov> Subject: Nonblocking functions and handlers. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Bill Gropp writes: (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. I'm not so sure this is a radical position Bill since even nonsynchronized communication will need to be defined formally this way. Nonsynchronized communication is in effect creating a parallel process that has the job of passing the communication on. Al Geist earlier asked the question wheather buffers used by nonsynchronized communication should be accessible after the communication has started - the answer should be - no, unless by some explicit mechanism that formally amounts to a communication with the process mentioned above. Any nonexplicit interaction (e.g. a write to the buffer) would have to be specified as formally equivalent to an explicit interaction. Also, there is quite a range of terminology in use. One common error: "Asynchronous" and "synchronous" has quite a particular meaning in EE and when CS people use the terms in relation to message passing they usually mean NONSYNCHRONIZED and SYNCHRONIZED. Also BLOCKING = SYNCHRONIZED. Let us begin a glossary that defines the terms we use - if no-one else volunteers I'll take this to be the responsibility of the Formal Specification Subcommittee. So I'm looking for volunteers from that subcommittee. Steven From owner-mpi-pt2pt@CS.UTK.EDU Sat Nov 28 19:07:59 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08243; Sat, 28 Nov 92 19:07:59 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22503; Sat, 28 Nov 92 18:46:34 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22499; Sat, 28 Nov 92 18:46:32 -0500 Message-Id: <9211282346.AA22499@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6099; Sat, 28 Nov 92 18:45:57 EST Date: Sat, 28 Nov 92 18:35:53 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: Our work Reply-To: SNIR@watson.ibm.com First, a technical issue: If somebody cannot print postscript, or has strong preferences for Latex, pls let me know. Now for content. I would like to use the outline I just sent as a working document for our subcommittee, and modify it periodically to reflect input I receive from the remaining group members. I shall update this document every weekend, or every 2 weekends, according to the number of comments I get, and the amount of spare time my family grants me. Of course, decisions will be taken when we meet, but I feel it is useful to have a working document that reflects in a somewhat coherent manner the state of our deliberations. So please, start sending your comments to mpi-pt2pt, assuming the next checkpoint is end of next week. I want to replace as soon as possible the I's by We's. From owner-mpi-pt2pt@CS.UTK.EDU Thu Dec 3 10:39:15 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01501; Thu, 3 Dec 92 10:39:15 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA16013; Thu, 3 Dec 92 10:20:18 -0500 Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16008; Thu, 3 Dec 92 10:20:13 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA25545 (5.65c/IDA-1.4.4 for ); Thu, 3 Dec 1992 16:19:46 +0100 Received: by f1neuman.gmd.de id AA14744; Thu, 3 Dec 1992 16:19:30 +0100 Date: Thu, 3 Dec 1992 16:19:30 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9212031519.AA14744@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: gmap10@f1neuman.gmd.de I would like to make some comments on the document Marc Snir has distributed. 1. I agree on all points in the list of goals. I find it especially important to define something which is not too far away from common practice, otherwise nobody would follow us. A point which is missing on the list (perhaps because it is too obvious): the interface should not contain more functions than are really necessary. We must avoid to end up with a mixed bag which contains multiple mechanisms for the same thing. 2. I like Marc's typification scheme for message passing functions based on the 4 suboperations. I understand those suboperations more as a way to describe the semantics rather than the implementation of the (compound) functions which the user will see. I'm not sure that the channel based operations are necessary on the user level. Basically there are the two addressing schemes for message passing: either a message is sent to a named process, or it is sent along a named connection (channel), in which case the processes don't even need to have identifiers. We had a long discussion here in Europe on which scheme is better. In the end it turns out that both are more or less equivalent. Most existing user interfaces tend towards the named process approach. The only reason for supporting both concepts in the same interface seems to be the possibility of some additional optimization. If suboperations (1) and (4) are separated, ressource allocation and deallocation can be saved. However, I think that a clever operating system can detect frequently used communication paths automatically, and it can establish a permanent link for them. (Link caching) 3. We will have various message passing mechanisms, especially synchronous and asynchronous communication (global/local send termination). I strongly recommend that the corresponding send and receive functions must match. On one hand this gives the opportunity for some implementation optimization, and also it might be used for correctness checking. 4. As Marc said, in MPI1 termination is either local or global, at the user's choice. I really think that this is necessary, and a restriction on either synchronous or asynchronous communication would be hard to accept. 5. Marc suggests that there should be no "unsuccessful return" of a message passing operation. I agree. 6. On page 7, Marc discusses the different protocols for message passing. I'm not sure that we want to define this in detail, since it is very much implementation dependent. On the other hand we will have to accept the fact that a program will run on a machine with huge system buffers and not on a smaller system. Somehow this is true with other system resources, also. 7. Page 7, further down: if we deal with heterogeneous networks, a contiguous message buffer may contain sections with different data type. The reason is always that one wants to send as few messages as possible (startup minimization). Personally I prefer to have a separate library call before the send which specifies the contents of the message buffer. This information is then used or ignored by the send routine depending on the data representations in the sender and receiver processors. 8. If our goal is to define a scalable interface, setting up groups by specifying member lists in all participating processes is a problem. This is especially so for the handling of very regular group structures, like grids or tori, since these will most likely be used by applications with very large process numbers. I think the Virtual Topology subcommittee has to deal with this problem. 9. Communication context: I very much favor option 1. (i.e. context switch by some special functions) against explicitely specifying the context in a (collective) communication statement. I do not see that the explicit specification would lead to less user errors, and it would add another item to the argument lists. 10. Receive criteria: I do not see a good point for having "don't care" addresses in sends. For "don't care" in receive I prefer named constants over "magic values". Be sure not to use arguments for both input to and output from a function, especially if it may contain wildcards on input. There is no better way to make sure your constants get overwritten! 11. Matching of data: In Fortran I think option 2. is acceptable. If the receive buffer is too short, this is a user error. In C there is another option which should be supported by the interface: if the receive buffer pointer is NULL, the system allocates one with the right size automatically. 12. Syntax: multiple modes like in MPI1 are okay with me, but I hate to see character strings there. Talk to any implementor who cares about efficiency, and you know what I mean. 13. Also, in Fortran I find it better to use procedure calls for most functions. 14. Miscelania: a) We will need a nonblocking AWAIT operation. This is a great help in organizing the overlap of computation and communication. b) How could we survive without a PROBE function? c) What about providing an inquiry subroutine, which returns the message length, the sender, and the type of the "last message" which was received? For non-blocking messages this is not well defined, but a solution should be possible. This has become a longer message than I had planned. Sorry for bothering you all so much! Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 16:39:48 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21532; Wed, 9 Dec 92 16:39:48 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22026; Wed, 9 Dec 92 16:32:54 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22022; Wed, 9 Dec 92 16:32:51 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA05364; Wed, 9 Dec 92 15:32:41 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA00525; Wed, 9 Dec 92 15:32:39 CST Date: Wed, 9 Dec 92 15:32:39 CST Message-Id: <9212092132.AA00525@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: lusk@antares.mcs.anl.gov Thanks to Marc for laying out many of the issues so clearly. We find that we agree with many and disagree with some. Goals: (1) We think that those most in need of a standard are those writing libraries, and therefor the standard should cater to their needs as well as those of end users. It will be easy to design simple application programming interfaces on top of the MPI standard. (2) We strongly agree with this point (allow efficient communication). (3) We are not sure what "allow (but no mandate)" for heterogeneous communications means. If this means that vendors of homogeneous massively parallel systems may implement only the homogeneous operations, then we agree, as long as the heterogeneous operations are defined by the MPI standard. We agree in principle with the others. Framework: We also find the described framework useful for thinking about the operations, but not necessarily the right structure for the MPI. Issues: An exchange operation can be useful. At least two MPP vendors (TMC and Intel) provide such operations. We also agree an leaving out RPC, active messages, etc, as these are topics of rapidly moving research. We believe that there may be something more inclusive than channels that can serve as the basis for specifying high-performance communication operations. We do not find the discussion of "local versus global" termination to be a good way to consider the issues of blocking, nonblocking, and synchronous communication. In particular, blocking and nonblocking refer to the state of the buffer containing the message on return from the routine (be it send, receive, or wait), and synchronous (in the MPI proposal) really refers to an alternate protocol for the delivery of a message. We believe that relying on an exception handling mechanism for error handling will make the system too fragile. We propose allowing the return of error conditions. An alternate proposal allows the user to specify an error routine to be called in the event of an error. Note that library implementors may need to handle errors explicitly. Correctness Criteria: We believe that it is important to allow the user to determine (at least a bound on) the amount of resources available. Further, it MAY be appropriate to specify a minimum size that is supported (otherwise zero may be consistent with the letter of the standard). Communication Parameters: Our draft implementation includes an extension to the current proposal that permits the communication of messages of mixed datatype (we interpret "type" on page 7 to mean "datatype" rather than "message type"). This is a simple extension to the data description vector that allows for heterogeneous communication. Groups and context: We agree that the collective communication committee should be the one to discuss these issues. We do feel that the destination and source processor ids should not be relative to some "current" group. Note that such a functionality (with some restrictions) can be provided by a simple layer on top of MPI. For an example of a problem with making processor id's relative to a current "group", consider recv(source=DONT_CARE,...). Does this mean only in the current group? What is the result of MPI_INFOS() on the received message? What if two processors A and B are both in two groups alpha and beta, but with different rankings (ie., A = 2 and B = 5 in alpha, and A = 12 and B = 0 in beta). Now A (current group alpha) sends to B (current group beta). B does recv(source=DONT_CARE,...). Does B receive this message? What is the value of MPI_INFOS()? If B doesn't receive this message, then how is a communication context different from a group that is a copy of the current group (which would seem to supply the same semantics)? Matching of send and receive: We believe that there should be two types of matching operations: a simple (efficient) one on message type, and a general one using a user-defined predicate (matching function). Our experience has been that once the users have "select by source", they'll then want "select by source range and/or type range and/or message size and/or ...". We also agree that truncation of a received message into a too-short buffer should be an error (or at least detectable!). Syntax: We agree with multiple function names rather than multiple modes. It is easier to provide a efficient implementation and clear semantics in this case (for example, having the return code always have the same meaning independent of the input arguments). Note also that it is easy, given a multiple function names approach, to produce one-routine-does-everything interfaces but not vice-versa. Miscelania: We must have a non-blocking AWAIT. We'd like to AWAIT several messages. We must have a PROBE. We agree that the issue of returning data on received messages (particularly those done with nonblocking operations) is very important. One approach (used by one MPP vendor) is to ask for this info by handle, and then explicitly free the handle. This eliminates unnecessary global state (the currently most recently received message) from the semantics, and makes it possible to easily describe what exactly is returned by these "INFO" routines. Bill Gropp and Rusty Lusk From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 17:09:55 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22659; Wed, 9 Dec 92 17:09:55 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22401; Wed, 9 Dec 92 16:51:16 -0500 Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22397; Wed, 9 Dec 92 16:51:13 -0500 Received: from b125 (b125.super.org) by super.super.org (4.1/SMI-4.1) id AA08797; Wed, 9 Dec 92 16:51:11 EST Received: by b125 (4.1/SMI-4.1) id AA04506; Wed, 9 Dec 92 16:51:10 EST Date: Wed, 9 Dec 92 16:51:10 EST From: lederman@b125.super.org (Steve Lederman) Message-Id: <9212092151.AA04506@b125> To: mpi-pt2pt@cs.utk.edu Subject: thoughts I have a few thoughts on the ideas that have been passed around. I'm sorry for not responding sooner but a cold has slowed me down. First, on Marc's document of Nov. 28: - Goal 4 is for a C, C++, F77 and F90 binding. The draft Intro that Jack sent around on Nov. 30 states in the subsection for Who Should Use This Standard? that it is for Fortran 77 and/or C. Do we wish to include C++ and F90 which would likely to have different interfaces? - I find the SEND_RECV operation an interesting option. On the old Mark/?? machines from Caltech/JPL there was a call (if my recollection is correct) that allowed for the same user buffer to serve as the send and receive buffer. I assume this is not what is intended here? I think SEND_RECV would be used with some frequency by users. Would it simply be a convenience or can it be made faster than issuing the two calls. I would think it would be faster on some architectures (even more than the two call overhead). If it is faster then my inclination is to include it even though it adds another call. comments? - I have a question regarding when a communication operation is completed. On a send, does it terminate locally if the user buffer is copied into a system buffer but has not left the node? (I believe this is done on the CM5.) Analogously, does it terminate globally if it gets to a system buffer on the receiving node or until it is in the user buffer? As for choosing local, global or both I vote for the current choice of both. I can see applications for both. - I think that Marc's suggestion that MPI functions not return unsuccessfully has merit. It would be pain to be constantly checking the status after each operation to be safe. Along these lines is what to do if the send and recv message lengths are different. The current standard allows for fewer bytes to be received and the recv returns the actual number. Again, I think it would be a pain to have to check every time. If this is a desirable feature to allow different lengths, then I vote to make it users selectable. I know I have found many bugs by not having messages that were a different size being received. - I think that a PROBE function would be a good idea. Given that the standard is not currently going to support active messages, etc. it seems that being able periodically check if any messages have arrived would be a useful feature. Another thought: At the meeting in Minneapolis, several people, including myself, discussed the advantage of having a send where the sending process knows the receiving process has already posted a receive. This could relieve the send process of having to check with the receiving process since it knows it will not have to buffer the message in system space and it knows it is ready to receive the message. If a system could not take advantage of this then it could revert to a normal send. That is, the standard would not specify the way the send would occur, it would allow for an alternative protocol if the vendor wished. This is the rr (receiver ready) option in Bill Gropp and Ewing Lusk's implementation. I would like to propose we include this in the standard. Thoughts? Well enough for one message. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Dec 14 15:48:42 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24886; Mon, 14 Dec 92 15:48:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17233; Mon, 14 Dec 92 15:48:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Dec 1992 20:48:20 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17227; Mon, 14 Dec 92 15:48:19 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA03749; Mon, 14 Dec 92 15:48:17 -0500 Date: Mon, 14 Dec 92 15:48:17 -0500 Message-Id: <9212142048.AA03749@thud.cs.utk.edu> To: mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum Forwarding: Mail from '"Dr. C.D. Wright" ' dated: Mon, 14 Dec 92 12:16:10 GMT ---------- Begin Forwarded Message ---------- >From @ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK Mon Dec 14 07:20:05 1992 Return-Path: <@ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK> Received: from mail.liv.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22696; Mon, 14 Dec 92 07:19:55 -0500 Received: from ibm.liverpool.ac.uk by mailhub.liverpool.ac.uk via JANET with NIFTP (PP) id <21042-0@mailhub.liverpool.ac.uk>; Mon, 14 Dec 1992 12:19:28 +0000 Received: from UK.AC.LIVERPOOL by MAILER(4.4.t); 14 Dec 1992 12:20:02 GMT Date: Mon, 14 Dec 92 12:16:10 GMT From: "Dr. C.D. Wright" Subject: Re: Message Passing Interface Forum To: dongarra@edu.utk.cs Message-Id: <"mailhub.li.044:14.11.92.12.19.28"@liverpool.ac.uk> Status: RO Hi. Since I am in the UK it is clear that I can't actively participate in the MPI Forum. I do, however, have one particular problem with every comms library I have used so far that I would like to see addressed in any new "standard", and I hope you can pass this on to whoever is the appropriate person to deal with it. In many packages such as PVM, PARMACS, p4, etc, it is possible to probe for and/or receive messages selectively, the selection being based on the message type (usually in integer) and/or the sender. This is overly restrictive. It would be far more useful if the message's format were sufficiently well defined for the user to be able to provide their own selection function to be passed in and used as the basis for reception and/or probing. That's it. Hope you can do something with this gripe/suggestion. Colin. ----------- End Forwarded Message ----------- From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 15 19:28:34 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21803; Tue, 15 Dec 92 19:28:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15587; Tue, 15 Dec 92 19:28:11 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 16 Dec 1992 00:28:10 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from helios.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15573; Tue, 15 Dec 92 19:28:06 -0500 Received: by helios.llnl.gov (4.1/LLNL-1.18) id AA11599; Tue, 15 Dec 92 16:30:03 PST Date: Tue, 15 Dec 92 16:30:03 PST From: tony@helios.llnl.gov (Anthony Skjellum) Message-Id: <9212160030.AA11599@helios.llnl.gov> To: dongarra@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum That is what we have been talking about in Zipcode for a long time. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Dec 25 19:15:36 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20495; Fri, 25 Dec 92 19:15:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09422; Fri, 25 Dec 92 19:15:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 26 Dec 1992 00:15:05 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Fri, 25 Dec 92 19:15:04 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA14461; Fri, 25 Dec 92 19:14:59 -0500 Date: Fri, 25 Dec 92 19:14:59 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9212260014.AA14461@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: point-to-point proposal consistent with collective proposal Point-to-point Communication Proposal ------------------------------------- After reading Marc Snir's Message Passing Interface Outline and the other point-to-point comments, I would like to propose the following minimum set of point-to-point routines for discussion at the January meeting in Dallas. Stated Goals: 1. Design an application programming interface. 2. Design an interface that is not too different from current practice 3. Define an interface that can be quickly implemented on many vendor platforms. 4. Focus on a proposal that can be agreed upon in 6 months. 5. Provide a reliable communication interface. 6. Allow efficient communication. 7. Interface should not contain more functions than are really necessary. ----------------------------------------------------------------------------- General Comments: The collective communication interface should be an extension of the point-to-point interface. a. As Marc points out on page 2 "SEND and RECV are a particular case of broadcast in a group of size 2; this observation can be used to check if the definition of collective communication semantics are consistent with the definition of point-to-point communication." b. Collective routines like broadcast should provide the same message data format (contiguous, buffer with stride, etc.) as point-to-point routines. c. By using a structured name space (described on pages 7-8 of [Snir]) where all processes are identified by a (group,rank) pair for both the point-to-point and collective routines, then the users will have a consistent naming scheme across all the MPI communication routines. And those desiring a flat name space can have it by using the default group "ALL". d. Syntax guidelines - short, mnemonic names; systematic ordering of parameters; systematic naming convention. ------------------------------------------------------------------------------- Proposed point-to-point routines: 1. info = MPI_SEND( buf, bytes, type, gid, dest ) Function: Sends a contiguous message of length "bytes" pointed to by "buf" to the process identified by ("gid","dest") with a message tag of "type". On return "info" contains the error code. Conceptually the user can view this function as "I want this message delivered to the specified process in the fastest reliable manner." Implementation comments: On return user can consider buf is safe for reuse. MPI_SEND need not be strictly blocking. Users do not want to have to dance around the send buffer with STAUS and WAIT checking for when the send buffer is safe to change. If a compiler can determine that (or rearrange code so that) the send buffer is not touched before the send completes, then the implementation could be non-blocking. If a multiprocessor requires synchronous communication or routing along a particular topology, then this could be set in a global routine that affects all point-to-point and collective routines. 2. info = MPI_RECV( buf, bytes, type, gid, source ) Function: Receive the message specified by "type", "gid", and "source" into "buf". The message must be <= "bytes" in size or "info" returns error. Routine waits until specified message has arrived [like blocking recv]. On return "info" contains the error code. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. Conceptually the user can view this function as "I must have the specified message before I can do anything else. I will wait until it arrives." 3. info = MPI_NRECV( buf, bytes, type, gid, source ) Function: Checks to see if the specified message has arrived [like probe]. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. If the message has arrived it is returned in "buf" and "info" = 1. The message must be <= "bytes" in size or "info" returns error. If the message has not arrived, then the routine returns with "info" = 0. Values less than 0 returned by "info" indicate an error. MPI_NRECV can be called multiple times to check on the arrival of a message, and the user can later call MPI_RECV to wait on the same message. Conceptually the user can view this function as "I would like to receive the specified message now, but if it is not here, I will not wait for it. I will go do something else." Implementation is not specified here, but for efficiency, implementors may wish to post a receive on the first call to MPI_NRECV and just do checks [like status] on repeat calls. If implemented like this, MPI_RECV must be able to check both the system queue and the posted queue. 4. type = MPI_TYPE() 5. bytes = MPI_BYTES() 6. gid = MPI_GID() 7. source = MPI_SOURCE() Function: These routines return the actual values for the last message returned by MPI_RECV or MPI_NRECV. Values < 0 indicate an error. Allows the user to determine the size of the received message and the values of "don't care" arguments. 8. bytes = MPI_PACK( data_type, nitems, stride, inbuf, outbuf ) 9. bytes = MPI_UNPACK( data_type, nitems, stride, inbuf, outbuf ) Function: General purpose routines for packing and unpacking an arbitrarily complex message buffer. Allows for non-contiguous messages, mixed data types in a single message, and heterogeneity between machines. Same routines can be used to build collective communication buffers, for example for MPI_BCAST. On return "outbuf" contains the (potentially) encoded data, and "bytes" is the size in bytes of the encoded data. "bytes" < 0 indicates an error. The following data types would be supported: BYTE INTEGER*2 INTEGER*4 REAL*4 REAL*8 COMPLEX*8 COMPLEX*16 I would also like to see the following data types considered: CHARACTER_STRING INTEGER*8 REAL*16 COMPLEX*32 "nitems" is the total number of items of this data type to pack or unpack from "inbuf". "stride" can be used for packing rows of a matrix in Fortran or columns of a matrix in C. MPI_PACK can be called multiple times to construct a single message buffer. MPI_UNPACKs must be called by the user to decode such a buffer on receipt. Neither of these routines need be called if the user does not require non-contiguous messages, mixed data types, or heterogeneity. Having separate packing and unpacking routines allows send and receive to be optimized for a contiguous stream of bytes, and simplifies future extensions of MPI into areas such as heterogeneity. =============================================================================== For completeness here is the present proposal from the collective communication subcommittee. 1. info = MPI_BCAST( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return the contents of "buf" on "root" is contained in "buf" on all group members. On return "info" contains the error code. 2. info = MPI_GATHER( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return all the individual "buf" are concatenated into the "root" buf, which must be of size at least gsize*bytes where gsize is group size. The data is laid in the "root" buf in rank order that is | gid,0 data | gid,1 data | ...| gid, root data | ...| gid, gsize-1 data | Other member's "buf" are unchanged on return. On return "info" contains the error code. 3. info = MPI_GLOBAL_OP( inbuf, bytes, type, gid, op, outbuf ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "op". On return the "outbuf" of all group members contains the result of the global operation "op" applied pointwise to the collective "inbuf". For example, if the op is max and inbuf contains two float point numbers then outbuf(1) = global max( inbuf(1)) and outbuf(2) = global max( inbuf(2)) A set of standard operations are supplied with MPI including: global max - for each data type global min - for each data type global sum - for each data type global mult- for each data type global AND - for integer and logical type global OR - for integer and logical type global XOR - for integer and logical type Optionally the users may define their own global functions for this routine. On return "info" contains the error code. 4. info = MPI_SYNCH( gid ) Function: Called by all members of the group "gid" Returns only when all members have called this function. On return "info" contains the error code. 5. gid = MPI_MKGROUP( list_of_processes ) Function: Called by all processes in the list. Forms a logical group containing the listed processes and assigns each process a unique rank in the group. The ranks are consecutively numbered from 0 to gsize-1. On return "gid" is an MPI assigned group ID (or error code if < 0) 6. gsize = MPI_GROUPSIZE( gid ) Function: Can be called by any process. On return "gsize" is the number of members in the group "gid" (or error code if < 0). 7. rank = MPI_MYRANK( gid ) Function: Can be called only by members of group "gid". On return "rank" is the rank of the calling process in group "gid" (an integer between 0 and gsize-1) or error code if < 0. =========================================================================== From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:50:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19230; Mon, 4 Jan 93 08:50:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07603; Mon, 4 Jan 93 08:49:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:49:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07594; Mon, 4 Jan 93 08:49:43 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06801 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:49:38 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15719; Mon, 4 Jan 93 13:49:35 GMT Date: Mon, 4 Jan 93 13:49:35 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041349.AA15719@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02659; Mon, 4 Jan 93 13:49:02 GMT To: mpi-pt2pt@cs.utk.edu Subject: Multifarious Content-Length: 6062 Gentlepeople, Firstly a general point ======================= I'd like to add an additional criterion to the goals of the message passing interface. 8) The implementation and semantics of the interface should remain simple and "obvious" when multiple "lightweight" threads share the same address space. I'd like this because 1) Lightweight thread libraries on the Posix model are becoming available on most workstations (networks of which are one of our targets). 2) There is a clear industry trend to having a few processors on a logically (and often physically) shared memory both in workstations (e.g. SparcStation 10), and servers (SparcServer 690, RS6000 ??). It would be foolish to go down a path which made exploitation of such machines hard. Undoubtedly this trend will also occur in future MPP machines (it would "only" take a software change to the code on the Paragon to make it be like this. After all it has all the hardware there already). As a consequence of this I'd like to remove as much hidden state as possible from the message passing system, so 1) I dislike the functions which return information about "the last message", since in a single process with multiple threads there can be many "last" messages. The suggestion that these operations be performed on descriptors returned from the probe is much preferable (and semantically cleaner). 2) I dislike the idea of setting options in a global routine, because there may be other threads performing operations at the same time as the options are changing, and the semantics becomes very hard to pin down. Secondly A few comments on Al Geist's proposal of Christmas Day =============================================================== 1. MPI_SEND I liked the separation in the original draft between the concepts of blocking (the buffer is now free) and synchronous (the message has been received at the other end) communications. I'd like to preserve both possibilities. 3. MPI_NRECV I don't understand how this is intended to work. Consider the following code. char buffer[100]; int i; for (i=0; i<10; i++) { if (&MPI_NRECV(buffer[10*i],10,1,1,1)) break; } Questions : 1) Into which elements of buffer will the receive occur ? Al's implementation note suggests just the first, so what then do the other (up to) 9 calls achieve ? Will they receive the next nine matching messages ? (this doesn't appear very intuitive for a "polling" routine) 2) Should I now do a blocking MPI_RECV as well ? Or does this only make sense if the MPI_NRECVs all fail ? But if they do fail, is the receive buffer still queued to receive a matching message when it becomes available ? (and if so how do I cancel the outstanding receive and get back ownership of my buffer ?) 4,5,6,7 See comments above about this style of working. 8,9 If we're only having one form of packing, then it should probably be the more general one with arrays of pointers and counts, since it can subsume (admittedly at some cost) the strided style. Conclusion ========== I prefer the original draft to Al's version. I'd like to change the mode argument in the original draft (see my mail to mpi-lang). Removing internal state ======================= I'd also like to remove the hidden state ("last message"), there are various ways of achieving this, none of which is exceptionally attractive. 1) Return a structure from MPI_RECV (and friends) This is fine in C, but not good in Fortran 77 (and may be more expensive than we want when you don't need any of the information). e.g. typedef struct MPI_REPLY { int status; int nbytes; int whoFrom; int tag; /* More ? */ } MPI_REPLY; { MPI_REPLY res; res = MPI_RECV(...); if (res.whoFrom == 0) ... etc ... } 2) Return results indirect through individual parameters (arguments for the Fortran inclined) This works, but is very error prone, particularly in Fortran when the arguments are INOUT. (We have an interface like this at the moment, and it does cause problems). 3) Return a tag on which queries can be made. This means that you must then have another call to the system to dispose of the tag (and associated system storage). This is unpleasant on a simple blocking receive when you don't want any of the information (since it's another call to the library). 4) Have additional (possibly optional) OUT arguments through which the results are returned if required. Using optional keyword arguments in F90 this works fine. C can be made to work more or less. F77 can't (within the standard). 5) Have one required argument which is a pointer to a reply structure like that used in 1) which is then filled in by the function. This can work in all the languages. In C you could also allow NULL to mean no such results are required. IMHO Solution 5 is best. If we wanted to (and we may) then we could disallow direct access to this structure and provide functions to pick it apart (these could be macros in C, inline functions in C++). This would avoid having to specify the contents of the structure and allow for future expansion. Comments on Steve Lederman's note ================================= > At the meeting in Minneapolis, several people, including myself, > discussed the advantage of having a send where the sending process > knows the receiving process has already posted a receive. We have this in our current message passing model. It has proved to be an awful mistake. 98% of user's don't know when to use it, and then complain that their code no longer works when they've used it incorrectly. Please let's not do this. (I'm speaking from experience here). See you all in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:59:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19312; Mon, 4 Jan 93 08:59:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08031; Mon, 4 Jan 93 08:59:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:59:32 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08019; Mon, 4 Jan 93 08:59:30 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06920 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:59:25 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15744; Mon, 4 Jan 93 13:59:21 GMT Date: Mon, 4 Jan 93 13:59:21 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041359.AA15744@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02661; Mon, 4 Jan 93 13:58:46 GMT To: mpi-pt2pt@cs.utk.edu Subject: Incorrect Example Content-Length: 311 Appologies, the example should (of course) read for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 09:33:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19844; Mon, 4 Jan 93 09:33:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09262; Mon, 4 Jan 93 09:32:48 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 14:32:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09242; Mon, 4 Jan 93 09:32:36 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA07241 (5.65c/IDA-1.4.4); Mon, 4 Jan 1993 09:32:32 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15874; Mon, 4 Jan 93 14:32:28 GMT Date: Mon, 4 Jan 93 14:32:28 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041432.AA15874@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02690; Mon, 4 Jan 93 14:31:52 GMT To: mpi-pt2pt@cs.utk.edu, mpi-lang@cs.utk.edu Subject: Profilers etc. Content-Length: 2528 Gentlepeople, I have an implementation issue which I would like to raise in the MPI forum, since it is unclear where it fits into the sub-committee structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to those of you who receive this message twice. Issue ===== The major objective of MPI1 is to achieve portability of applications. This has major benefits for us all (not least in legitimising and therefore growing the MPP marketplace). One of the benefits which it would also be nice to achieve would be the wide availability of different tools which support programming in the MPI1 model. The most immediately obvious such tools (to me at least !) are 1) HPF to Fortran + MPI1 translators 2) Performance monitoring/tuning tools 3) Debuggers Support for the first is easy (since it just requires what we're already doing). Portable support for the second is not so trivial, since the collection of useful performance information is much more intrusive. Portable support for the third is harder still, and I won't discuss it further. Options ======= We have various possible options which we can take. 1) Ignore the problem Provide no support for portable performance monitoring tools, and leave each tool provider with a large porting problem. I don't like this solution, it loses some of the benefit of the standard, which should be attracting people to build tools. 2) Document specific implementation hooks as part of MPI1. In effect these would be callbacks from the library to profiling code which could then do whatever it liked. 3) As 2, but without REQUIRING that a conforming implementation provide the functions. They're there as a recommendation, rather than being mandatory. I think we should be concerned about this, and I'd like us at least to make some recommendation. Personally I'd probably implement two separate interface to the library, one of which provided the hooks, and the other of which didn't so that you don't pay the cost of checking the profiling hooks unless you asked to. Of course even if we do nothing it's not too hard to escape (using horrible macros) in C, but Fortran doesn't always have macros, so a properly specified internal solution is definitely preferable. Thoughts ??? Flame me at Dallas. I'm travelling tomorrow. When are we having a meeting in Europe ??? -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:26:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21722; Mon, 4 Jan 93 10:26:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11937; Mon, 4 Jan 93 10:25:39 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:25:38 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11921; Mon, 4 Jan 93 10:25:36 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA02891; Mon, 4 Jan 93 10:25:20 -0500 Date: Mon, 4 Jan 93 10:25:20 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301041525.AA02891@msr.EPM.ORNL.GOV> To: jim@meiko.co.uk, mpi-pt2pt@cs.utk.edu Subject: NRECV example explained. Hi Jim, Your points are well taken given your suggested additional goals for MPI. You asked some questions about an example you gave. char buffer[100]; int i; for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } > 1) Into which elements of buffer will the receive occur ? The MPI_NRECV call says I want to receive a message not to exceed 10 bytes which matches type = 1, groupID = 1, source = 1. If the message has already arrived before the NRECV call, then it will be placed in buffer[0:9]. If it arrives while the 'for' loop is still executing, then it will be placed in buffer[10*i:10*i+10]. If it arrives later, then the user will need to ask for it later with either MPI_NRECV or MPI_RECV. All he can say for sure is the message did not arrive while I was polling for it. It is a strange example in that the user asks for a single message but specifies 10 different places where it should be stored. Ultimately, it the successful RECV or NRECV that specifies where the message is stored. > 2) Should I now do a blocking MPI_RECV as well ? Or does this only > make sense if the MPI_NRECVs all fail ? But if they do fail, is > the receive buffer still queued to receive a matching message when > it becomes available ? (and if so how do I cancel the outstanding > receive and get back ownership of my buffer ?) It only makes sense to call MPI_RECV if all MPI_NRECVs fail. It is not specified that the recv buffer ever has to be queued. An efficient immplementation may queue a failed NRECV in the educated guess that the user will ask for the message later in the same buffer. Then when the user does ask and the RECV or NRECV is successful the system avoids one copy of the message. Technically the user never loses ownership of the buffer. He can do anything he wants with it. If he is sure the posted message will never arrive, then he can post a message from another source to that same buffer. Cheers! Al From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:44:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22293; Mon, 4 Jan 93 10:44:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12552; Mon, 4 Jan 93 10:44:12 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:44:11 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12544; Mon, 4 Jan 93 10:44:08 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA08401 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 10:44:05 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16038; Mon, 4 Jan 93 15:44:00 GMT Date: Mon, 4 Jan 93 15:44:00 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041544.AA16038@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02725; Mon, 4 Jan 93 15:43:24 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Mon, 4 Jan 93 10:25:20 -0500 <9301041525.AA02891@msr.EPM.ORNL.GOV> Subject: NRECV example explained. Content-Length: 1350 Thanks Al, I now understand what you're trying to do, but I still don't think that your suggested "efficient implementation" is safe. Consider an example like this (less perverted than my previous one, which was of course constructed solely to make a point). strcpy(buffer0,"Something"); if (MPI_NRECV(buffer0, 10, 1, 1, 1)) /* Whatever */ else { sleep(20); /* Wait for the message to arrive, * For the sake of argument assume it does. */ MPI_RECV(buffer1, 10, 1, 1, 1); if (strcmp(buffer0,"Something") != SAME) bug("Buffer changed"); } Your suggested "efficient" implementation will surely overwrite buffer0 (which it should NOT do in your semantic model, since "Ultimately, it the successful RECV or NRECV that specifies where the message is stored."). I have to say I much prefer a model in which the queuing and dequeuing of buffers is explicit. Then it is clear where the current ownership of the buffer resides at any time, and when the additional copy can be avoided. (Though of course there is a lovely race to be resolved in the cancel...) Maybe we can talk about this further over a beer if you're going to be in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 12:37:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20368; Tue, 5 Jan 93 12:37:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13588; Tue, 5 Jan 93 12:37:19 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 17:37:18 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13576; Tue, 5 Jan 93 12:37:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA22236; Tue, 5 Jan 93 17:37:12 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00742; Tue, 5 Jan 93 10:36:12 MST Date: Tue, 5 Jan 93 10:36:12 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301051736.AA00742@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Multifarious I have a few short comments on Jim's comments... > > Removing internal state > ======================= > > I'd also like to remove the hidden state ("last message"), there are > various ways of achieving this, none of which is exceptionally > attractive. > > 1) Return a structure from MPI_RECV (and friends) > This is fine in C, but not good in Fortran 77 (and may be more > expensive than we want when you don't need any of the information). > > e.g. > typedef struct MPI_REPLY > { > int status; > int nbytes; > int whoFrom; > int tag; > /* More ? */ > } MPI_REPLY; > > { > MPI_REPLY res; > > res = MPI_RECV(...); > > if (res.whoFrom == 0) > > ... etc ... > } > > > 2) Return results indirect through individual parameters (arguments for the > Fortran inclined) > > This works, but is very error prone, particularly in Fortran when > the arguments are INOUT. (We have an interface like this at the > moment, and it does cause problems). > > 3) Return a tag on which queries can be made. > > This means that you must then have another call to the system to > dispose of the tag (and associated system storage). This is > unpleasant on a simple blocking receive when you don't want any of > the information (since it's another call to the library). > > 4) Have additional (possibly optional) OUT arguments through which the > results are returned if required. > > Using optional keyword arguments in F90 this works fine. > C can be made to work more or less. > F77 can't (within the standard). > > 5) Have one required argument which is a pointer to a reply structure > like that used in 1) which is then filled in by the function. > > This can work in all the languages. > In C you could also allow NULL to mean no such results are > required. > > > IMHO Solution 5 is best. > > If we wanted to (and we may) then we could disallow direct access to > this structure and provide functions to pick it apart (these could be > macros in C, inline functions in C++). This would avoid having to > specify the contents of the structure and allow for future expansion. > > > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England I also like Solution 5 and like the idea of providing functions to access the reply structure. "Hiding" the data structure in this way makes it easy to change things in later revisions when we (inevitably) find out we've forgotten something. Tom Henderson Forecast Systems Laboratory (NOAA) hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 15:52:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01504; Tue, 5 Jan 93 15:52:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22365; Tue, 5 Jan 93 15:51:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 20:51:34 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22325; Tue, 5 Jan 93 15:50:59 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA21624; Tue, 5 Jan 93 14:50:42 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA17798; Tue, 5 Jan 93 14:50:40 CST Date: Tue, 5 Jan 93 14:50:40 CST Message-Id: <9301052050.AA17798@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Comments on "ready receiver" part of draft implementation Here is some text supporting the idea of "ready receivers". Bill Gropp and Rusty Lusk %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: rr.dvi %%Pages: 3 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}B /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}{initclip}ifelse ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}B /@endspecial{grestore clear SpecialSave restore end}B /@defspecial{SDict begin}B /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}B /st{stroke SaveX SaveY moveto}B /fil{fill SaveX SaveY moveto}B /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 58 123 df<001F83E000F06E3001C078780380F8 780300F03007007000070070000700700007007000070070000700700007007000FFFFFF800700 700007007000070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070007FE3FF001D20809F 1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<7038F87CFC7EFC7E743A 0402040204020804080410081008201040200F0E7E9F17>34 D<70F8FCFC740404040808101020 40060E7C9F0D>39 D<0020004000800100020006000C000C001800180030003000300070006000 60006000E000E000E000E000E000E000E000E000E000E000E000E0006000600060007000300030 003000180018000C000C000600020001000080004000200B2E7DA112>I<800040002000100008 000C00060006000300030001800180018001C000C000C000C000E000E000E000E000E000E000E0 00E000E000E000E000E000C000C000C001C001800180018003000300060006000C000800100020 00400080000B2E7DA112>I<70F8FCFC74040404080810102040060E7C840D>44 DI<70F8F8F87005057C840D>I<000100030003000600060006000C000C 000C00180018001800300030003000600060006000C000C000C001800180018003000300030006 00060006000C000C000C00180018001800300030003000600060006000C000C000C000102D7DA1 17>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0700380700380700380 7807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 80038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F8 03C0F803C02007C00007C0000780000780000F00000E00001C0000380000700000600000C00001 80000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00 100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C0000380003F000 003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780400780400F00 200E001C3C0003F000121F7E9D17>I<007C000182000701000E03800C07801C07803803003800 00780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003 C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>54 D<03F0000C0C001006003003002001806001806001806001807001807803003E03003F06001FC8 000FF00003F80007FC000C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C000 806001802001001002000C0C0003F000121F7E9D17>56 D<70F8F8F87000000000000000000000 70F8F8F87005147C930D>58 D<000100000003800000038000000380000007C0000007C0000007 C0000009E0000009E0000009E0000010F0000010F0000010F00000207800002078000020780000 403C0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F0002000780 0200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 D<000FC040007030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C00 00407C00004078000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000407C0000403C0000401C0000401E0000800E00008007000100 0380020001C0040000703800000FC0001A217D9F21>67 DI70 D73 D76 DII<001F800000F0F00001C0380007801E000F000F000E0007001E0007803C 0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0 F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007 800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C198010078030038060 0180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE0003 FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300F0 0600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030 800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>I87 D<080410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>92 D<1FE000303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E 00F00E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E0 0E00E00E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E0 0D01C00CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F0 00F00070007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380 00038000038000038000038000038000038000038003E380061B801C0780380380380380700380 700380F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B80 03E3F815207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF 80F00000F00000F000007000007000003800801800800C010007060001F80011147F9314>I<00 7C00C6018F038F07060700070007000700070007000700FFF00700070007000700070007000700 070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C 301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000 003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000 C01C038003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I< 1C001E003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000 E0000000000000000000000000007007F000F00070007000700070007000700070007000700070 007000700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I< 0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF0 0E03C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E00 0E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F 00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7F E023147F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800 070E001C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E0 7000E03801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00 E00E00F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01 C00EC3000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319 >I<03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0 03807003807803803803803807801C0B800E138003E38000038000038000038000038000038000 0380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0 0078007F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002 00060006000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E08 0E080E080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1 FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00 007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E01 1E011C0338027006700EFFFE10147F9314>I E /Fb 14 122 df<01FFFFC0001E00F0001E0078 001E0038001E003C003C003C003C003C003C003C003C003C0078007800780078007800F0007801 E000F0078000FFFE0000F00F8000F003C001E001C001E001E001E001E001E001E003C001E003C0 01E003C001E003C001C0078003C00780078007800F0007801E000F007800FFFFE0001E1F7D9E20 >66 D<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>71 D<01FFF800001F0000001E000000 1E0000001E0000003C0000003C0000003C0000003C000000780000007800000078000000780000 00F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0008003C0010003C001 0003C0030003C00200078006000780060007800C0007801C000F007800FFFFF800191F7D9E1D> 76 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C0078 007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C0001E01E 0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C0807803C080780 3C100F801C10FFF00C20000007C01D207D9E21>82 D<00C001E001E001C0000000000000000000 00000000000E003300230043804300470087000E000E000E001C001C001C003840388030807080 310033001C000B1F7C9E0E>105 D<01E0000FE00001C00001C00001C00001C000038000038000 0380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80001D00001E0000 1FC00038E000387000387000383840707080707080707080703100E03100601E0013207D9F15> 107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C00 38003800380038007000700070007100E200E200E200E200640038000A207C9F0C>I<007C0001 C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F0 0700700F00700E0030180018700007C00013147C9317>111 D<01C1E002621804741C04781C04 701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380780380700380E003 C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC000171D8093 17>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF00CE00840102060 1F8010147D9313>I<018001C0038003800380038007000700FFF007000E000E000E000E001C00 1C001C001C003800380038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E0 2301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20 180E20180E201C1E200C264007C38014147C9318>I<0E00C03300E02301C04381C04301C04701 C08703800E03800E03800E03801C07001C07001C07001C07001C0E00180E00180E001C1E000C3C 0007DC00001C00001C00003800F03800F07000E06000C0C0004380003E0000131D7C9316>121 D E /Fc 20 117 df45 D<000003800000000007C00000000007C0000000000FE0000000000FE0000000000FE000000000 1FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC0000000073FC00 000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003C0FF80000003 807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE000001E001FF0 00001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC0000700003FC000070 0003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF00FFFE001FFF FEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D 72 D77 DI80 D<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F 800FC03F801F803F803F801F007F800E007F0000007F000000FF000000FF000000FF000000FF00 0000FF000000FF000000FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007 E0070003F01E0000FFFC00001FE0001A1B7E9A1F>99 D<003FE00001FFF80003F07E0007C01F00 0F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFF E0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0 038003F81F0000FFFE00001FF0001B1B7E9A20>101 D<0007F0003FFC00FE3E01F87F03F87F03 F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7EA915>I< 00FF81F003FFE7F80FC1FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F 007F007F007F007F003F007E001F007C001F80FC000FC1F8001FFFE00018FF8000380000003800 00003C0000003E0000003FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C 0001F8F80000F8F80000F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00 007FF0001E287E9A22>II<07000F801FC0 3FE03FE03FE01FC00F8007000000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFE FFFE0F2B7DAA14>I108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E0 7F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007 F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003F E0001D1B7E9A22>I114 D<03FE300FFFF01E03F03800F0700070F00070 F00070F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007C E0003CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00161B7E9A1B>I<0070000070000070 0000700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F0 3807F03807F03807F03807F03803F03803F87001F86000FFC0001F8015267FA51B>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 117 7 a Fc(A)23 b(Note)f(on)i(Access)e(to)h(High-P)n(erformance)e (Message-P)n(assing)832 94 y(Proto)r(cols)869 240 y Fb(Bil)r(l)16 b(Gr)n(opp)864 297 y(R)o(usty)g(Lusk)526 353 y Fa(Mathematics)f(and)g (Computer)g(Science)i(Division)678 409 y(Argonne)e(National)h(Lab)q(oratory) 145 546 y(The)22 b(MPI)h(Implemen)o(tation)g(T)l(estb)q(ed)g(con)o(tains)g (what)f(are)g(called)i(there)e(\\Ready-Receiv)o(er")74 602 y(v)o(ersions)17 b(of)g(the)g(message-passing)g(primitiv)o(es.)27 b(The)17 b(purp)q(ose)h(of)e(this)i(brief)f(note)g(is)h(to)e(giv)o(e)h(some) 74 659 y(explanation)e(of)f(what)g(wh)o(y)f(w)o(e)h(think)h(something)g(lik)o (e)g(this)f(needs)h(to)f(b)q(e)h(considered)g(as)f(part)f(of)h(the)74 715 y(ev)o(en)o(tual)i(standard)e(and)i(to)e(rep)q(ort)h(on)g(some)g(exp)q (erimen)o(ts)h(that)f(supp)q(ort)g(this)h(opinion.)145 802 y(It)21 b(is)g(quite)h(p)q(ossible)h(to)e(enco)q(de)h(message-passing)f (algorithms)g(with)h(only)f(a)g(single)h(t)o(yp)q(e)g(of)74 858 y(send)d(and)g(receiv)o(e,)h(sa)o(y)e(a)g(blo)q(c)o(king)i(send)f(and)g (a)f(blo)q(c)o(king)i(receiv)o(e.)30 b(Most)18 b(of)g(the)g(v)m(ariations)h (are)74 914 y(to)c(enable)h(increased)g(p)q(erformance,)f(p)q(ossibly)i(at)d (the)h(exp)q(ense)i(of)d(increased)i(program)e(complexit)o(y)l(.)74 971 y(An)19 b(example)h(of)f(this)g(is)g(the)g(general)h(consensus)f(that)g (w)o(e)f(need)i(non-blo)q(c)o(king)h(send)e(and)g(receiv)o(e)74 1027 y(op)q(erations)c(in)i(order)d(to)h(o)o(v)o(erlap)g(computation)g(with)h (comm)o(unication.)145 1114 y(It)11 b(is)h(imp)q(ortan)o(t)f(not)g(to)g(stop) g(to)q(o)g(so)q(on,)g(ho)o(w)o(ev)o(er,)g(in)h(pro)o(viding)h(supp)q(ort)e (for)g(high)h(p)q(erformance.)74 1170 y(As)k(time)g(go)q(es)f(b)o(y)l(,)h(v)o (endors)f(will)j(pro)o(vide)e(op)q(erations)g(that)f(are)g(faster)g(than)g (the)h(ones)g(that)f(can)g(b)q(e)74 1227 y(sp)q(eci\014ed)i(with)d(the)g(op)q (erations)h(in)g(the)f(initial)j(standard)c(prop)q(osal.)20 b(It)15 b(is)f(p)q(ossible)i(to)e(an)o(ticipate)h(at)74 1283 y(least)g(some)g(of)g(these)h(op)q(erations)f(and)g(pro)o(vide)h(for)f(them)g (in)h(the)f(MPI)g(standard.)145 1370 y(One)j(example)g(o)q(ccurs)f(in)h(the)g (situation)f(when)h(receiv)o(es)g(ha)o(v)o(e)f(b)q(een)h(issued)g(prior)g(to) e(the)i(send)74 1426 y(op)q(eration)e(that)f(is)i(exp)q(ected)g(to)e(satisfy) h(them.)22 b(In)16 b(suc)o(h)g(a)g(case)g(a)f(considerable)j(amoun)o(t)d(of)h (hand-)74 1483 y(shaking)23 b(can)f(b)q(e)g(eliminated)i(since)f(the)f (bu\013ers)g(ha)o(v)o(e)g(already)g(b)q(een)h(allo)q(cated.)41 b(On)22 b(the)g(In)o(tel)74 1539 y(IPSC/860)c(and)g(the)g(Delta,)g(one)h(can) f(tak)o(e)f(adv)m(an)o(tage)h(of)g(this)g(b)o(y)g(using)h(\\force)f(t)o(yp)q (es".)28 b(Almost)74 1595 y(all)20 b(users)f(of)f(the)h(Delta)f(who)h(are)f (seeking)i(go)q(o)q(d)e(p)q(erformance)h(use)g(this)g(tec)o(hnique.)32 b(Its)19 b(name)g(is)74 1652 y(misleading,)d(since)f(it)f(do)q(esn't)g(ha)o (v)o(e)g(an)o(ything)g(to)f(do)h(with)h(t)o(yp)q(es.)k(The)14 b(imp)q(ortan)o(t)g(thing)g(to)g(realize)74 1708 y(is)19 b(that)f(it)i(is)f (not)f(just)h(an)f(arcane)h(v)o(endor-sp)q(eci\014c)i(optimization,)f(but)f (a)f(general)i(situation)f(that)74 1765 y(transcends)13 b(an)o(y)g (particular)h(implemen)o(tation.)20 b(The)13 b(general)h(situation)f(is)h (that)e(when)i(receiv)o(es)g(ha)o(v)o(e)74 1821 y(b)q(een)i(issued)h(ahead)e (of)g(time,)g(message)g(latency)h(can)f(b)q(e)h(greatly)f(decreased.)145 1908 y(W)l(e)e(conducted)h(some)f(preliminary)h(exp)q(erimen)o(ts)g(on)f(the) g(IPSC/860)g(with)g(the)g(MPI)g(implemen-)74 1964 y(tation)19 b(testb)q(ed,)g(using)h(the)f(\\rr")f(v)o(ersions)h(of)f(send)i(and)f(receiv) o(e)h(that)e(it)h(supplies.)33 b(\(\\rr")17 b(stands)74 2021 y(for)d(\\ready)g(receiv)o(er".\))20 b(The)15 b(test)f(program)f(just)i (ping-p)q(ongs)g(a)g(message)f(bac)o(k)g(and)h(forth)f(b)q(et)o(w)o(een)74 2077 y(t)o(w)o(o)20 b(no)q(des.)39 b(The)22 b(follo)o(wing)g(graphs)f(sho)o (w)g(the)h(p)q(erformance)f(impro)o(v)o(emen)o(ts)g(pro)o(vided)i(b)o(y)e (the)74 2134 y(ready-receiv)o(er)16 b(op)q(erations.)145 2220 y(This)g(next)f(test)f(sho)o(ws)h(the)g(b)q(eha)o(vior)h(in)g(sending)g (around)g(a)e(ring.)963 2790 y(1)p eop %%Page: 2 2 bop 74 2161 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ftime.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 710 569 d 36 850 569 d 18 991 569 d 36 1132 569 d 18 1272 569 d 36 1413 569 d 18 1553 569 d 36 1694 569 d 18 1834 569 d 36 1975 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 817 553 m 1 -2 r -1 -1 r -1 1 r 2 v 1 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 2 -2 r -3 v -2 -2 r -3 -3 r -7 -3 r -2 -1 r -3 -2 r -1 -4 r -4 v s n 827 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 816 533 m 1 2 r 3 h 6 -3 r 4 h 2 1 r 2 2 r s n 820 535 m 6 -4 r 5 h 1 1 r 2 3 r 2 v s n 849 558 m -4 -2 r -2 -3 r 841 546 l -4 v 843 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 849 558 m -2 -2 r -2 -1 r -1 -2 r 843 546 l -4 v 844 536 l 1 -3 r 2 -1 r 2 -1 r s n 852 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 875 558 m -4 -2 r -3 -3 r 867 546 l -4 v 868 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 875 558 m -3 -2 r -1 -1 r -1 -2 r 868 546 l -4 v 870 536 l 1 -3 r 1 -1 r 3 -1 r s n 877 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s -24 1108 555 d -27 1110 558 d n 1110 558 m 1096 539 l 20 h s 9 1105 531 e n 1130 558 m -4 -2 r -2 -3 r 1123 546 l -4 v 1124 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1130 558 m -2 -2 r -2 -1 r -1 -2 r 1124 546 l -4 v 1125 536 l 1 -3 r 2 -1 r 2 -1 r s n 1133 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1156 558 m -4 -2 r -2 -3 r 1148 546 l -4 v 1150 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1156 558 m -3 -2 r -1 -1 r -1 -2 r 1150 546 l -4 v 1151 536 l 1 -3 r 1 -1 r 3 -1 r s n 1159 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1393 554 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -2 2 r -4 h -4 -2 r -2 -2 r -2 -3 r -1 -5 r -7 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 1 v -1 4 r -3 2 r -4 2 r -1 h -4 -2 r -2 -2 r -2 -4 r s n 1387 558 m -3 -2 r -2 -2 r -1 -3 r -2 -5 r -7 v 2 -4 r 2 -3 r 3 -1 r s n 1388 531 m 3 1 r 2 3 r 2 4 r 1 v -2 4 r -2 2 r -3 2 r s n 1411 558 m -4 -2 r -2 -3 r 1404 546 l -4 v 1405 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1411 558 m -2 -2 r -2 -1 r -1 -2 r 1405 546 l -4 v 1406 536 l 1 -3 r 2 -1 r 2 -1 r s n 1414 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1437 558 m -4 -2 r -2 -3 r 1429 546 l -4 v 1431 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1437 558 m -3 -2 r -1 -1 r -1 -2 r 1431 546 l -4 v 1432 536 l 1 -3 r 1 -1 r 3 -1 r s n 1440 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1665 558 m -3 -2 r -2 -2 r -4 v 2 -2 r 3 -2 r 6 h 3 2 r 2 2 r 4 v -2 2 r -3 2 r c s n 1665 558 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r s n 1671 546 m 2 2 r 1 2 r 4 v -1 2 r -2 2 r s n 1665 546 m -3 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 3 -1 r 6 h 3 1 r 2 1 r 1 3 r 5 v -1 3 r -2 1 r -3 1 r s n 1665 546 m -2 -1 r -1 -1 r -2 -3 r -5 v 2 -3 r 1 -1 r 2 -1 r s n 1671 531 m 2 1 r 1 1 r 2 3 r 5 v -2 3 r -1 1 r -2 1 r s n 1692 558 m -3 -2 r -3 -3 r 1685 546 l -4 v 1686 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1692 558 m -2 -2 r -1 -1 r -2 -2 r 1686 546 l -4 v 1687 536 l 2 -3 r 1 -1 r 2 -1 r s n 1695 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1718 558 m -4 -2 r -2 -3 r 1710 546 l -4 v 1712 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1718 558 m -2 -2 r -2 -1 r -1 -2 r 1712 546 l -4 v 1713 536 l 1 -3 r 2 -1 r 2 -1 r s n 1721 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1931 553 m 3 1 r 4 4 r -27 v s -25 1936 556 d 12 1931 531 e n 1961 558 m -4 -2 r -3 -3 r 1953 546 l -4 v 1954 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1961 558 m -3 -2 r -1 -1 r -1 -2 r 1954 546 l -4 v 1956 536 l 1 -3 r 1 -1 r 3 -1 r s n 1963 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1986 558 m -3 -2 r -3 -3 r 1979 546 l -4 v 1980 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1986 558 m -2 -2 r -1 -1 r -2 -2 r 1980 546 l -4 v 1981 536 l 2 -3 r 1 -1 r 2 -1 r s n 1989 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2012 558 m -4 -2 r -2 -3 r 2004 546 l -4 v 2006 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2012 558 m -2 -2 r -2 -1 r -1 -2 r 2006 546 l -4 v 2007 536 l 1 -3 r 2 -1 r 2 -1 r s n 2015 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 1440 569 569 d 36 569 569 e 18 569 713 e 36 569 857 e 18 569 1001 e 36 569 1145 e 18 569 1289 e 36 569 1433 e 18 569 1577 e 36 569 1721 e 18 569 1865 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 393 867 m 2 -1 r -2 -1 r -1 1 r 1 v 1 3 r 2 1 r 3 2 r 6 h 3 -2 r 2 -1 r 1 -3 r -2 v -1 -3 r -4 -2 r 398 857 l -2 -1 r -3 -3 r -1 -4 r -3 v s n 404 873 m 2 -2 r 1 -1 r 2 -3 r -2 v -2 -3 r -3 -2 r 398 857 l s n 392 848 m 1 1 r 3 h 6 -2 r 4 h 3 1 r 1 1 r s n 396 849 m 6 -3 r 5 h 2 1 r 1 2 r 3 v s n 425 873 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 873 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 846 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 873 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 873 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 846 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 873 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 846 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 873 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s -25 404 1158 d -27 405 1160 d n 405 1160 m 391 1141 l 20 h s 9 400 1133 e n 425 1160 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1160 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1133 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1160 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1160 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1133 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1160 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1133 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1160 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 407 1444 m -1 -1 r 1 -1 r 2 1 r 1 v -2 3 r -2 1 r -4 h 397 1447 l -2 -3 r -2 -2 r -1 -5 r -8 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 1 v -1 4 r -3 3 r -4 1 r -1 h 397 1437 l -2 -3 r -2 -4 r s n 401 1448 m -3 -1 r -2 -3 r -1 -2 r -2 -5 r -8 v 2 -4 r 2 -2 r 3 -2 r s n 402 1421 m 3 2 r 2 2 r 2 4 r 1 v -2 4 r -2 3 r -3 1 r s n 425 1448 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 3 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1448 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 2 -2 r s n 428 1421 m 3 2 r 1 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1448 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 454 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1448 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -1 r 3 -2 r s n 479 1421 m 3 2 r 1 1 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1448 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 3 -2 r s n 505 1421 m 3 2 r 1 1 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1448 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 531 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 398 1736 m -3 -1 r -2 -3 r -4 v 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 4 v -2 3 r -3 1 r c s n 398 1736 m -2 -1 r -1 -3 r -4 v 1 -2 r 2 -1 r s n 404 1725 m 2 1 r 1 2 r 4 v -1 3 r -2 1 r s n 398 1725 m -3 -2 r -2 -1 r -1 -3 r -5 v 1 -2 r 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 1 2 r 5 v -1 3 r -2 1 r -3 2 r s n 398 1725 m -2 -2 r -1 -1 r -2 -3 r -5 v 2 -2 r 1 -2 r 2 -1 r s n 404 1709 m 2 1 r 1 2 r 2 2 r 5 v -2 3 r -1 1 r -2 2 r s n 425 1736 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1736 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1709 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1736 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1736 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1709 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1736 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1709 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1736 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 2019 m 3 1 r 4 4 r -27 v s -26 375 2023 d 12 370 1997 e n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 710 2009 d -36 850 2009 d -18 991 2009 d -36 1132 2009 d -18 1272 2009 d -36 1413 2009 d -18 1553 2009 d -36 1694 2009 d -18 1834 2009 d -36 1975 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 713 e -36 2009 857 e -18 2009 1001 e -36 2009 1145 e -18 2009 1289 e -36 2009 1433 e -18 2009 1577 e -36 2009 1721 e -18 2009 1865 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s n 870 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -6 2 r -3 h -5 -2 r -4 -3 r -1 -3 r -2 -6 r -8 v 2 -5 r 1 -4 r 4 -3 r 5 -2 r 3 h 6 2 r 3 3 r 2 4 r s n 858 2065 m -3 -2 r -4 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 4 -3 r 3 -2 r s n 892 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 892 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 896 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 920 2053 d -24 921 2053 d n 921 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 933 2053 m 4 -2 r 2 -3 r -19 v s n 940 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 952 2053 m 4 -2 r 1 -3 r -19 v s 6 915 2053 e 12 915 2029 e 12 933 2029 e 12 952 2029 e -24 976 2053 d -24 978 2053 d n 978 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 990 2053 m 3 -2 r 2 -3 r -19 v s n 997 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1009 2053 m 3 -2 r 2 -3 r -19 v s 7 971 2053 e 12 971 2029 e 12 990 2029 e 12 1009 2029 e n 1033 2053 m -19 v 2 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1035 2053 m -19 v 1 -3 r 4 -2 r s -24 1052 2053 d -24 1053 2053 d 7 1028 2053 e 6 1047 2053 e 7 1052 2029 e -24 1071 2053 d -24 1072 2053 d n 1072 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1084 2053 m 4 -2 r 1 -3 r -19 v s 7 1065 2053 e 12 1065 2029 e 12 1084 2029 e n 1108 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1108 2053 d -24 1110 2053 d 7 1103 2053 e 12 1103 2029 e n 1144 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1134 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1160 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1175 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1175 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 4 2 r 3 3 r s n 1163 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1194 2065 m -29 v 2 -5 r 3 -2 r 3 h 4 2 r 2 3 r s n 1196 2065 m -29 v 1 -5 r 2 -2 r s 13 1189 2053 e n 1220 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1220 2053 d -24 1221 2053 d 7 1214 2053 e 12 1214 2029 e n 1245 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1245 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1249 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1273 2053 d -24 1274 2053 d n 1274 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1286 2053 m 4 -2 r 2 -3 r -19 v s 6 1268 2053 e 12 1268 2029 e 12 1286 2029 e n 1324 2050 m 2 3 r -7 v -2 4 r -2 1 r -3 2 r -7 h -3 -2 r -2 -1 r -4 v 2 -1 r 3 -2 r 9 -4 r 3 -1 r 2 -2 r s n 1307 2048 m 2 -2 r 3 -1 r 9 -4 r 3 -2 r 2 -1 r -5 v -2 -2 r -3 -2 r -7 h -4 2 r -1 2 r -2 3 r -7 v 2 4 r s -36 1367 2065 d -36 1369 2065 d n 1362 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -5 -2 r -13 h s n 1382 2065 m 4 -2 r 1 -1 r 2 -4 r -5 v -2 -3 r -1 -2 r -4 -2 r s 12 1362 2029 e n 1403 2043 m 20 h 3 v -1 4 r -2 1 r -3 2 r -6 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1422 2043 m 5 v -2 3 r s n 1411 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s -24 1437 2053 d -24 1439 2053 d n 1439 2043 m 2 5 r 3 3 r 3 2 r 6 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1432 2053 e 12 1432 2029 e n 1475 2063 m -2 -1 r 2 -2 r 2 2 r 1 v -2 2 r -4 h -3 -2 r -2 -3 r -31 v s n 1471 2065 m -1 -2 r -2 -3 r -31 v s 14 1461 2053 e 12 1461 2029 e n 1495 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1495 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s n 1499 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1523 2053 d -24 1525 2053 d n 1525 2043 m 1 5 r 4 3 r 3 2 r 5 h 2 -2 r -1 v -2 -2 r -1 2 r 1 1 r s 7 1518 2053 e 12 1518 2029 e -24 1552 2053 d -24 1554 2053 d n 1554 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1566 2053 m 3 -2 r 2 -3 r -19 v s n 1573 2048 m 3 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1584 2053 m 4 -2 r 2 -3 r -19 v s 7 1547 2053 e 12 1547 2029 e 12 1566 2029 e 12 1584 2029 e n 1608 2050 m -2 v -1 h 2 v 1 1 r 4 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1624 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1624 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 3 2 r 4 3 r s n 1612 2043 m -4 -2 r -1 -3 r -4 v 1 -3 r 4 -2 r s -24 1643 2053 d -24 1644 2053 d n 1644 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1656 2053 m 4 -2 r 2 -3 r -19 v s 6 1638 2053 e 12 1638 2029 e 12 1656 2029 e n 1698 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1687 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1711 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1730 2043 m 5 v -2 3 r s n 1720 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 657 974 614 806 b 20 -7 647 983 i 21 -7 647 984 i n 657 981 m -1 2 r 3 v 2 v 2 2 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -2 -2 r -2 -1 r s n 656 988 m 2 1 r 3 1 r 3 h 2 h 2 -2 r 1 -3 r 1 -2 r s 1 4 646 980 i 21 -8 654 1004 i 20 -7 655 1004 i 2 3 653 1001 i 2 7 674 993 i n 666 1016 m -4 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -3 r c s n 666 1016 m 1 -3 r 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s n 681 1013 m -1 2 r -1 3 r -2 2 r -2 h -4 h -2 -1 r -2 -2 r s n 678 1040 m 1 -2 r 1 1 r 1 v -1 1 r -3 -2 r -2 -1 r -1 -3 r -3 v 2 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 3 r s n 673 1035 m 1 -2 r 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s 20 -8 672 1052 i 21 -8 672 1053 i 6 -13 683 1060 i 9 3 687 1053 i 10 3 686 1052 i 1 4 671 1049 i 3 6 681 1057 i 3 6 691 1042 i 697 1058 695 1052 b n 679 1072 m 1 -1 r 1 h 2 v c s 14 -5 686 1070 i 14 -5 686 1071 i 1 4 685 1067 i 701 1069 699 1062 b 13 -5 690 1080 i 14 -5 690 1081 i n 693 1080 m -1 3 r 3 v 1 2 r 2 3 r 2 h 11 -4 r s n 693 1088 m 1 2 r 3 h 10 -4 r s 1 4 689 1077 i 3 7 702 1072 i 709 1090 706 1083 b n 699 1104 m -2 v 1 -1 r 1 -2 r 2 -1 r 2 1 r 2 h 1 2 r 1 2 r 2 v -1 1 r -1 2 r -2 1 r -3 h -1 -1 r -2 -2 r c s n 699 1102 m 1 -1 r 4 -2 r 3 h s n 709 1105 m -2 2 r -4 1 r -2 h s n 702 1109 m 1 v -1 2 r 1 h -2 v s n 705 1099 m 1 -2 r 2 -1 r -1 v 3 1 r 2 2 r 2 5 r 2 3 r 1 h s n 708 1095 m 2 1 r 2 3 r 2 4 r 2 3 r 2 h 1 h 1 -2 r -3 v -2 -6 r -2 -2 r -2 -1 r -1 1 r -2 1 r 4 v s 41 -155 708 1114 i 7 10 749 959 i 18 -12 755 991 i 18 -12 756 992 i n 764 986 m 3 v 2 v 1 2 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -1 r -3 -1 r s n 765 993 m 2 1 r 3 h 3 h 2 -2 r 2 -2 r -3 v -2 v s 756 992 754 988 b 18 -12 767 1009 i 18 -12 768 1010 i 2 4 766 1006 i 787 1000 783 994 b n 782 1018 m -1 -4 r 1 -2 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r -1 3 r -2 3 r -1 1 r -4 1 r -2 -1 r -3 -2 r c s n 782 1018 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 h 2 1 r s n 795 1011 m 2 v 3 v -2 3 r -2 1 r -3 1 r -3 -1 r -2 -1 r s n 799 1038 m 1 -1 r 1 h 1 v -1 1 r -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 3 r 3 v s n 793 1035 m -3 v 1 -3 r 2 -2 r 1 -1 r 3 -1 r 3 h 2 2 r s 18 -12 796 1051 i 17 -12 797 1052 i 3 -15 808 1057 i 10 1 811 1049 i 10 1 810 1048 i 3 3 794 1049 i 3 5 807 1054 i 4 6 812 1037 i 822 1051 818 1046 b n 808 1069 m -1 v 2 h -1 2 r c s 12 -8 814 1065 i 12 -8 815 1066 i 3 3 812 1063 i 828 1061 824 1055 b 12 -8 820 1075 i 12 -9 821 1076 i n 823 1074 m 3 v 1 3 r 1 1 r 2 2 r 3 h 9 -6 r s n 825 1081 m 2 2 r 2 -1 r 10 -6 r s 2 4 819 1072 i 4 6 831 1064 i 841 1079 837 1073 b n 835 1096 m -1 -2 r 1 -2 r 1 -2 r 1 -1 r 3 h 1 h 2 1 r 1 2 r 1 2 r -1 2 r -1 2 r -1 1 r -3 h -1 h -2 -1 r c s n 834 1094 m 2 -2 r 3 -3 r 2 h s n 845 1094 m -1 2 r -4 3 r -2 h s n 839 1099 m 2 v 2 v 1 -1 r -1 -1 r s n 840 1089 m -2 v 1 -2 r 1 h 2 h 3 2 r 3 4 r 2 2 r 2 h s n 842 1085 m 1 h 3 2 r 3 4 r 2 2 r 3 h 1 -1 r 1 -2 r -1 -3 r -4 -5 r -2 -2 r -2 1 r -1 h -1 2 r 3 v s 32 42 852 1113 i 45 52 884 1155 i 36 36 929 1207 i 16 -15 968 1264 i 15 -15 969 1265 i n 976 1258 m 3 v 1 2 r 2 1 r 2 2 r 3 h 3 -2 r 2 -1 r 1 -3 r -3 v -1 -3 r -2 -1 r -2 -1 r -3 h s n 979 1264 m 2 1 r 3 h 3 -1 r 1 -2 r 1 -3 r -3 v -2 v s 969 1265 966 1262 b 16 -15 983 1280 i 16 -15 984 1280 i 3 3 981 1277 i 1002 1267 997 1262 b n 999 1286 m -1 -3 r -3 v 1 -3 r 2 -2 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -1 3 r -1 1 r -3 2 r -3 h -3 -2 r c s n 999 1286 m -3 v -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 1 r s n 1011 1277 m 1 2 r 3 v -2 3 r -1 1 r -3 2 r -3 h -2 -1 r s n 1020 1303 m -2 v 2 h 2 v -1 h -3 h -2 h -2 -3 r -2 -3 r -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 3 1 r 1 2 r 2 3 r 3 v s n 1014 1300 m -1 -2 r -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s 16 -15 1019 1316 i 16 -15 1020 1317 i 1 -14 1032 1319 i 10 -1 1033 1311 i 10 -2 1033 1311 i 3 3 1017 1314 i 5 5 1030 1317 i 5 5 1033 1299 i 1045 1312 1041 1307 b n 1035 1332 m -2 v 1 h 2 v c s 10 -10 1040 1327 i 11 -11 1040 1328 i 3 3 1037 1325 i 1053 1320 1048 1314 b 10 -10 1048 1335 i 11 -10 1048 1336 i n 1050 1333 m 3 v 2 3 r 1 2 r 3 1 r 2 h 8 -8 r s n 1053 1341 m 2 h 3 h 8 -8 r s 3 3 1045 1333 i 5 5 1056 1323 i 1069 1336 1064 1331 b n 1066 1353 m -1 -2 r -1 v 1 -3 r 1 -1 r 2 -1 r 2 h 2 1 r 1 1 r 1 3 r 1 v -1 2 r -1 2 r -2 h -2 h -2 h c s n 1065 1351 m 1 -2 r 3 -3 r 2 -1 r s n 1075 1350 m -1 2 r -3 3 r -2 h s n 1071 1355 m 2 v 2 v 1 -1 r -1 -1 r s n 1069 1345 m -1 v 1 -2 r 1 -1 r 2 -1 r 3 2 r 3 3 r 3 2 r 2 h s n 1071 1341 m 1 h 3 1 r 4 4 r 3 2 r 2 -1 r 1 -1 r -2 v -1 -3 r -4 -4 r -3 -2 r -3 1 r 1 v -1 2 r 1 3 r s 23 22 1086 1366 i 45 37 1109 1388 i 45 34 1154 1425 i 21 15 1199 1459 i 11 -19 1229 1494 i 11 -19 1230 1494 i n 1235 1485 m 1 3 r 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r -3 v -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1239 1491 m 3 h 2 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1230 1494 1227 1492 b 11 -19 1248 1504 i 11 -19 1249 1505 i 4 2 1245 1503 i 1262 1488 1256 1484 b n 1265 1506 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 1 2 r -1 4 r -1 1 r -2 2 r -3 1 r -3 h c s n 1265 1506 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s n 1274 1494 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -2 h s n 1290 1516 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -2 r 1 -4 r 1 -1 r 2 -3 r 3 h 3 h 2 1 r 2 3 r 1 2 r s n 1283 1516 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 11 -19 1293 1530 i 10 -19 1294 1530 i -4 -14 1306 1529 i 9 -4 1305 1521 i 9 -5 1304 1521 i 4 2 1290 1528 i 5 3 1304 1527 i 6 4 1301 1509 i 1316 1518 1311 1515 b n 1312 1540 m -1 -1 r 2 -1 r 2 v c s 7 -13 1315 1534 i 7 -13 1316 1535 i 3 3 1313 1532 i 1326 1524 1320 1520 b 7 -13 1325 1540 i 7 -12 1326 1540 i n 1327 1537 m 1 3 r 2 3 r 2 1 r 3 h 2 -1 r 6 -10 r s n 1332 1544 m 2 h 2 -2 r 6 -9 r s 4 2 1322 1538 i 7 3 1329 1526 i 1346 1535 1339 1531 b n 1347 1552 m -1 -2 r -1 v -2 v 1 -2 r 2 -1 r 1 -1 r 2 h 2 1 r 2 2 r 2 v 2 v -1 2 r -2 1 r -2 h -2 h c s n 1346 1550 m -2 v 2 -4 r 2 -1 r s n 1356 1546 m -1 3 r -2 3 r -2 1 r s n 1353 1553 m 1 v 2 2 r -1 v -2 -1 r s n 1349 1544 m -1 -2 r 1 -2 r -1 v 2 -1 r 3 h 5 3 r 3 1 r 1 -1 r s n 1349 1539 m 1 h 4 h 4 3 r 3 h 2 -1 r 1 -1 r -2 v -2 -2 r -6 -4 r -3 h -2 1 r 1 v 2 v 2 3 r s 9 4 1370 1559 i 45 31 1379 1563 i 45 26 1424 1594 i 45 23 1469 1620 i 6 2 1514 1643 i 7 -20 1532 1663 i 7 -21 1533 1664 i n 1536 1654 m 1 3 r 2 1 r 2 1 r 3 h 3 -1 r 2 -3 r 1 -1 r -4 v -2 -2 r -2 -2 r -2 -1 r -2 h -3 1 r s n 1541 1659 m 2 h 3 -1 r 2 -3 r 1 -2 r -3 v -1 -3 r -2 -1 r s 1533 1664 1529 1662 b 7 -20 1552 1671 i 7 -20 1553 1671 i 4 1 1549 1670 i 1563 1652 1557 1650 b n 1569 1670 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r 3 v 2 v -2 2 r -3 2 r -3 -1 r c s n 1569 1670 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 2 h s n 1576 1657 m 1 2 r 1 2 r 4 v 2 v -2 2 r -3 1 r -2 h s n 1595 1676 m -1 -1 r 1 -1 r 1 2 r 1 v -3 1 r -2 h -3 -1 r -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 h 2 2 r 2 3 r s n 1588 1677 m -2 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 2 h s 7 -20 1600 1689 i 7 -21 1601 1690 i -6 -14 1613 1687 i 9 -5 1610 1679 i 9 -6 1609 1679 i 4 2 1597 1688 i 6 3 1610 1685 i 6 3 1605 1668 i 1621 1674 1615 1672 b n 1620 1697 m -1 -1 r 2 -1 r 1 v c s 6 -13 1622 1690 i 6 -14 1623 1691 i 3 2 1620 1689 i 1631 1678 1625 1676 b 5 -13 1633 1694 i 5 -14 1634 1695 i n 1635 1692 m 1 2 r 3 2 r 2 1 r 3 h 2 -1 r 4 -11 r s n 1641 1697 m 2 h 2 -2 r 4 -10 r s 4 2 1630 1693 i 7 2 1635 1680 i 1653 1686 1646 1684 b n 1657 1703 m -2 -1 r -2 v -2 v -2 v 2 -1 r 1 -1 r 3 h 2 1 r 1 1 r 1 2 r 2 v -1 2 r -1 1 r -2 1 r -2 h c s n 1655 1702 m -3 v 2 -3 r 1 -2 r s n 1664 1696 m 3 v -1 3 r -2 2 r s n 1663 1703 m 2 v 2 1 r -1 v -2 h s n 1657 1695 m -1 -2 r -2 v 1 -1 r 1 -1 r 4 h 4 2 r 4 h 1 -1 r s n 1657 1690 m 1 h 3 h 5 1 r 3 1 r 2 -2 r -1 v -2 v -2 -2 r -6 -2 r -3 -1 r -2 2 r 1 v 2 v 2 2 r s 13 4 1681 1707 i 45 21 1694 1711 i 45 18 1739 1732 i 45 17 1784 1750 i 12 3 1829 1767 i 5 -21 1855 1787 i 5 -20 1856 1787 i n 1858 1777 m 2 3 r 1 2 r 2 h 4 h 2 -1 r 2 -3 r 1 -2 r -1 -3 r -1 -3 r -3 -2 r -2 h -2 h -2 2 r s n 1863 1782 m 3 h 2 -2 r 2 -2 r 1 -2 r -1 -4 r -1 -2 r -2 -2 r s 1856 1787 1852 1786 b 6 -21 1875 1793 i 6 -20 1876 1793 i 4 1 1872 1792 i 1885 1773 1878 1771 b n 1892 1790 m -3 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 3 -1 r 2 1 r 3 2 r 1 2 r 4 v 2 v -2 2 r -3 2 r -3 h c s n 1892 1790 m -2 -1 r -1 -3 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s n 1898 1777 m 2 2 r 1 2 r 3 v 2 v -2 3 r -3 1 r -2 1 r s n 1919 1795 m -1 -2 r 1 h 1 1 r 1 v -3 1 r -2 1 r -3 -1 r -3 -2 r -1 -2 r -4 v -2 v 2 -2 r 3 -2 r 3 h 2 1 r 2 2 r 2 2 r s n 1912 1796 m -2 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s 6 -21 1925 1807 i 6 -20 1926 1807 i -7 -12 1938 1803 i 9 -6 1934 1796 i 9 -7 1933 1796 i 4 1 1922 1806 i 6 2 1935 1802 i 7 2 1928 1785 i 1944 1790 1939 1789 b n 1945 1813 m -1 v 1 -1 r 1 1 r c s 4 -14 1947 1806 i 4 -13 1948 1806 i 3 1 1945 1805 i 1955 1793 1948 1791 b 4 -14 1958 1809 i 4 -13 1959 1809 i n 1960 1807 m 2 2 r 2 2 r 2 h 4 h 1 -1 r 3 -11 r s n 1966 1811 m 3 h 1 -2 r 3 -11 r s 4 1 1955 1808 i 7 1 1959 1795 i 1977 1800 1970 1798 b n 1983 1816 m -2 -1 r -2 v -1 -2 r 1 -2 r 1 -1 r 2 -1 r 2 -1 r 2 1 r 2 1 r 2 v 1 2 r -1 2 r -1 2 r -2 h -2 1 r c s n 1981 1815 m -3 v 1 -4 r 2 -1 r s n 1990 1808 m 3 v -1 4 r -2 1 r s n 1989 1816 m 1 v 2 2 r -1 v -2 -1 r s n 1982 1808 m -2 v -1 -2 r 1 -1 r 1 -2 r 3 h 5 2 r 3 -1 r 2 h s n 1982 1803 m 1 -1 r 3 h 5 2 r 3 -1 r 2 -1 r -1 v -3 v -3 -1 r -6 -2 r -3 h -2 2 r 1 v 2 v 3 2 r s 2 2007 1818 e 45 144 614 748 i 8 21 659 892 i 8 -12 634 897 i 8 -12 635 897 i n 636 895 m 1 2 r 2 3 r 2 1 r 3 1 r 2 -1 r 652 892 l s n 641 901 m 2 1 r 2 -2 r 651 891 l s 4 2 631 895 i 6 4 639 883 i 655 893 649 889 b n 656 912 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 3 r 2 v -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 656 912 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 666 901 m 1 2 r 3 v -1 3 r -1 2 r -2 2 r -3 h -2 h s 8 -12 670 921 i 8 -12 670 922 i n 672 919 m 1 3 r 2 2 r 1 2 r 3 h 2 -1 r 7 -9 r s n 676 926 m 3 h 2 -1 r 687 915 l s 3 3 667 919 i 6 4 675 907 i 690 918 684 914 b 12 -18 684 940 i 12 -18 685 940 i n 691 932 m 3 v 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -2 r -1 -1 r -3 -1 r -3 1 r s n 694 938 m 2 h 3 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 685 940 682 938 b 12 -18 702 952 i 12 -17 703 952 i 4 2 699 950 i 718 936 712 932 b n 719 955 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 2 r 3 v -1 3 r -1 2 r -3 2 r -2 1 r -4 -1 r c s n 719 955 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 729 944 m 1 2 r 3 v -1 3 r -1 2 r -2 1 r -3 1 r -3 h s n 743 967 m -1 -1 r 2 -1 r 2 v -1 1 r -2 h -3 h -2 -2 r -2 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 3 v s n 736 966 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 2 h s 13 -18 744 981 i 12 -17 745 981 i -3 -14 758 981 i 10 -3 757 973 i 10 -4 756 973 i 3 2 742 979 i 5 4 755 979 i 6 4 754 961 i 768 971 763 968 b 19 -42 730 956 i 45 68 749 914 i 28 36 794 982 i 11 -9 825 1030 i 11 -9 826 1031 i n 828 1029 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -8 r s n 830 1037 m 2 1 r 3 -1 r 8 -7 r s 3 3 823 1028 i 5 5 834 1019 i 846 1033 841 1027 b n 842 1051 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 1 r 2 2 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r c s n 842 1051 m -2 v -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s n 855 1043 m 2 v 3 v -2 3 r -1 1 r -3 1 r -3 h -3 -1 r s 11 -9 853 1063 i 11 -9 853 1064 i n 856 1062 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -7 r s n 858 1070 m 2 1 r 2 -1 r 9 -7 r s 2 3 851 1061 i 4 5 862 1052 i 874 1066 869 1060 b 16 -14 862 1085 i 17 -14 862 1086 i n 870 1079 m 3 v 1 3 r 1 1 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -3 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 872 1086 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 862 1086 860 1083 b 16 -14 876 1102 i 17 -14 876 1103 i 2 4 874 1099 i 895 1091 890 1086 b n 891 1109 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 2 r 2 1 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -2 -1 r c s n 891 1109 m -2 v -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 2 1 r s n 904 1101 m 2 v 3 v -2 3 r -1 1 r -3 2 r -3 -1 r -2 h s n 911 1127 m -1 v 1 h 2 v -1 h -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 3 r 3 v s n 904 1125 m -3 v -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 2 1 r s 16 -14 909 1141 i 16 -14 910 1142 i 1 -14 922 1145 i 10 923 1137 e 11 922 1136 e 3 3 907 1139 i 4 4 920 1143 i 5 5 923 1125 i 935 1138 931 1133 b 41 39 933 1150 i 45 48 974 1189 i 36 35 1019 1237 i 9 -11 1061 1283 i 10 -11 1061 1284 i n 1063 1282 m 1 2 r 1 3 r 2 2 r 3 1 r 2 -1 r 7 -9 r s n 1067 1289 m 2 h 2 -1 r 7 -9 r s 3 3 1058 1281 i 6 5 1067 1270 i 1082 1282 1076 1277 b n 1081 1300 m -2 -2 r -3 v 1 -3 r 2 -2 r 2 -2 r 3 h 3 1 r 2 2 r 2 2 r 3 v -1 3 r -2 2 r -2 2 r -3 h -3 -1 r c s n 1081 1300 m -1 -2 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 2 h s n 1092 1291 m 1 2 r 3 v -1 3 r -1 1 r -3 2 r -3 h -2 h s 9 -11 1094 1311 i 9 -11 1095 1311 i n 1097 1309 m 3 v 2 3 r 1 1 r 3 1 r 2 -1 r 7 -8 r s n 1100 1316 m 2 1 r 2 -1 r 8 -9 r s 4 2 1091 1309 i 5 4 1101 1298 i 1115 1310 1109 1305 b 13 -17 1107 1331 i 14 -16 1107 1331 i n 1114 1324 m 2 v 1 3 r 2 1 r 3 1 r 3 h 2 -2 r 2 -2 r 1 -3 r -2 v -2 -3 r -2 -2 r -2 h -3 h s n 1117 1330 m 2 h 3 h 3 -2 r 1 -1 r 1 -3 r -3 v -1 -2 r s 1107 1331 1104 1329 b 14 -17 1123 1345 i 14 -16 1124 1345 i 3 2 1121 1343 i 1140 1331 1135 1326 b n 1140 1349 m -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 2 1 r 1 3 r 1 3 r -2 3 r -1 1 r -3 2 r -3 h -3 -1 r c s n 1140 1349 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s n 1151 1339 m 2 v 1 3 r -1 3 r -2 2 r -3 1 r -2 1 r -3 -1 r s n 1162 1364 m -2 v 2 h 1 v -1 1 r -3 h -2 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 1 1 r 2 3 r 3 v s n 1156 1362 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s 14 -16 1163 1377 i 13 -17 1164 1378 i -1 -14 1176 1379 i 10 -2 1176 1371 i 10 -2 1175 1370 i 3 3 1161 1375 i 5 4 1174 1377 i 6 4 1174 1359 i 1188 1370 1183 1366 b 11 8 1188 1382 i 45 30 1199 1390 i 45 28 1244 1420 i 45 23 1289 1448 i 1 1 1334 1471 i 8 -12 1342 1482 i 8 -12 1343 1482 i n 1345 1480 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1349 1486 m 3 h 2 -1 r 6 -10 r s 3 2 1340 1480 i 6 4 1347 1468 i 1363 1478 1357 1474 b n 1365 1496 m -2 -3 r -3 v -3 v 1 -1 r 3 -3 r 3 h 3 h 2 1 r 2 3 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1365 1496 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 -1 r 3 h s n 1375 1484 m 1 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -2 h s 8 -12 1379 1504 i 8 -13 1380 1505 i n 1382 1502 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1386 1508 m 3 1 r 2 -2 r 6 -9 r s 3 2 1377 1503 i 6 4 1384 1490 i 1400 1500 1394 1496 b 11 -18 1395 1522 i 11 -19 1396 1523 i n 1401 1514 m 1 3 r 1 2 r 2 1 r 3 h 3 h 2 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1405 1520 m 2 h 3 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1396 1523 1392 1520 b 11 -18 1414 1533 i 12 -19 1414 1534 i 3 2 1411 1532 i 1428 1517 1422 1513 b n 1430 1535 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1430 1535 m -1 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 h 2 h s n 1440 1524 m 1 2 r 1 3 r -1 3 r -1 1 r -3 3 r -2 h -3 h s n 1455 1546 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 h 4 h 1 1 r 2 3 r 1 3 r s n 1448 1546 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h s 11 -19 1458 1560 i 12 -18 1458 1560 i -4 -14 1471 1559 i 9 -3 1470 1551 i 9 -4 1469 1551 i 3 2 1455 1558 i 6 3 1468 1558 i 6 3 1466 1540 i 1481 1549 1476 1546 b 31 15 1483 1561 i 45 19 1514 1576 i 45 13 1559 1595 i 39 18 1604 1608 i 5 -13 1651 1635 i 5 -14 1652 1636 i n 1653 1633 m 1 3 r 3 2 r 2 h 3 1 r 2 -2 r 4 -10 r s n 1659 1638 m 2 h 2 -1 r 4 -11 r s 4 2 1648 1634 i 6 3 1654 1621 i 1671 1628 1664 1625 b n 1676 1645 m -3 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 3 h 2 1 r 3 2 r 1 3 r -1 3 r 2 v -2 2 r -3 1 r -3 h c s n 1676 1645 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1683 1633 m 2 2 r 1 2 r 4 v -1 1 r -2 3 r -3 1 r -2 h s 5 -14 1691 1652 i 5 -13 1692 1652 i n 1693 1649 m 1 3 r 3 2 r 2 1 r 3 h 2 -2 r 4 -10 r s n 1699 1655 m 2 h 2 -2 r 4 -10 r s 4 1 1688 1651 i 6 3 1694 1637 i 1711 1644 1704 1641 b 8 -20 1709 1667 i 8 -20 1710 1667 i n 1714 1658 m 1 2 r 2 2 r 2 1 r 3 h 3 -1 r 2 -2 r -2 v 1 -4 r -1 -2 r -3 -2 r -2 -1 r -2 h -3 1 r s n 1719 1663 m 2 h 3 -1 r 2 -3 r -2 v 1 -3 r -1 -3 r -2 -1 r s 1710 1667 1706 1666 b 8 -20 1729 1675 i 8 -20 1730 1675 i 4 1 1726 1674 i 1741 1657 1735 1654 b n 1746 1674 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1746 1674 m -1 -2 r -1 -2 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1754 1662 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1772 1681 m -1 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 3 r 1 2 r s n 1765 1682 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1777 1694 i 8 -20 1778 1695 i -6 -13 1790 1692 i 8 -5 1788 1684 i 8 -5 1787 1684 i 4 2 1774 1693 i 6 2 1787 1691 i 7 3 1782 1673 i 1798 1680 1793 1678 b 27 8 1802 1692 i 45 9 1829 1700 i 45 22 1874 1709 i 45 15 1919 1731 i 1 1964 1746 e 44 12 1965 1746 i 45 143 614 750 i 6 16 659 893 i 20 -8 658 924 i 20 -8 658 925 i n 657 921 m 4 12 r 2 2 r 2 1 r 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -3 -8 r s n 661 933 m 2 1 r 1 1 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v s 2 7 677 913 i n 670 926 m 1 2 r 2 h 7 h 2 1 r 1 v 1 v s n 671 928 m 3 h 7 -1 r 1 1 r 1 2 r -1 1 r -1 1 r s n 678 939 m 5 12 r -2 1 r -2 -1 r -2 h -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 -1 r 3 2 r 2 2 r 2 v 1 3 r -1 3 r s n 682 950 m -3 1 r -2 h s n 674 946 m 1 -2 r 1 -3 r 2 -2 r 2 h 3 -1 r 3 2 r 2 1 r s n 682 960 m 1 h -1 v -1 h -1 1 r 3 v 1 3 r 2 2 r 1 1 r 3 h 6 -3 r 3 h 1 1 r s n 685 969 m 9 -4 r 2 h 2 2 r 1 v s n 687 968 m 1 -2 r -1 -6 r -1 -3 r 2 -2 r 2 h 2 h 2 2 r 1 3 r 2 v -1 3 r s n 687 960 m -2 v 1 -2 r 2 h 2 h 2 1 r s 20 -8 685 992 i 20 -8 685 993 i n 694 988 m -2 -1 r -2 -1 r -1 -2 r -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 1 r 3 2 r 2 v 2 v -1 3 r s n 689 984 m -3 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 1 r s 1 4 684 989 i 706 988 705 984 b 16 695 999 e 13 696 1000 e n 700 1010 m 711 999 l 3 -3 r 1 -3 r -2 v -1 v -1 -1 r -1 2 r 1 h s 3 6 694 997 i 701 1012 698 1007 b 20 -8 696 1021 i 20 -8 697 1022 i n 695 1018 m 5 11 r 2 3 r 1 h 2 1 r 2 -1 r 2 -2 r -1 v -4 v -3 -7 r s n 700 1029 m 1 2 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v s 3 6 715 1010 i n 708 1023 m 2 1 r 1 1 r 8 h 1 h 1 1 r -1 2 r s n 710 1024 m 2 h 7 h 2 h 1 2 r -2 2 r -1 h s n 718 1047 m -1 v 2 1 r -1 1 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 1 r 4 v -1 2 r s n 713 1043 m -2 v 1 -3 r 3 -2 r 1 -1 r 4 h 2 1 r 2 2 r s 15 1 718 1055 i 13 718 1056 e 11 -11 722 1067 i 2 6 717 1053 i 723 1069 721 1063 b 14 -5 725 1074 i 13 -5 726 1075 i n 731 1073 m -2 2 r -1 3 r 2 v 1 3 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 4 724 1071 i 740 1073 738 1066 b 12 28 737 1090 i 45 90 749 1118 i 18 31 794 1208 i 18 -12 808 1255 i 18 -12 808 1256 i n 806 1253 m 7 10 r 2 2 r 2 h 2 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 813 1263 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r 1 -1 r -1 -3 r s 4 6 824 1241 i n 820 1255 m 2 1 r 1 h 8 -1 r 1 h 1 1 r 1 v s n 822 1256 m 2 h 7 -3 r 2 1 r 1 1 r -1 2 r -1 1 r s n 831 1266 m 6 10 r -1 1 r -3 1 r -1 -1 r -2 -1 r -2 -2 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 2 1 r 4 v 2 v s n 837 1275 m -3 2 r -2 h s n 828 1274 m -3 v 1 -3 r 2 -2 r 1 -1 r 4 -1 r 2 1 r 2 1 r s n 839 1285 m -1 v -1 1 r 1 v 2 v 2 4 r 2 1 r 2 h 2 h 6 -4 r 2 h 2 h s n 844 1293 m 7 -5 r 3 h 2 1 r 1 v s n 845 1292 m 1 -1 r -3 -6 r -1 -3 r 1 -2 r 2 -1 r 2 -1 r 3 2 r 2 3 r 2 v -1 3 r s n 843 1285 m -2 v 1 -2 r 2 -2 r 2 h 2 1 r s 18 -12 848 1316 i 18 -12 848 1317 i n 856 1311 m -3 -1 r -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 2 r -1 3 r s n 850 1307 m -2 v 1 -3 r 2 -2 r 1 -2 r 3 h 3 h 2 1 r s 2 3 846 1314 i 868 1308 866 1304 b 16 -3 859 1321 i 13 -3 860 1322 i n 866 1331 m 9 -13 r 2 -4 r 1 -3 r -1 -2 r -1 v -2 h 1 v 1 1 r s 4 5 858 1319 i 867 1333 864 1328 b 19 -12 864 1342 i 18 -12 865 1343 i n 863 1340 m 7 10 r 2 2 r 1 h 3 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 870 1350 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -1 v -3 v s 4 6 881 1328 i n 876 1342 m 2 1 r 2 h 8 -1 r 1 h 1 1 r -1 1 r s n 878 1343 m 3 h 7 -3 r 1 1 r 1 1 r -1 2 r -1 1 r s n 891 1364 m -1 v 2 h -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 4 r 2 v s n 885 1361 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 16 -3 892 1372 i 13 -3 893 1373 i 9 -13 899 1382 i 4 5 891 1370 i 900 1384 897 1379 b 12 -8 904 1389 i 12 -8 904 1390 i n 909 1386 m -2 3 r 3 v 2 v 2 3 r 1 h 1 -1 r -1 v -1 h 1 v s 2 4 902 1386 i 918 1385 914 1379 b 11 15 918 1402 i 45 59 929 1417 i 45 46 974 1476 i 7 7 1019 1522 i 14 -17 1027 1545 i 14 -17 1028 1546 i n 1025 1543 m 9 8 r 3 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 v -2 -3 r -7 -5 r s n 1034 1551 m 3 h 1 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 -2 r s 6 5 1038 1526 i n 1038 1541 m 3 1 r 1 -1 r 7 -3 r 2 h 2 v s n 1041 1542 m 2 -1 r 6 -5 r 1 h 2 1 r -1 3 r s n 1052 1548 m 10 8 r -1 2 r -2 1 r -2 h -2 -1 r -3 -2 r -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 1 2 r 2 2 r 3 v s n 1061 1555 m -2 3 r -2 1 r s n 1052 1556 m -2 v -1 -3 r 1 -3 r 2 -1 r 2 -2 r 3 h 3 h s n 1066 1565 m -1 v -1 v -1 1 r 1 v 1 2 r 3 3 r 2 h 2 h 2 -1 r 5 -5 r 2 -1 r 1 h s n 1073 1570 m 6 -7 r 2 -1 r 2 1 r 1 h s n 1074 1569 m -2 v -4 -4 r -2 -3 r 1 -2 r 1 -2 r 2 -1 r 3 1 r 3 2 r 2 v 1 3 r s n 1070 1563 m -1 -2 r -3 v 2 -1 r 2 -1 r 2 h s 13 -17 1084 1591 i 13 -17 1085 1592 i n 1090 1583 m -3 h -2 h -1 -1 r -2 -3 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 3 1 r 1 2 r 1 2 r 1 3 r s n 1084 1582 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 2 h 3 h s 4 3 1081 1589 i 1101 1577 1097 1574 b 14 -7 1096 1592 i 12 -7 1097 1593 i n 1106 1600 m 4 -15 r 1 -5 r -3 v -1 -2 r -1 h -1 h 1 v 1 h s 5 4 1095 1591 i 1108 1601 1103 1597 b 13 -17 1108 1611 i 13 -17 1109 1611 i n 1105 1609 m 10 7 r 3 2 r 1 -1 r 3 -1 r 1 -1 r -2 v -2 v -2 -3 r -6 -5 r s n 1115 1616 m 2 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 -2 r -1 -2 r s 6 4 1119 1592 i n 1119 1606 m 2 1 r 2 h 7 -4 r 1 h 1 1 r 1 v s n 1121 1607 m 2 -1 r 7 -4 r 1 h 2 1 r -1 2 r -1 1 r s n 1140 1623 m -1 v 1 h 1 v 1 v -3 h -3 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 -1 r 3 2 r 2 1 r 2 3 r 3 v s n 1133 1622 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 3 h 2 1 r s 14 -7 1143 1630 i 12 -6 1144 1631 i 4 -15 1153 1638 i 5 4 1142 1629 i 1155 1639 1150 1635 b 10 -11 1159 1643 i 9 -11 1160 1644 i n 1164 1639 m -1 3 r 3 v 1 2 r 3 2 r 1 h 1 -1 r -1 v -2 h 1 v s 3 3 1157 1641 i 1172 1635 1166 1630 b 22 20 1177 1651 i 45 32 1199 1671 i 45 26 1244 1703 i 24 18 1289 1729 i 10 -19 1317 1763 i 10 -20 1318 1764 i n 1314 1762 m 11 5 r 4 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 -2 r -2 -2 r -7 -4 r s n 1325 1767 m 3 h 1 h 2 -2 r 1 -1 r -3 v -1 -1 r -1 -2 r s 7 4 1324 1742 i n 1327 1757 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 2 r s n 1330 1757 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 3 v -1 1 r s n 1342 1761 m 11 6 r -1 1 r -1 2 r -2 h -2 h -3 -1 r -2 -3 r -1 -2 r -4 v 1 -1 r 3 -3 r 2 -1 r 4 1 r 2 1 r 2 2 r 1 3 r s n 1352 1766 m -1 3 r -2 1 r s n 1344 1769 m -1 -2 r -1 -3 r -3 v 1 -2 r 3 -2 r 2 -1 r 3 h s n 1359 1774 m -1 v -1 h 2 v 2 2 r 3 2 r 3 h 1 -1 r 2 -1 r 3 -7 r 2 -1 r 1 h s n 1367 1778 m 4 -8 r 2 -1 r 2 h 1 h s n 1368 1776 m -1 v -5 -4 r -3 -2 r -2 v 1 -2 r 2 -2 r 3 1 r 3 1 r 1 2 r 1 3 r s n 1363 1771 m -2 -2 r -2 v 1 -2 r 2 -1 r 2 h s 10 -19 1382 1796 i 10 -20 1383 1797 i n 1387 1787 m -3 1 r -2 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 1380 1787 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s 4 2 1379 1795 i 1396 1779 1392 1777 b 12 -10 1395 1794 i 11 -9 1395 1795 i n 1406 1800 m 1 -16 r -4 v -1 -3 r -2 -2 r -1 h -1 h 2 v 2 -1 r s 5 3 1393 1793 i 1407 1801 1402 1798 b 9 -19 1410 1810 i 9 -20 1411 1811 i n 1407 1809 m 11 5 r 3 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 v -2 -3 r -8 -4 r s n 1418 1814 m 2 h 2 h 1 -1 r 1 -2 r -3 v -1 v -1 -2 r s 6 3 1417 1790 i n 1420 1804 m 2 h 1 -1 r 6 -5 r 2 h 1 h 2 v s n 1422 1804 m 2 -2 r 5 -5 r 1 h 2 h 3 v 1 v s n 1444 1816 m -1 -1 r 2 -1 r 1 v 1 v -3 1 r -3 h -2 -1 r -3 -2 r -3 v -3 v 1 -2 r 2 -3 r 3 h 3 h 2 1 r 2 2 r 1 3 r s n 1437 1816 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 12 -10 1449 1822 i 10 -8 1450 1822 i 1 -16 1460 1828 i 5 3 1447 1821 i 1461 1828 1456 1826 b 6 -13 1467 1831 i 6 -13 1468 1832 i n 1471 1826 m -1 3 r 1 3 r 1 2 r 3 1 r 2 h -1 v -1 v -2 h 1 2 r s 4 2 1464 1830 i 1477 1820 1471 1817 b 29 10 1485 1835 i 45 6 1514 1845 i 45 17 1559 1851 i 40 17 1604 1868 i 6 -21 1651 1900 i 6 -20 1652 1900 i n 1648 1899 m 11 4 r 4 h 1 -1 r 2 -2 r -2 v -2 v -1 -1 r -2 -2 r -8 -3 r s n 1659 1903 m 3 -1 r 1 h 2 -2 r -2 v -2 v -1 -1 r -1 -2 r s 7 3 1654 1878 i n 1660 1892 m 2 h 1 -1 r 5 -6 r 2 -1 r 1 1 r 1 v s n 1662 1892 m 1 -2 r 5 -6 r 1 -1 r 2 1 r 2 v 1 v s n 1675 1894 m 12 4 r -1 1 r -1 2 r -2 1 r -2 h -3 -1 r -2 -2 r -2 -2 r -4 v 1 -1 r 2 -3 r 2 -1 r 4 h 2 h 2 2 r 2 3 r s n 1686 1897 m -1 3 r -2 2 r s n 1678 1901 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1694 1904 m -1 v -1 h 1 v 1 v 2 2 r 4 1 r 2 -1 r 1 h 2 -2 r 2 -7 r 2 -1 r 1 -1 r s n 1702 1907 m 3 -9 r 2 -2 r 2 h 1 h s n 1703 1905 m -1 -2 r -5 -2 r -3 -2 r -3 v -2 v 2 -1 r 3 h 3 1 r 2 1 r 1 3 r s n 1697 1901 m -2 -2 r -2 v -2 v 2 -2 r 2 h s 7 -20 1720 1922 i 7 -20 1721 1922 i n 1723 1912 m -2 2 r -3 h -2 -1 r -2 -2 r -1 -2 r -1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 h 2 h 1 2 r 2 2 r s n 1716 1913 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s 4 1 1717 1921 i 1731 1903 1727 1902 b 10 -12 1732 1918 i 9 -11 1733 1919 i n 1744 1922 m -2 -16 r -4 v -2 -3 r -1 -1 r -1 -1 r -2 1 r 1 1 r 1 h s 6 2 1730 1918 i 1746 1923 1740 1921 b 6 -20 1750 1931 i 6 -21 1751 1932 i n 1747 1930 m 11 4 r 4 h 1 h 2 -2 r -2 v -2 v -1 -2 r -3 -2 r -7 -2 r s n 1758 1934 m 3 h 1 -1 r 2 -1 r -2 v -3 v -1 -1 r -2 -2 r s 7 2 1753 1910 i n 1759 1924 m 2 -1 r 1 -1 r 5 -5 r 2 -1 r 1 h 1 v s n 1761 1923 m 1 -1 r 5 -7 r 1 h 2 h 2 v 1 v s n 1784 1932 m -1 -2 r 2 h 1 v 1 v -3 1 r -2 1 r -3 -1 r -2 -2 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 4 h 2 1 r 2 1 r 2 3 r s n 1777 1933 m -1 -2 r -2 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 3 h s 10 -12 1790 1937 i 9 -10 1791 1937 i -2 -15 1802 1940 i 6 2 1788 1936 i 1804 1941 1798 1939 b 4 -14 1810 1943 i 4 -13 1811 1943 i n 1812 1937 m 4 v 2 2 r 1 2 r 3 1 r 2 -1 r -1 v -1 -1 r -1 h 1 2 r s 4 1 1807 1942 i 1818 1930 1811 1928 b 1 1828 1944 e 45 11 1829 1944 i 45 11 1874 1955 i 45 15 1919 1966 i 26 5 1964 1981 i 19 4 1990 1986 i end %%EndDocument @endspecial 80 2259 a Fa(Figure)16 b(1:)j(Comm)o(unications)d(rates)e(for)h (\\ping-p)q(ong")h(b)q(et)o(w)o(een)f(t)o(w)o(o)f(no)q(des)i(on)f(an)g(In)o (tel)h(ipsc/860.)963 2790 y(2)p eop %%Page: 3 3 bop 74 2133 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ring.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 713 569 d 36 857 569 d 18 1001 569 d 36 1145 569 d 18 1289 569 d 36 1433 569 d 18 1577 569 d 36 1721 569 d 18 1865 569 d 36 2009 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 814 553 m 2 1 r 4 4 r -27 v s -25 819 556 d 11 814 531 e n 843 558 m -4 -2 r -2 -3 r 835 546 l -4 v 837 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 843 558 m -3 -2 r -1 -1 r -1 -2 r 837 546 l -4 v 838 536 l 1 -3 r 1 -1 r 3 -1 r s n 846 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 869 558 m -4 -2 r -3 -3 r 861 546 l -4 v 862 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 869 558 m -3 -2 r -1 -1 r -1 -2 r 862 546 l -4 v 864 536 l 1 -3 r 1 -1 r 3 -1 r s n 871 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 894 558 m -3 -2 r -3 -3 r 887 546 l -4 v 888 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 894 558 m -2 -2 r -1 -1 r -2 -2 r 888 546 l -4 v 889 536 l 2 -3 r 1 -1 r 2 -1 r s n 897 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1099 553 m 1 -2 r -1 -1 r -2 1 r 2 v 2 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -6 -3 r -3 -1 r -2 -2 r -2 -4 r -4 v s n 1109 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 1097 533 m 2 2 r 2 h 7 -3 r 4 h 2 1 r 1 2 r s n 1101 535 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 1131 558 m -4 -2 r -3 -3 r 1123 546 l -4 v 1124 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1131 558 m -3 -2 r -1 -1 r -1 -2 r 1124 546 l -4 v 1126 536 l 1 -3 r 1 -1 r 3 -1 r s n 1133 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s n 1157 558 m -4 -2 r -3 -3 r 1149 546 l -4 v 1150 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1157 558 m -3 -2 r -1 -1 r -2 -2 r 1150 546 l -4 v 1151 536 l 2 -3 r 1 -1 r 3 -1 r s n 1159 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1182 558 m -4 -2 r -2 -3 r 1175 546 l -4 v 1176 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1182 558 m -2 -2 r -2 -1 r -1 -2 r 1176 546 l -4 v 1177 536 l 1 -3 r 2 -1 r 2 -1 r s n 1185 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1387 554 m 1 -1 r -1 -2 r -2 2 r 1 v 3 2 r 4 2 r 5 h 4 -2 r 1 -2 r -4 v -1 -2 r -4 -2 r -4 h s n 1397 558 m 2 -2 r 2 -2 r -4 v -2 -2 r -2 -2 r s n 1397 546 m 2 -1 r 3 -3 r 1 -2 r -4 v -1 -3 r -1 -1 r -4 -1 r -5 h -4 1 r -1 1 r -2 3 r 1 v 2 2 r 1 -2 r -1 -1 r s n 1401 544 m 1 -4 r -4 v -1 -3 r -2 -1 r -2 -1 r s n 1419 558 m -4 -2 r -3 -3 r 1411 546 l -4 v 1412 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1419 558 m -3 -2 r -1 -1 r -1 -2 r 1412 546 l -4 v 1414 536 l 1 -3 r 1 -1 r 3 -1 r s n 1421 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1444 558 m -3 -2 r -3 -3 r 1437 546 l -4 v 1438 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1444 558 m -2 -2 r -1 -1 r -2 -2 r 1438 546 l -4 v 1439 536 l 2 -3 r 1 -1 r 2 -1 r s n 1447 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1470 558 m -4 -2 r -2 -3 r 1462 546 l -4 v 1464 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1470 558 m -2 -2 r -2 -1 r -1 -2 r 1464 546 l -4 v 1465 536 l 1 -3 r 2 -1 r 2 -1 r s n 1473 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s -24 1685 555 d -27 1686 558 d n 1686 558 m 1672 539 l 20 h s 9 1681 531 e n 1707 558 m -4 -2 r -3 -3 r 1699 546 l -4 v 1700 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1707 558 m -3 -2 r -1 -1 r -2 -2 r 1700 546 l -4 v 1701 536 l 2 -3 r 1 -1 r 3 -1 r s n 1709 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1732 558 m -4 -2 r -2 -3 r 1725 546 l -4 v 1726 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1732 558 m -2 -2 r -2 -1 r -1 -2 r 1726 546 l -4 v 1727 536 l 1 -3 r 2 -1 r 2 -1 r s n 1735 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1758 558 m -4 -2 r -2 -3 r 1750 546 l -4 v 1752 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1758 558 m -3 -2 r -1 -1 r -1 -2 r 1752 546 l -4 v 1753 536 l 1 -3 r 1 -1 r 3 -1 r s n 1761 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s 1961 545 1964 558 b n 1961 545 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 2 r 2 -2 r -2 -1 r s n 1971 549 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 1964 558 e n 1964 556 m 6 h 6 2 r s n 1994 558 m -3 -2 r -3 -3 r 1987 546 l -4 v 1988 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1994 558 m -2 -2 r -1 -1 r -2 -2 r 1988 546 l -4 v 1989 536 l 2 -3 r 1 -1 r 2 -1 r s n 1997 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2020 558 m -4 -2 r -2 -3 r 2012 546 l -4 v 2014 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2020 558 m -2 -2 r -2 -1 r -1 -2 r 2014 546 l -4 v 2015 536 l 1 -3 r 2 -1 r 2 -1 r s n 2023 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 2046 558 m -4 -2 r -3 -3 r 2038 546 l -4 v 2039 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 2046 558 m -3 -2 r -1 -1 r -1 -2 r 2039 546 l -4 v 2041 536 l 1 -3 r 1 -1 r 3 -1 r s n 2048 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s 1440 569 569 d 36 569 569 e 18 569 641 e 18 569 713 e 18 569 785 e 18 569 857 e 36 569 929 e 18 569 1001 e 18 569 1073 e 18 569 1145 e 18 569 1217 e 36 569 1289 e 18 569 1361 e 18 569 1433 e 18 569 1505 e 18 569 1577 e 36 569 1649 e 18 569 1721 e 18 569 1793 e 18 569 1865 e 18 569 1937 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 392 932 395 945 b n 392 932 m 3 2 r 3 2 r 4 h 4 -2 r 3 -2 r 1 -4 r -3 v -1 -4 r -3 -2 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 1 r 2 -1 r -2 -1 r s n 402 936 m 3 -2 r 2 -2 r 2 -4 r -3 v -2 -4 r -2 -2 r -3 -1 r s 12 395 945 e n 395 943 m 6 h 6 2 r s n 425 945 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 945 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 918 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 945 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 945 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 918 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 945 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 918 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 945 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 370 1299 m 3 2 r 4 3 r -27 v s -26 375 1303 d 12 370 1277 e n 400 1304 m 396 1303 l -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 400 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 402 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 425 1304 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1304 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1277 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1304 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1277 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1304 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1304 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 1659 m 3 1 r 4 4 r -27 v s -26 375 1663 d 12 370 1637 e 392 1651 395 1664 b n 392 1651 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 2 r -1 2 r 2 v 1 1 r 2 -1 r -2 -2 r s n 402 1655 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 395 1664 e n 395 1663 m 6 h 6 1 r s n 425 1664 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1664 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1637 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1664 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1664 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1637 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1664 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1637 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1664 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 368 2019 m 1 -1 r -1 -2 r -2 2 r 1 v 2 2 r 1 2 r 4 1 r 5 h 4 -1 r 1 -2 r 1 -2 r -3 v -1 -2 r 379 2011 l 373 2009 l -3 -2 r -2 -2 r -2 -4 r -4 v s n 378 2024 m 2 -1 r 2 -2 r 1 -2 r -3 v -1 -2 r 378 2011 l 373 2009 l s n 366 2000 m 2 1 r 2 h 7 -3 r 3 h 3 2 r 1 1 r s n 370 2001 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 713 2009 d -36 857 2009 d -18 1001 2009 d -36 1145 2009 d -18 1289 2009 d -36 1433 2009 d -18 1577 2009 d -36 1721 2009 d -18 1865 2009 d -36 2009 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 641 e -18 2009 713 e -18 2009 785 e -18 2009 857 e -36 2009 929 e -18 2009 1001 e -18 2009 1073 e -18 2009 1145 e -18 2009 1217 e -36 2009 1289 e -18 2009 1361 e -18 2009 1433 e -18 2009 1505 e -18 2009 1577 e -36 2009 1649 e -18 2009 1721 e -18 2009 1793 e -18 2009 1865 e -18 2009 1937 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s -36 645 2065 d -36 646 2065 d n 640 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -3 v -2 -4 r -2 -1 r -5 -2 r -14 h s n 660 2065 m 4 -2 r 1 -1 r 2 -4 r -3 v -2 -4 r -1 -1 r -4 -2 r s 12 640 2029 e n 655 2048 m 3 -2 r 2 -1 r 5 -12 r 2 -2 r 2 h 1 2 r s n 658 2046 m 2 -3 r 4 -12 r 1 -2 r 4 h 1 4 r 1 v s n 682 2065 m -1 -2 r 1 -1 r 2 1 r c s -24 682 2053 d -24 684 2053 d 7 677 2053 e 12 677 2029 e -24 701 2053 d -24 703 2053 d n 703 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 715 2053 m 3 -2 r 2 -3 r -19 v s 7 696 2053 e 12 696 2029 e 12 715 2029 e n 744 2053 m -3 -2 r -2 -1 r -2 -4 r -3 v 2 -4 r 2 -1 r 3 -2 r 4 h 3 2 r 2 1 r 1 4 r 3 v -1 4 r -2 1 r -3 2 r c s n 741 2051 m -2 -3 r -7 v 2 -3 r s n 751 2038 m 2 3 r 7 v -2 3 r s n 753 2050 m 1 1 r 4 2 r -2 v -4 h s n 739 2039 m -2 -1 r -1 -4 r -1 v 1 -4 r 5 -2 r 9 h 5 -1 r 2 -2 r s n 736 2033 m 1 -2 r 5 -2 r 9 h 5 -2 r 2 -3 r -2 v -2 -3 r -5 -2 r -10 h -5 2 r -2 3 r 2 v 2 3 r 5 2 r s n 819 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -5 2 r -3 h -6 -2 r -3 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 3 -3 r 6 -2 r 3 h 5 2 r 3 3 r 2 4 r s n 808 2065 m -4 -2 r -3 -3 r -2 -3 r -2 -6 r -8 v 2 -5 r 2 -4 r 3 -3 r 4 -2 r s n 842 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 842 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 845 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 869 2053 d -24 871 2053 d n 871 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 883 2053 m 3 -2 r 2 -3 r -19 v s n 890 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 902 2053 m 3 -2 r 2 -3 r -19 v s 7 864 2053 e 12 864 2029 e 12 883 2029 e 12 902 2029 e -24 926 2053 d -24 927 2053 d n 927 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 939 2053 m 4 -2 r 2 -3 r -19 v s n 946 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 958 2053 m 4 -2 r 1 -3 r -19 v s 6 921 2053 e 12 921 2029 e 12 939 2029 e 12 958 2029 e n 982 2053 m -19 v 2 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 984 2053 m -19 v 2 -3 r 3 -2 r s -24 1001 2053 d -24 1003 2053 d 7 977 2053 e 7 996 2053 e 7 1001 2029 e -24 1020 2053 d -24 1022 2053 d n 1022 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1034 2053 m 3 -2 r 2 -3 r -19 v s 7 1015 2053 e 12 1015 2029 e 12 1034 2029 e n 1058 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1058 2053 d -24 1059 2053 d 6 1053 2053 e 12 1053 2029 e n 1094 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1083 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1109 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1124 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1124 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1112 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1143 2065 m -29 v 2 -5 r 3 -2 r 4 h 3 2 r 2 3 r s n 1145 2065 m -29 v 2 -5 r 1 -2 r s 14 1138 2053 e n 1169 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1169 2053 d -24 1171 2053 d 7 1164 2053 e 12 1164 2029 e n 1195 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 1195 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1198 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1222 2053 d -24 1224 2053 d n 1224 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1236 2053 m 3 -2 r 2 -3 r -19 v s 7 1217 2053 e 12 1217 2029 e 12 1236 2029 e n 1274 2050 m 1 3 r -7 v -1 4 r -2 1 r -4 2 r -6 h -4 -2 r -2 -1 r -4 v 2 -1 r 4 -2 r 8 -4 r 4 -1 r 1 -2 r s n 1256 2048 m 2 -2 r 4 -1 r 8 -4 r 4 -2 r 1 -1 r -5 v -1 -2 r -4 -2 r -7 h -3 2 r -2 2 r -2 3 r -7 v 2 4 r s -36 1316 2065 d -36 1318 2065 d n 1311 2065 m 21 h 5 -2 r 2 -1 r 1 -4 r -5 v -1 -3 r -2 -2 r -5 -2 r -14 h s n 1332 2065 m 3 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -3 -2 r s 12 1311 2029 e n 1352 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1371 2043 m 5 v -2 3 r s n 1361 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s -24 1387 2053 d -24 1388 2053 d n 1388 2043 m 2 5 r 3 3 r 4 2 r 5 h 2 -2 r -1 v -2 -2 r -2 2 r 2 1 r s 7 1381 2053 e 12 1381 2029 e n 1424 2063 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -3 h -4 -2 r -1 -3 r -31 v s n 1421 2065 m -2 -2 r -2 -3 r -31 v s 13 1411 2053 e 12 1411 2029 e n 1445 2053 m -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 2 5 r 4 v -2 5 r -4 3 r -5 2 r c s n 1445 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1448 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1472 2053 d -24 1474 2053 d n 1474 2043 m 2 5 r 3 3 r 4 2 r 5 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1467 2053 e 12 1467 2029 e -24 1501 2053 d -24 1503 2053 d n 1503 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1515 2053 m 4 -2 r 1 -3 r -19 v s n 1522 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1534 2053 m 3 -2 r 2 -3 r -19 v s 7 1496 2053 e 12 1496 2029 e 12 1515 2029 e 12 1534 2029 e n 1558 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1573 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1573 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1561 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s -24 1592 2053 d -24 1594 2053 d n 1594 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1606 2053 m 3 -2 r 2 -3 r -19 v s 7 1587 2053 e 12 1587 2029 e 12 1606 2029 e n 1647 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1637 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1661 2043 m 20 h 3 v -1 4 r -2 1 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1680 2043 m 5 v -2 3 r s n 1669 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1733 2072 m -4 -3 r -3 -6 r -4 -6 r -1 -9 r -7 v 1 -8 r 4 -7 r 3 -5 r 4 -4 r s n 1729 2069 m -3 -7 r -2 -5 r -2 -9 r -7 v 2 -8 r 2 -6 r 3 -6 r s -24 1746 2053 d -24 1748 2053 d n 1748 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1760 2053 m 4 -2 r 1 -3 r -19 v s 7 1741 2053 e 12 1741 2029 e 12 1760 2029 e -36 1784 2053 d -36 1786 2053 d n 1786 2048 m 3 3 r 4 2 r 3 h 5 -2 r 4 -3 r 1 -5 r -4 v -1 -5 r -4 -3 r -5 -2 r -3 h -4 2 r -3 3 r s n 1796 2053 m 4 -2 r 3 -3 r 2 -5 r -4 v -2 -5 r -3 -3 r -4 -2 r s 7 1779 2053 e 12 1779 2017 e 31 1846 2050 e 31 1846 2039 e -33 1904 2062 d -36 1906 2065 d n 1906 2065 m 1887 2039 l 27 h s 12 1899 2029 e n 1923 2072 m 3 -3 r 4 -6 r 3 -6 r 2 -9 r -7 v -2 -8 r -3 -7 r -4 -5 r -3 -4 r s n 1926 2069 m 4 -7 r 1 -5 r 2 -9 r -7 v -2 -8 r -1 -6 r -4 -6 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 579 675 569 569 b 586 757 579 675 b 22 -2 575 770 i 22 -2 575 771 i n 585 770 m -1 3 r -1 2 r 2 v 1 3 r 3 2 r 3 h 2 h 3 -1 r 2 -2 r -3 v -3 v -1 -1 r -2 -2 r s n 583 777 m 1 2 r 2 2 r 4 h 2 h 3 -1 r 1 -2 r 1 -2 r s 4 575 767 d 22 -2 577 792 i 22 -2 577 793 i 4 577 789 d 1 7 598 787 i n 586 807 m 1 -4 r 1 -2 r 3 -1 r 2 h 4 h 2 2 r 1 3 r 2 v 3 v -2 3 r -3 1 r -2 h -3 -1 r -3 -1 r -1 -3 r c s n 586 807 m 1 -3 r 1 -2 r 3 -1 r 3 h 3 h 2 2 r 1 2 r s n 600 807 m 2 v -2 3 r -3 1 r -2 h -4 -1 r -2 -1 r -1 -2 r s n 591 833 m 1 -1 r 1 1 r 1 v -1 h -3 -2 r -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r s n 588 827 m 1 -2 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 2 r s 22 -2 582 843 i 22 -2 582 844 i 10 -12 590 854 i 9 5 596 848 i 9 5 596 847 i 4 582 840 d 1 6 590 851 i 1 7 603 838 i 1 6 604 849 i 4 597 865 d 9 -125 597 869 i 9 35 606 744 i 7 23 615 779 i 21 -7 613 817 i 20 -7 614 818 i n 623 814 m -1 3 r 2 v 2 v 2 3 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -1 -2 r -3 -1 r s n 622 821 m 2 2 r 3 1 r 3 h 2 h 2 -2 r 2 -3 r -2 v s 2 4 612 814 i 21 -7 620 837 i 21 -7 620 838 i 1 4 619 834 i 2 6 640 828 i n 632 850 m -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 2 r 2 2 r 1 2 r 3 v -1 3 r -3 2 r -2 1 r -3 h -3 -2 r -2 -2 r c s n 632 850 m -3 v 1 -2 r 3 -2 r 2 -1 r 3 h 3 2 r 1 1 r s n 646 847 m 2 v -2 3 r -2 2 r -2 1 r -3 h -3 -2 r -2 -1 r s n 643 874 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -1 2 r s n 638 869 m -2 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s 21 -7 636 886 i 21 -7 636 887 i 7 -13 646 895 i 9 3 651 888 i 10 3 650 887 i 1 4 635 883 i 2 6 645 892 i 2 7 656 876 i 2 6 659 887 i 6 20 655 903 i 10 25 661 923 i 9 16 671 948 i 9 31 680 964 i 9 17 689 995 i 10 18 698 1012 i 9 13 708 1030 i 9 1 717 1043 i 9 25 726 1044 i 11 -18 737 1084 i 12 -19 737 1085 i n 743 1076 m 3 v 2 2 r 1 1 r 4 h 2 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -2 -1 r -2 h -3 1 r s n 746 1082 m 3 h 3 -1 r 2 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 3 2 734 1083 i 11 -18 755 1095 i 11 -19 756 1096 i 4 2 752 1094 i 6 4 764 1075 i n 772 1097 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 772 1097 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 h 2 h s n 781 1086 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 h -2 h s n 796 1108 m -1 v 2 -1 r 2 v -1 1 r -2 h -3 h -2 -1 r -3 -3 r -3 v -3 v 2 -2 r 2 -2 r 3 h 3 h 2 1 r 2 3 r 1 3 r s n 790 1108 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 2 h s 11 -19 799 1122 i 11 -18 800 1122 i -3 -14 812 1121 i 10 -4 811 1114 i 10 -4 810 1113 i 4 2 796 1120 i 5 3 810 1120 i 6 3 808 1102 i 6 3 817 1108 i 4 3 823 1122 i 9 4 827 1125 i 10 10 836 1129 i 9 -1 846 1139 i 9 1 855 1138 i 37 12 864 1139 i 37 21 901 1151 i 37 6 938 1172 i 20 -11 975 1178 i -7 -21 1007 1177 i -7 -20 1008 1176 i n 1005 1167 m 2 1 r 2 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -2 -3 r -2 -1 r -4 h -2 1 r -1 1 r -2 3 r s n 1011 1168 m 2 -2 r 1 -2 r 1 -4 r -1 -2 r -2 -2 r -2 -2 r -3 h s 4 -2 1004 1178 i -6 -20 1027 1170 i -6 -21 1028 1170 i 4 -1 1024 1171 i 7 -2 1018 1151 i n 1040 1159 m -3 h -3 -1 r -2 -3 r -2 v -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -3 2 r c s n 1040 1159 m -2 h -3 -2 r -2 -2 r -2 v -4 v 1 -2 r 2 -2 r s n 1038 1145 m 2 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -2 2 r s n 1065 1148 m -2 h 1 -2 r 1 1 r 1 1 r -2 3 r -1 1 r -3 1 r -4 h -2 -1 r -2 -3 r -1 -2 r -3 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r s n 1060 1153 m -3 h -2 -2 r -2 -2 r -1 -2 r -3 v 2 -3 r 1 -2 r s -7 -21 1077 1156 i -6 -20 1077 1155 i -13 -6 1085 1145 i 3 -10 1079 1141 i 3 -9 1078 1141 i 3 -1 1074 1156 i 6 -1 1082 1146 i 7 -2 1067 1136 i 6 -2 1078 1133 i 30 30 1092 1138 i 37 -13 1122 1168 i 33 103 1159 1155 i 9 -20 1194 1272 i 9 -19 1195 1272 i n 1199 1263 m 2 3 r 1 1 r 2 1 r 3 h 3 -1 r 2 -2 r 1 -2 r -3 v -1 -3 r -2 -2 r -2 -1 r -3 h -2 1 r s n 1204 1268 m 2 h 3 -1 r 2 -2 r 1 -2 r -4 v -1 -2 r -1 -2 r s 1195 1272 1192 1271 b 9 -19 1214 1280 i 9 -20 1215 1281 i 4 2 1211 1279 i 1227 1262 1220 1259 b n 1231 1280 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1231 1280 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s n 1239 1268 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1257 1288 m -2 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r s n 1250 1288 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 3 -1 r 2 h s 9 -20 1261 1301 i 9 -20 1262 1301 i -6 -14 1275 1299 i 9 -5 1272 1291 i 9 -5 1271 1290 i 3 1 1259 1300 i 6 3 1272 1297 i 7 3 1267 1280 i 1283 1287 1278 1284 b 22 14 1284 1297 i 37 -19 1306 1311 i 37 -16 1343 1292 i 37 6 1380 1276 i 37 9 1417 1282 i 5 1454 1291 e -2 -21 1470 1302 i -2 -22 1471 1302 i n 1470 1292 m 2 1 r 2 1 r 2 h 3 -1 r 2 -3 r 1 -3 r -2 v -2 -3 r -2 -2 r -3 h -2 h -2 1 r -2 2 r s n 1476 1294 m 2 -1 r 2 -2 r 1 -4 r -2 v -2 -3 r -2 -1 r -2 -1 r s 4 1467 1302 e -2 -22 1491 1300 i -2 -22 1492 1300 i 4 1488 1300 e 1493 1278 1486 1279 b n 1506 1291 m -3 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 3 r 1 2 r -1 3 r -2 2 r -3 2 r c s n 1506 1291 m -2 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 1 -1 r s n 1507 1276 m 2 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r -2 2 r s n 1532 1285 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 1 r -3 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 3 -2 r 2 h 3 1 r 2 2 r s n 1526 1289 m -2 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 2 -2 r s -3 -21 1543 1294 i -3 -21 1544 1294 i -11 -9 1553 1286 i 6 -9 1547 1280 i 5 -8 1546 1280 i 5 -1 1539 1295 i 6 1550 1286 e 7 -1 1537 1273 i 1555 1271 1548 1272 b 2 1562 1280 e 37 5 1564 1280 i 37 -21 1601 1285 i 37 -36 1638 1264 i 36 -36 1675 1228 i 3 3 1711 1192 i 34 44 1714 1195 i 10 89 569 569 i 8 83 579 658 i 1 -14 520 742 i 1 -14 521 742 i n 522 739 m 2 3 r 3 1 r 2 h 3 -1 r 1 -2 r 534 729 l s n 529 743 m 2 -1 r 1 -2 r 533 729 l s 4 517 742 e 7 518 728 e 537 729 529 728 b n 547 744 m -3 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 3 h 2 h 3 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 547 744 m -2 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 2 h s n 550 730 m 2 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -2 1 r s 1 -14 563 745 i -14 565 745 d n 565 742 m 2 2 r 3 1 r 2 h 3 h 1 -2 r 577 731 l s n 572 745 m 2 -1 r 1 -2 r 576 731 l s 5 560 745 e 7 1 561 730 i 7 573 731 e 1 -21 586 753 i 1 -21 587 753 i n 587 743 m 2 2 r 2 2 r 2 h 3 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -3 -1 r -2 h -2 1 r -2 2 r s n 593 747 m 2 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -2 -1 r s 4 583 753 e 1 -22 607 755 i 1 -22 608 755 i 4 1 604 754 i 8 605 733 e n 623 748 m -3 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 3 -1 r 2 h 3 1 r 2 3 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 623 748 m -2 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 2 -1 r s n 626 734 m 2 1 r 2 2 r 1 4 r 2 v -2 3 r -2 2 r -2 1 r s n 650 747 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 h -3 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 2 r 2 2 r s n 644 750 m -2 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r s 1 -22 659 758 i 1 -22 660 758 i -9 -11 670 751 i 7 -8 665 745 i 7 -8 664 745 i 5 1 655 757 i 6 667 751 e 7 657 736 e 6 668 737 e 2 -25 604 742 i 9 35 606 717 i 10 34 615 752 i 9 44 625 786 i 5 30 634 830 i 14 -4 637 872 i 14 -4 637 873 i n 640 872 m -2 2 r 4 v 1 2 r 1 2 r 2 1 r 11 -3 r s n 639 880 m 1 1 r 2 1 r 11 -3 r s 1 4 636 869 i 2 7 650 865 i 654 883 652 876 b n 643 898 m -4 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 3 r 2 v 3 v -1 3 r -3 2 r -2 h -3 h -3 -2 r -2 -2 r c s n 643 898 m -3 v 2 -2 r 3 -2 r 2 h 3 h 2 1 r 2 2 r s n 657 896 m 2 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -1 r s 14 -4 647 914 i 14 -4 647 915 i n 650 914 m -2 2 r 4 v 1 2 r 1 2 r 3 1 r 11 -3 r s n 649 922 m 1 1 r 2 1 r 11 -3 r s 1 4 646 911 i 2 7 660 907 i 664 925 663 918 b 21 -5 645 937 i 21 -5 645 938 i n 655 936 m -1 2 r -1 3 r 1 2 r 1 2 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -1 -2 r -3 -1 r s n 654 943 m 1 1 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -2 r s 645 938 644 934 b 21 -5 650 958 i 21 -5 650 959 i 1 4 649 955 i 672 957 670 950 b n 661 972 m -4 v 1 -2 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -2 r c s n 661 972 m -3 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s n 675 970 m 2 v -2 3 r -3 2 r -2 h -3 h -2 -2 r -2 -1 r s n 670 997 m 1 -1 r 1 h -1 2 r -1 h -2 -2 r -2 -1 r -1 -3 r 1 -4 r 1 -2 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -2 3 r s n 665 992 m 1 -3 r 2 -2 r 2 -2 r 2 h 4 h 2 1 r 1 2 r s 21 -5 662 1008 i 21 -5 662 1009 i 7 -13 672 1018 i 9 4 677 1011 i 10 4 676 1010 i 1 4 661 1005 i 1 6 671 1015 i 2 7 682 1000 i 686 1017 685 1011 b 1 4 679 1027 i 9 24 680 1031 i 9 689 1055 e 10 6 698 1055 i 9 15 708 1061 i 9 11 717 1076 i 9 9 726 1087 i 9 7 735 1096 i 10 -24 744 1103 i 9 -18 754 1079 i 9 2 763 1061 i 9 2 772 1063 i 1 781 1065 d 13 -5 758 1023 i 13 -5 758 1024 i n 761 1023 m -1 3 r 3 v 1 2 r 2 2 r 2 h 11 -4 r s n 761 1031 m 2 1 r 2 h 10 -4 r s 2 4 756 1020 i 3 6 770 1015 i 777 1032 774 1025 b n 768 1048 m -3 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r c s n 768 1048 m -2 v 1 -3 r 3 -2 r 2 -1 r 3 h 3 1 r 1 1 r s n 782 1044 m 2 v -1 3 r -3 2 r -1 1 r -4 h -2 -1 r -2 -1 r s 13 -6 775 1063 i 13 -6 775 1064 i n 778 1063 m -1 2 r 4 v 1 2 r 2 2 r 2 h 11 -4 r s n 778 1071 m 1 1 r 3 h 10 -4 r s 2 4 773 1060 i 2 6 787 1055 i 794 1072 791 1065 b 20 -9 777 1087 i 20 -9 777 1088 i n 787 1084 m -1 2 r -1 3 r 1 1 r 2 3 r 3 1 r 3 h 2 -1 r 3 -2 r 1 -3 r -1 -3 r -2 v -2 -2 r -3 -1 r s n 786 1090 m 2 2 r 3 1 r 3 h 2 -1 r 2 -2 r 1 -3 r -2 v s 777 1088 776 1084 b 20 -8 785 1106 i 19 -8 786 1107 i 2 3 784 1104 i 807 1102 804 1095 b n 798 1118 m -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -4 h -2 -1 r -2 -2 r c s n 798 1118 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s n 812 1114 m 2 v -1 3 r -3 2 r -2 1 r -3 h -3 -1 r -1 -1 r s n 811 1141 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r s n 806 1137 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s 20 -9 805 1154 i 19 -9 806 1155 i 6 -13 816 1161 i 10 3 820 1154 i 9 3 820 1153 i 2 4 804 1151 i 2 5 815 1159 i 3 6 824 1143 i 831 1159 828 1153 b 7 70 802 1115 i 9 -5 809 1185 i 9 21 818 1180 i 9 7 827 1201 i 10 -19 836 1208 i 9 20 846 1189 i 9 5 855 1209 i 2 -4 864 1214 i -7 -12 865 1216 i -7 -12 866 1216 i n 864 1213 m 3 1 r 3 -1 r 2 -1 r 2 -2 r -3 v -6 -9 r s n 872 1212 m 1 -2 r -2 v -6 -10 r s 3 -2 863 1218 i 6 -4 855 1206 i 871 1196 865 1200 b n 888 1202 m -3 1 r -3 -1 r -3 -2 r -1 -1 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 3 2 r 1 2 r 3 v 3 v -2 2 r c s n 888 1202 m -2 1 r -3 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 882 1189 m 2 h 3 h 3 3 r 1 1 r 1 3 r -1 3 r -1 2 r s -8 -13 902 1194 i -8 -12 903 1193 i n 901 1191 m 3 h 3 -1 r 2 -1 r 2 -2 r -2 v -6 -10 r s n 909 1189 m 1 -2 r -2 v -6 -9 r s 4 -2 899 1195 i 6 -4 892 1183 i 907 1173 901 1177 b -11 -18 925 1188 i -12 -18 926 1187 i n 920 1179 m 3 h 3 h 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -2 r -3 -1 r -3 1 r -2 1 r -1 2 r -1 3 r s n 927 1178 m 2 -2 r -3 v -1 -3 r -1 -2 r -2 -2 r -3 h -2 h s 926 1187 922 1190 b -11 -19 943 1177 i -11 -18 944 1176 i 3 -2 941 1178 i 935 1156 929 1160 b n 953 1162 m -4 1 r -2 -1 r -3 -2 r -1 -1 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 3 r -1 3 r -2 2 r c s n 953 1162 m -3 1 r -2 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 947 1149 m 2 h 3 1 r 2 2 r 2 1 r 3 v 3 v -2 2 r s n 974 1146 m -2 -1 r 1 -1 r 1 h 1 1 r -1 3 r -1 2 r -3 2 r -3 h -3 h -2 -2 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 1 -1 r 4 -1 r 2 1 r s n 970 1152 m -2 h -3 -1 r -2 -2 r -2 -2 r -3 v -3 v 2 -2 r s -11 -19 987 1150 i -11 -18 988 1149 i -14 -4 993 1138 i 1 -10 985 1135 i 1 -11 984 1136 i 4 -2 984 1151 i 6 -3 990 1139 i 6 -4 973 1133 i 988 1124 983 1127 b 24 6 987 1135 i 37 14 1011 1141 i 37 5 1048 1155 i 37 55 1085 1160 i 5 1 1122 1215 i 11 -9 1117 1212 i 11 -9 1118 1213 i n 1120 1211 m 3 v 1 3 r 1 1 r 3 2 r 2 h 9 -7 r s n 1122 1218 m 2 1 r 3 h 8 -7 r s 3 3 1115 1210 i 5 5 1126 1201 i 1138 1215 1133 1209 b n 1134 1233 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -2 3 r -1 1 r -3 1 r -3 h -3 -2 r c s n 1134 1233 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r s n 1146 1225 m 1 3 r 2 v -2 3 r -2 2 r -3 1 r -3 -1 r -2 -1 r s 11 -9 1144 1246 i 11 -9 1145 1246 i n 1147 1244 m 3 v 1 3 r 1 2 r 3 2 r 2 -1 r 9 -7 r s n 1149 1252 m 2 1 r 3 -1 r 9 -7 r s 3 3 1142 1243 i 5 6 1153 1234 i 1165 1249 1161 1243 b 17 -14 1153 1268 i 17 -13 1153 1268 i n 1161 1262 m 3 v 1 2 r 1 2 r 3 2 r 3 h 3 -1 r 1 -2 r 2 -2 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 1163 1269 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 1153 1268 1151 1265 b 17 -13 1166 1284 i 17 -13 1167 1285 i 3 3 1164 1282 i 1186 1274 1181 1269 b n 1182 1292 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r 2 v -2 3 r -1 1 r -3 2 r -3 -1 r -3 -1 r c s n 1182 1292 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s n 1194 1285 m 1 2 r -1 3 r -1 2 r -2 2 r -3 1 r -3 h -2 -1 r s n 1201 1311 m -2 v 2 1 r -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -2 -3 r 1 -3 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 1 4 r 2 v s n 1195 1308 m -1 -2 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s 17 -13 1199 1324 i 17 -13 1199 1325 i 2 -15 1211 1329 i 10 -1 1213 1321 i 11 1212 1320 e 2 3 1197 1322 i 4 5 1209 1326 i 4 6 1214 1308 i 1225 1322 1221 1317 b 23 -25 1210 1319 i 36 39 1233 1294 i 37 4 1269 1333 i 37 -12 1306 1337 i 9 2 1343 1325 i 5 -13 1353 1333 i 5 -14 1354 1334 i n 1355 1331 m 1 3 r 2 2 r 2 h 4 1 r 1 -2 r 4 -10 r s n 1360 1336 m 3 h 1 -1 r 5 -11 r s 4 2 1350 1332 i 7 2 1355 1319 i 1372 1326 1366 1323 b n 1378 1343 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -2 1 r -4 h c s n 1378 1343 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1385 1331 m 1 1 r 1 3 r 3 v 2 v -3 3 r -2 1 r -3 h s 5 -13 1393 1349 i 5 -14 1394 1350 i n 1395 1347 m 1 2 r 3 3 r 1 h 4 1 r 1 -2 r 5 -11 r s n 1400 1352 m 3 h 1 -1 r 5 -11 r s 4 2 1390 1348 i 7 2 1395 1335 i 1412 1342 1406 1339 b 8 -20 1411 1364 i 8 -20 1412 1365 i n 1416 1355 m 1 3 r 2 2 r 2 h 3 1 r 2 -2 r 3 -2 r -2 v -3 v -1 -3 r -2 -2 r -2 -1 r -2 h -3 2 r s n 1421 1360 m 2 h 2 -1 r 3 -3 r -1 v 1 -4 r -2 -2 r -1 -2 r s 1412 1365 1408 1363 b 8 -20 1431 1372 i 8 -20 1432 1373 i 4 2 1428 1371 i 1443 1354 1436 1351 b n 1448 1371 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 4 h 1 1 r 3 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1448 1371 m -1 -1 r -1 -3 r -3 v -2 v 3 -3 r 2 -1 r 3 h s n 1455 1359 m 2 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1474 1378 m -1 v 1 -1 r 1 2 r -1 1 r -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 2 r 1 3 r s n 1467 1379 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1479 1391 i 8 -20 1480 1392 i -6 -14 1492 1389 i 8 -5 1490 1381 i 8 -6 1489 1381 i 4 2 1476 1390 i 6 2 1489 1388 i 7 3 1484 1370 i 1500 1377 1495 1374 b 30 16 1497 1385 i 37 -10 1527 1401 i 37 16 1564 1391 i 37 -3 1601 1407 i 23 4 1638 1404 i 87 5 1661 1408 i 10 54 569 569 i 9 50 579 623 i 9 49 588 673 i 4 23 597 722 i 21 -4 591 756 i 21 -4 591 757 i n 591 753 m 2 13 r 2 2 r 1 1 r 2 1 r 2 -1 r 2 -1 r 1 -1 r -3 v 601 755 l s n 593 766 m 1 1 r 2 1 r 2 1 r 2 -1 r 2 -1 r -1 v 1 -2 r s 1 7 612 749 i n 602 761 m 2 1 r 1 1 r 8 2 r 1 1 r 1 v -1 1 r s n 604 762 m 2 1 r 7 1 r 2 h 2 v -2 2 r -1 h s 22 -4 597 789 i 21 -4 598 790 i n 608 787 m -3 -2 r -1 -2 r -1 -2 r 1 -3 r 1 -2 r 3 -2 r 2 h 3 h 3 2 r 1 2 r 1 3 r -1 2 r -1 2 r s n 603 781 m 1 -2 r 2 -2 r 2 -2 r 2 h 4 h 2 2 r 1 1 r s 1 4 597 786 i 4 619 785 d 15 4 607 797 i 13 3 607 798 i n 609 810 m 622 801 l 3 -3 r 2 -3 r 1 -2 r -1 v -2 -1 r 2 v 1 h s 1 6 606 795 i 609 812 608 805 b 22 -4 603 819 i 21 -4 604 820 i n 603 816 m 2 12 r 2 3 r 1 1 r 2 h 2 h 2 -1 r 1 -2 r -3 v -1 -8 r s n 605 828 m 2 2 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 1 7 624 812 i n 615 823 m 1 2 r 1 1 r 8 1 r 1 1 r 1 v 1 v s n 616 825 m 2 h 8 1 r 1 1 r 2 v -1 1 r -1 1 r s n 620 849 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -3 v -3 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 3 r 2 v 3 v -2 3 r s n 616 844 m -2 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 2 r s 16 3 618 857 i 13 3 619 858 i 13 -9 621 869 i 1 6 618 855 i 621 871 620 865 b 14 -3 622 877 i 14 -3 623 878 i n 629 877 m -3 2 r -2 2 r -1 2 r 1 3 r 1 1 r 1 h 1 -1 r -1 -1 r -1 1 r s 1 4 622 874 i 637 878 636 871 b 3 22 631 893 i 9 34 634 915 i 9 40 643 949 i 9 23 652 989 i 10 5 661 1012 i 9 -2 671 1017 i 9 21 680 1015 i 5 10 689 1036 i 18 -12 681 1050 i 18 -11 682 1050 i n 680 1047 m 6 10 r 3 2 r 1 1 r 3 -1 r 1 -1 r 2 -2 r -1 v -1 -3 r -4 -7 r s n 686 1057 m 2 2 r 2 h 2 h 2 -2 r 1 -2 r -1 v -2 v s 4 6 698 1035 i n 693 1049 m 2 1 r 2 1 r 8 -2 r 1 h 1 v 2 v s n 695 1050 m 3 h 7 -2 r 1 h 1 2 r -1 2 r -1 h s 18 -12 700 1077 i 18 -12 700 1078 i n 708 1071 m -3 h -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 2 r -1 3 r s n 702 1068 m -2 v -3 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 2 3 698 1075 i 720 1069 718 1065 b 16 -3 711 1082 i 13 -3 712 1083 i n 718 1092 m 727 1079 l 2 -4 r -3 v -2 v -1 v -2 h 1 v 1 h s 3 5 710 1080 i 719 1094 716 1089 b 18 -12 716 1103 i 18 -12 717 1104 i n 715 1100 m 7 11 r 2 2 r 2 h 2 h 2 -2 r 1 -1 r -2 v -1 -3 r -4 -7 r s n 722 1111 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -2 v -2 v s 4 6 733 1088 i n 728 1102 m 2 2 r 2 h 8 -2 r 1 1 r 1 1 r -1 1 r s n 730 1104 m 3 -1 r 7 -2 r 1 h 1 2 r -1 2 r -1 1 r s n 743 1125 m -2 v 2 1 r 1 v -1 1 r -3 -1 r -2 -1 r -2 -3 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 3 r 3 v s n 737 1121 m -2 v -3 v 3 -2 r 1 -2 r 3 h 3 h 2 1 r s 15 -2 745 1132 i 13 -2 745 1133 i 9 -13 751 1143 i 4 5 743 1131 i 752 1144 749 1139 b 12 -8 756 1150 i 12 -7 756 1150 i n 762 1147 m -2 3 r -1 2 r 3 v 2 2 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 3 754 1147 i 770 1145 766 1139 b 1 1 762 1149 i 9 28 763 1150 i 9 26 772 1178 i 9 -3 781 1204 i 10 12 790 1201 i 9 9 800 1213 i 9 7 809 1222 i 9 10 818 1229 i 9 9 827 1239 i 10 5 836 1248 i 9 4 846 1253 i 9 20 855 1257 i 2 864 1277 e 9 -19 869 1292 i 9 -20 869 1293 i n 866 1291 m 11 5 r 3 h 2 h 1 -2 r 1 -1 r -3 v -1 v -3 -2 r -7 -4 r s n 877 1296 m 2 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 v -2 -2 r s 6 3 875 1271 i n 878 1285 m 3 h 1 h 6 -5 r 1 -1 r 1 1 r 1 1 r s n 881 1285 m 2 -1 r 4 -6 r 2 h 2 1 r 2 v -1 1 r s 9 -20 898 1306 i 9 -19 899 1306 i n 903 1297 m -3 1 r -2 h -2 -1 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 1 r 2 1 r 1 1 r 1 3 r s n 896 1297 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 3 1 896 1305 i 911 1288 907 1286 b 11 -11 911 1304 i 10 -9 912 1304 i n 922 1309 m -16 v -4 v -1 -3 r -1 -2 r -1 h -2 h 1 1 r 1 h s 6 2 909 1303 i 924 1310 918 1307 b 9 -20 926 1319 i 9 -19 927 1319 i n 924 1317 m 11 6 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v -2 v -3 -2 r -7 -3 r s n 935 1323 m 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -1 v -2 -2 r s 6 3 933 1298 i n 936 1312 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 1 r s n 939 1312 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 2 v -1 1 r s n 961 1323 m -1 -1 r 2 -1 r 2 v 1 v -3 1 r -2 h -3 -2 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 954 1323 m -2 -1 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 2 -1 r 3 h s 12 -11 966 1329 i 10 -9 967 1329 i 1 -16 977 1334 i 6 3 964 1328 i 979 1335 973 1332 b 6 -13 985 1337 i 6 -13 986 1338 i n 988 1332 m 4 v 1 2 r 1 2 r 3 1 r 1 h 1 -1 r -1 -1 r -1 h 1 v s 4 2 982 1336 i 994 1326 988 1323 b 10 8 1001 1339 i 37 -3 1011 1347 i 37 14 1048 1344 i 37 23 1085 1358 i 37 1122 1381 e 7 6 1159 1381 i 20 -6 1156 1396 i 20 -6 1156 1397 i n 1155 1393 m 3 12 r 2 3 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v -2 -8 r s n 1158 1405 m 2 2 r 1 h 2 1 r 2 -1 r 2 -2 r 1 -1 r -2 v s 2 7 1175 1387 i n 1167 1399 m 2 2 r 1 h 8 1 r 1 1 r 1 v 1 v s n 1169 1401 m 2 h 8 h 1 h 2 v -1 2 r -1 h s 21 -6 1165 1427 i 20 -6 1166 1428 i n 1175 1424 m -2 -1 r -2 -2 r -1 -2 r -3 v 2 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r -1 2 r -1 3 r s n 1170 1419 m 1 -2 r 1 -2 r 3 -2 r 2 -1 r 3 h 3 1 r 1 2 r s 2 4 1164 1424 i 1187 1425 1186 1421 b 16 2 1175 1435 i 13 1 1176 1436 i n 1179 1447 m 12 -10 r 3 -3 r 2 -3 r -2 v -1 v -2 -1 r 1 v 1 1 r s 2 6 1175 1433 i 1180 1449 1178 1443 b 20 -7 1175 1457 i 20 -7 1175 1458 i n 1174 1454 m 3 12 r 2 2 r 2 1 r 2 h 2 h 1 -2 r 1 -1 r -3 v -2 -8 r s n 1177 1466 m 2 1 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 2 6 1194 1448 i n 1186 1460 m 2 1 r 1 1 r 8 1 r 1 1 r 2 v s n 1188 1461 m 2 1 r 8 h 1 h 2 v -1 2 r -1 h s n 1194 1485 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r 3 v -2 3 r s n 1189 1480 m 1 -2 r 1 -3 r 3 -1 r 2 -1 r 3 h 3 1 r 1 2 r s 16 2 1193 1493 i 13 1 1194 1494 i 12 -10 1197 1505 i 2 6 1193 1491 i 1198 1507 1196 1501 b 14 -5 1199 1513 i 14 -5 1200 1514 i n 1206 1512 m -3 2 r -1 2 r -1 3 r 1 2 r 2 1 r 1 h -1 v -1 -1 r -1 1 r s 1 4 1199 1510 i 1214 1512 1212 1505 b 21 177 1208 1524 i -13 -17 1210 1731 i -13 -17 1211 1730 i n 1208 1732 m 9 -7 r 2 -3 r -1 v -3 v -1 -1 r -3 -1 r -1 h -3 1 r -7 5 r s n 1217 1725 m 1 -2 r -2 v -2 v -1 -1 r -2 -1 r -2 -1 r -2 1 r s 6 -4 1194 1715 i n 1209 1719 m 1 -2 r -2 v -2 -7 r -2 v 1 h 1 h s n 1210 1717 m -1 -3 r -3 -7 r -1 v 2 -1 r 2 1 r 1 h s -13 -17 1236 1710 i -13 -17 1237 1710 i n 1230 1702 m -1 3 r -1 2 r -1 1 r -3 2 r -3 -1 r -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 1 r s n 1227 1708 m -3 1 r -2 h -3 -2 r -1 -2 r -2 -3 r 1 -3 r 1 -2 r s 3 -2 1234 1712 i 1226 1691 1223 1693 b -4 -15 1240 1698 i -3 -13 1240 1698 i n 1249 1691 m 1236 1683 l -4 -2 r -3 h -3 h 1 v -1 1 r 2 1 r -2 v s 5 -4 1238 1700 i 1251 1690 1246 1693 b -13 -17 1260 1691 i -13 -17 1261 1691 i n 1258 1693 m 10 -7 r 1 -3 r 1 -1 r -1 -3 r -1 -1 r -2 -1 r -2 h -3 1 r -6 5 r s n 1268 1686 m 1 -2 r -2 v -1 -2 r -1 -2 r -2 -1 r -1 h -3 1 r s 5 -4 1245 1676 i n 1259 1680 m 1 -3 r -1 v -2 -8 r -1 v 1 -1 r 2 1 r s n 1260 1677 m -1 -2 r -2 -7 r -1 v 1 -1 r 3 1 r s n 1280 1663 m -1 h -1 v 1 h 1 1 r -1 2 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h s n 1277 1669 m -2 1 r -3 -1 r -3 -1 r -1 -2 r -1 -3 r -3 v 1 -2 r s -4 -15 1288 1661 i -3 -13 1288 1661 i -13 -8 1297 1654 i 5 -3 1286 1662 i 1299 1652 1294 1656 b -9 -12 1304 1649 i -9 -11 1305 1648 i n 1301 1643 m 3 2 r 2 h 3 h 2 -2 r -2 v -2 -1 r 2 v 1 h s 4 -2 1301 1650 i 1298 1635 1293 1639 b 25 -176 1280 1660 i -8 -20 1305 1497 i -8 -20 1306 1497 i n 1303 1498 m 11 -5 r 2 -2 r 1 -1 r -2 v -1 -2 r -1 -2 r -2 h -3 h -8 3 r s n 1314 1493 m 1 -1 r 1 -2 r -2 v -1 -2 r -1 -1 r -2 -1 r -2 h s 7 -2 1294 1478 i n 1307 1485 m 2 -2 r -1 v -8 v 1 -1 r 1 -1 r 1 1 r s n 1309 1483 m -2 v -1 -7 r -2 v 2 h 2 1 r 1 v s -9 -20 1336 1484 i -9 -20 1337 1484 i n 1332 1475 m -1 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 3 -2 r 2 -1 r 2 h 3 1 r s n 1327 1480 m -2 h -3 -1 r -2 -3 r -1 -2 r -3 v 1 -3 r 2 -1 r s 4 -1 1333 1485 i 1331 1463 1327 1464 b -16 1342 1474 d -13 1343 1473 d n 1354 1469 m -12 -11 r -3 -3 r -3 -1 r -2 h -1 h -1 1 r 2 1 r -1 v s 5 -2 1341 1474 i 1356 1468 1350 1470 b -8 -20 1364 1472 i -8 -20 1365 1472 i n 1361 1473 m 12 -4 r 2 -3 r 1 -1 r -2 v -1 -2 r -2 -2 r -1 h -3 h -8 3 r s n 1373 1469 m 1 -2 r 1 -2 r -2 v -1 -2 r -2 -1 r -1 -1 r -2 h s 7 -3 1353 1454 i n 1366 1460 m 1 -1 r 1 -2 r -8 v -1 v 1 h 2 h s n 1367 1459 m -3 v -7 v -2 v 2 h 2 1 r 1 v s n 1390 1450 m -1 -1 r 1 -1 r 1 1 r -1 3 r -1 2 r -3 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r s n 1386 1455 m -2 h -3 -1 r -2 -3 r -1 -1 r -4 v 1 -2 r 1 -2 r s -16 1398 1450 d -14 1399 1450 d -12 -11 1410 1445 i 6 -3 1396 1451 i 1412 1444 1406 1447 b -5 -13 1417 1442 i -5 -14 1418 1442 i n 1416 1436 m 2 2 r 3 1 r 2 1 r 3 -2 r -1 v -1 v -1 -1 r -1 2 r 1 h s 4 -1 1414 1443 i 1415 1427 1409 1430 b 36 14 1418 1436 i 36 4 1454 1450 i 37 2 1490 1454 i 37 2 1527 1456 i 29 2 1564 1458 i 3 -22 1600 1474 i 3 -21 1601 1474 i n 1597 1473 m 12 2 r 3 -1 r 2 h 1 -2 r -2 v -1 -3 r -1 v -3 -1 r -9 -1 r s n 1609 1475 m 2 -1 r 2 -1 r 1 -1 r -3 v -1 -2 r -1 v -2 -1 r s 7 1 1600 1452 i n 1608 1464 m 2 h 1 -1 r 4 -7 r 1 -1 r 1 h 1 2 r s n 1610 1464 m 1 -2 r 3 -7 r 1 -1 r 2 h 1 3 r 1 v s 3 -21 1633 1478 i 3 -21 1634 1478 i n 1634 1468 m -2 2 r -2 1 r -2 -1 r -3 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -3 r 2 -1 r 3 -1 r 3 h 1 2 r 2 2 r s n 1628 1470 m -2 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -2 r 2 -2 r 2 -1 r s 4 1630 1478 e 4 1636 1457 e 8 -13 1644 1472 i 7 -12 1645 1473 i n 1656 1474 m 1652 1459 l -2 -4 r -1 -3 r -2 -1 r -1 h -1 1 r 1 v 2 -1 r s 6 1 1642 1472 i 6 1652 1474 e 3 -21 1663 1482 i 3 -22 1664 1483 i n 1660 1482 m 12 2 r 4 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -3 -2 r -8 -1 r s n 1672 1484 m 3 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -2 -2 r s 7 1 1663 1461 i n 1671 1473 m 2 -1 r 1 -1 r 4 -6 r 1 -1 r 1 h 1 1 r s n 1673 1472 m 1 -2 r 3 -6 r 1 -1 r 2 h 1 2 r 1 v s n 1697 1477 m -1 -2 r 1 h 1 1 r 1 v -2 2 r -2 h -3 h -3 -2 r -2 -2 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 4 h 2 h 3 1 r 1 3 r s n 1691 1479 m -2 -1 r -2 -3 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 3 h s 8 -14 1704 1481 i 7 -12 1705 1481 i -4 -15 1716 1482 i 6 1 1702 1480 i 1718 1483 1712 1482 b 2 -14 1724 1483 i 2 -15 1725 1484 i n 1726 1477 m 1 4 r 2 2 r 1 1 r 3 1 r 2 -1 r -1 v -1 -1 r -1 1 r 1 1 r s 4 1 1721 1483 i 1730 1470 1723 1469 b 7 3 1741 1481 i p end %%EndDocument @endspecial 98 x Fa(Figure)15 b(2:)k(Comm)o(unications)c(rates)f(for)h (\\buc)o(k)o(et-brigade")f(messages)h(in)g(a)g(ring)g(of)f(four)g(pro)q (cessors)74 2287 y(on)h(an)g(In)o(tel)h(ipsc/860.)963 2790 y(3)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 17:49:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08019; Tue, 5 Jan 93 17:49:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27933; Tue, 5 Jan 93 17:49:38 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 22:49:37 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27924; Tue, 5 Jan 93 17:49:36 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA22393; Tue, 5 Jan 93 15:49:25 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0n9N5Q-0016ZKC; Tue, 5 Jan 93 15:49 MST Message-Id: Date: Tue, 5 Jan 93 15:49 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Comments on "ready receiver" part of draft implementation We would like to second the position taken by Gropp and Lusk. With our Sandia/Univ. of New Mexico OS (SUNMOS) project, we have also discovered the value of posting non-blocking receives for anticipated messages. In SUNMOS 1, we provided this facility at the kernel level. However, we also saw the benefit of having permanently posted receives. That is, once such a receive is posted, any matching message would get delivered to the specified location (overwriting previous data), with an associated integer flag being incremented upon message receipt. Many applications utilize repetitive boundary data exchanges; this strategy avoids the costs of independent receives. We have proceeded to a new kernel design that uses posted receives as the basis for all message communication. A message passing design document is currently in the works :-). We will be happy to electronically forward that document to you as it becomes available. Stephen Wheat and Barney Maccabe SNL UNM/SNL From owner-mpi-profile@CS.UTK.EDU Wed Jan 13 17:46:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11465; Wed, 13 Jan 93 17:46:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01528; Wed, 13 Jan 93 17:25:49 -0500 X-Resent-To: mpi-profile@CS.UTK.EDU ; Wed, 13 Jan 1993 17:25:47 EST Errors-To: owner-mpi-profile@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01521; Wed, 13 Jan 93 17:25:46 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA01688; Wed, 13 Jan 93 17:25:45 -0500 Date: Wed, 13 Jan 93 17:25:45 -0500 Message-Id: <9301132225.AA01688@thud.cs.utk.edu> To: mpi-profile@cs.utk.edu Subject: this is a test testing From owner-mpi-pt2pt@CS.UTK.EDU Fri Jan 15 12:26:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14370; Fri, 15 Jan 93 12:26:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03545; Fri, 15 Jan 93 12:25:16 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 15 Jan 1993 12:25:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03526; Fri, 15 Jan 93 12:25:09 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA19456; Fri, 15 Jan 93 17:24:54 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14219; Fri, 15 Jan 93 10:23:53 MST Date: Fri, 15 Jan 93 10:23:53 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301151723.AA14219@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: "Message Capsule" Proposal (long) Hi all, Here is a proposal that we think may help reduce the number of separate send and receive routines to something much less than "512" without eliminating any of the nice features that we haven't been able to say "no" to. The proposal also allows for "simple" versions of send and receive routines for novice users. The proposal borrows ideas from Jim Cownie, Paul Pierce, and others (without their consent-- don't blame anyone but us!). We hope that this proposal will at least start a few good arguments. :) Leslie Hart hart@fsl.noaa.gov Tom Henderson hender@fsl.noaa.gov Bernardo Rodriguez bernardo@fsl.noaa.gov NOAA Forecast Systems Lab ----------------------------------------------------------------------------- A PROPOSAL FOR "MESSAGE CAPSULES" 1 Introduction to Message Capsules To send a message, a user must first create and initialize a data structure called a "message capsule". (In Fortran77 a message capsule might be an integer array, possibly containing pointers used by underlying C routines. Another way might be to use the POSIX approach mentioned by Bob Knighten.) The message capsule is an opaque object: a user should only access the contents of a message capsule using a special set of routines. A message capsule contains a complete description of the user data in the message including user data location, data type(s), and optional stride information. The capsule will also contain information describing the data in an incoming message. (In the current proposal this information is transmitted with each message. This is only necessary if we allow reception of a message by a process that does not know its complete data description.) A message capsule also contains message tag, context, control flags, and "matching" information. "Matching" information is used by a receiving process to select a message for receipt. "Matching" criteria has not been defined for MPI. A receiving process must also create and initialize a message capsule. When a message is received, the receive routine uses the information in the message capsule to move data from system buffers (or communication hardware) to location(s) in user space described in the capsule. The information in the message capsule is used by the receive routine to guide any data translation that may be necessary in a heterogeneous system. Note that the use of message capsules does not imply extra data movement. The message capsule just tells the system "Here's where to put the data when it arrives...". In the current proposal, message capsules are dynamic data objects. A message capsule can be reused once the data described by it is no longer required for message passing communication. Routines are provided to create and destroy message capsules, attach descriptions of user data, inquire about the contents of a message capsule, and modify message context. Some proposed routines are described below. The send and receive routines have "message capsule" as an additional parameter. Context is not required as an explicit input parameter to these routines. Proposed (incomplete) syntax for these routines is described below along with syntax for check, wait, probe, and release routines. Proposed syntax for "simplified" versions of send and receive routines are also described below. 2 Program Flow General steps for typical send -> receive communications using message capsules: Case 1. Receiving process knows the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 2) RECEIVING PROCESS: Receiving process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 3) SENDING PROCESS: "Send" message. During a send operation, the send (or wait for a non-blocking send) routine uses information in the message capsule to collect user data, construct a "message" (using system buffer(s) if necessary), and send the message to a specified destination. Depending on the implementation, the send routine may also perform data translation in a heterogeneous system. While a send operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. A send routine will return an error if the message capsule has not been properly initialized. 4) RECEIVING PROCESS: "Receive" message. During a receive operation, the receive (or possibly wait for a non-blocking receive) routine distributes received data to user location(s) described in the message capsule. Depending on the implementation, the receive routine may also perform data translation in a heterogeneous system. While a receive operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. The receive (or wait) routine will return an error if the data in the received message does not match the description in the capsule (ie. mismatch in data length(s) or type(s)). An error will also be returned if the message capsule has not been properly initialized. 5) RECEIVING PROCESS: (OPTIONAL) "Information" about message described by a capsule. Information about a message may be obtained by calling various "info" routines with the corresponding message capsule specified as an argument. Information returned by these routines includes data types, data sizes, data strides, sending process, message tag, and message context. "Info" routines will return an error if a message has not been successfully "probed" or received. Case 2. Receiving process does not know the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines (same as Case 1). 2) RECEIVING PROCESS: Receiving process creates a message capsule by calling the capsule creation routine. The message capsule may optionally be initialized in this step by calling the data description routines (though it may need to be re-initialized after the message arrives). Message context specification routine may also be called to select context if "default" context is not desired. 3) SENDING PROCESS: "Send" message. Same as Case 1. 4) RECEIVING PROCESS: (NOT OPTIONAL) "Probe" message (inquire if a message with "matching" characteristics has arrived-- "matching" criteria have not yet been defined for MPI). If a message with "matching" characteristics has arrived, data description information is loaded into a message capsule specified as an argument to the probe routine. The message is removed from the system message queue so no other thread may probe or receive the message unless it is "released" by the thread (see "release" in step 5 below). A flag is set in the message capsule to indicate that the data description of the arrived message is valid. This flag is used by the "info" routines. Note that the message capsule must be able to separately store the user's data description and the data description of the "arrived" message. "Info" routines will only return information about "arrived" messages. 5) RECEIVING PROCESS: (NOT OPTIONAL) "Info" routines MUST be called before a receive is posted when data description is not known by the receiving process. In a C program, information returned by these routines may be used to guide dynamic allocation of user data structures to store message data. In a Fortran77 program, this information can be used to check if available static arrays are large enough to hold message data before a receive is posted. 6) RECEIVING PROCESS: (OPTIONAL) "Release" message. If a thread does not wish to receive a message after a successful probe, it must release the message so other threads will have an opportunity to probe or receive the message. The release routine clears the flag mentioned in step 4 and puts the message back into the system message queue. Return to step 4. 7) RECEIVING PROCESS: Receiving process initializes the message capsule by calling data description routine(s) using information returned by the "info" routines. A flag is set in the message capsule to indicate that it has been initialized by the user. If the message capsule was initialized in step 2 and it does not have the same description as the arrived message, then the data description must be "reset" using a special routine, and then initialized. 8) RECEIVING PROCESS: "Receive" message. Same as Case 1. 3 Proposed Message Capsule Initialization and Inquiry Routines Basic functionality of proposed routines are described below. A sample syntax is shown just to make explanation easier. (For example, the issue of whether these routines are functions or subroutines is not important at this point, parameter ordering has not been given any thought, etc.) Also, routine names have been chosen to be descriptive an do not conform to any string length limitations. 3.1 Message Capsule Creation and Destruction Routines MPI_CREATE_CAPSULE (CAPSULE) Create message capsule CAPSULE. MPI_DESTROY_CAPSULE (CAPSULE) Destroy message capsule CAPSULE. 3.2 Data Description Routines Each message capsule contains one or more "data items" to describe the data in the corresponding message. Each data item contains information about data type, data length, and storage location(s). Each data item is created by a single call to a MPI_ATTACH_xxx() routine. Two kinds of data items are supported: "vector" and "strided". A vector data item is used for user data stored in contiguous memory. A strided data item is used for non-contiguous user data that can be described as a sequence of identically sized data blocks separated by a fixed stride. More general user data arrangements may be described by multiple vector and strided data items. (Note that the vector data item is a subset of the strided data item. Vector data items could be removed to get rid of almost half of the MPI_ATTACH_xxx() routines. We included the vector data item because it simplifies the most commonly used data description.) MPI_ATTACH_INT_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH integers. Storage location is user array ARRAY. MPI_ATTACH_REAL_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH real numbers (floats) stored in user array ARRAY. (etc. for other data types)... MPI_ATTACH_INT_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of integers. Each block has BLOCK_LENGTH integers. Starting elements in each block are separated by STRIDE integers. Storage location of first block is user array ARRAY. MPI_ATTACH_REAL_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of real numbers (floats). Each block has BLOCK_LENGTH real numbers. Starting elements in each block are separated by STRIDE real numbers. Storage location of first block is user array ARRAY. (etc. for other data types)... MPI_ATTACH_RESET (CAPSULE) All data items are removed from message capsule CAPSULE. The flag that indicates that CAPSULE has been initialized by the user is cleared. This routine can be used in step 7 of Case 2 in Section 2 above to remove any existing data item(s) before re-initializing the message capsule. It must be used in this situation if the message capsule contains a user data description that does not match the data description of the arrived message. This routine is also used to discard data descriptions that are no longer needed allowing message capsules to be re-used. 3.3 Context Modification Routine MPI_SET_CONTEXT (CAPSULE, CONTEXT) The context for the message described by message capsule CAPSULE is set to context CONTEXT. 3.4 "Information" Routines These routines must be used when a receiving process does not know the data description of a message it wants to receive. Use of these routines in other cases is optional. In all cases, "length" refers to number of elements (integers, floats, ...) not number of bytes. All of these routines return an error if the specified message has not been successfully probed or received. Only information about messages that have actually arrived is returned. Note that no "info" routine is required for context since this must always be known by the receiving routine. MPI_INFO_NUM_DATA_ITEMS (CAPSULE) Returns the number of data items in the message. MPI_INFO_DATA_ITEM_DESCRIPTION (CAPSULE, ITEM_NUMBER) Returns a description of data item number ITEM_NUMBER in the message described by message capsule CAPSULE. Valid return values include INT_VECTOR_ITEM, REAL_VECTOR_ITEM, INT_STRIDE_ITEM, REAL_STRIDE_ITEM, etc. MPI_INFO_VECTOR_ITEM (CAPSULE, ITEM_NUMBER, LENGTH) LENGTH is set to the length of the vector data described by data item number ITEM_NUMBER in message capsule CAPSULE. An error is returned if the data item is not a vector data item. MPI_INFO_STRIDE_ITEM (CAPSULE, ITEM_NUMBER, NUM_BLOCKS, BLOCK_LENGTH) NUM_BLOCKS is set to the number of blocks in the strided data described by data item number ITEM_NUMBER in message capsule CAPSULE. BLOCK_LENGTH is set to the length of each block. An error is returned if the data item is not a strided data item. Note that stride is not returned since there is no requirement that strides used by the sending and receiving processes match. MPI_INFO_TAG (CAPSULE) Returns the message tag for the message described by message capsule CAPSULE. MPI_INFO_SENDER (CAPSULE) Returns the sending process for the message described by message capsule CAPSULE. 4 Proposed (Partial) Syntax for Send, Receive, Check, Wait, Probe, and Release Routines Syntax examples showing how message capsules affect send, receive, check, wait, probe, and release routines are shown below. Routine naming is for clarity only and does not conform to any of the MPI naming styles. 4.1 Send Syntaxes for the three flavors of send discussed at the Dallas meeting ("synchronous", "blocking", and "non-blocking") are shown below. (These three routines could be reduced to one if the "communication mode" parameter were used. It is also possible to embed even more information in the message capsule such as "communication mode", message tag, and "matching" criteria. We show examples for this case in Section 8.) MPI_SEND_SYNCH (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "synchronous" send. MPI_SEND_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "blocking" send. MPI_SEND_NON_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "non-blocking" send. 4.2 Receive Syntaxes for the two flavors of receive discussed at the Dallas meeting ("blocking" and "non-blocking") are shown below. (These two routines could be reduced to one if the "communication mode" parameter were used.) "Matching" criteria have been left undefined. MPI_RECV_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). MPI_RECV_NON_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "non-blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). (This will only happen if this routine is called when a "matching" message has already arrived.) 4.3 Wait and Check The wait and check routines use the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). These routines are used with the non-blocking send and receive routines. MPI_CHECK (CAPSULE) CALLED BY A SENDING PROCESS: Return value indicates whether a send operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking send has not been previously posted for CAPSULE. If the send operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. CALLED BY A RECEIVING PROCESS: Return value indicates whether a receive operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking receive has not been previously posted for CAPSULE. If the receive operation has completed, error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). If the receive operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. (Depending on the implementation, user data locations described in CAPSULE may be loaded by this routine, or they may already be loaded by the time this routine is called.) MPI_WAIT (CAPSULE) Same as MPI_CHECK() except that the routine will not return until a "matching" message arrives. 4.4 Probe and Release The probe routines also uses the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). Both wait and check styles of probe are proposed. MPI_PROBE_CHECK (CAPSULE, "MATCHING CRITERIA") If a message with characteristics satisfying "MATCHING CRITERIA" has arrived, copy its description into message capsule CAPSULE. The message is removed from the system "message queue" and may not be probed or received by another thread unless explicitly released using the release routine. "Information" routines can be called using CAPSULE to get information about the message. A receive can also be posted for this message by the thread that owns CAPSULE (after modifying CAPSULE using information from the "info" routines). If a "matching" message has not arrived, return immediately without modifying CAPSULE. Return value indicates whether a message has arrived or not. MPI_PROBE_WAIT (CAPSULE, "MATCHING CRITERIA") Same as MPI_PROBE_CHECK() except that the routine will not return until a "matching" message arrives. MPI_RELEASE (CAPSULE) The previously probed message described by message capsule CAPSULE is put back in the system "message queue". It may now be probed or received by any thread. An error will be returned if the message has not been probed. Data description of arrived message in CAPSULE is invalidated. 4.5 Simplified Versions of Send and Receive Routines Good arguments have been made (we think) for "simplified" versions of send and receive routines that are less flexible and "easier to use". These routines could be built on top of the routines proposed above or implemented directly. The simplified routines proposed below cannot be used with contexts or probe or "info" routines. Only contiguous user data is supported. Receiving process must know message data description. Actually, it is not clear to us that these routines are really easier to use. They do conform more closely to "common practice" though. Simplified versions of the check and wait routines can also be made (we've left them out-- this thing is long enough already!). MPI_SEND_SYNCH_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Send message consisting of LENGTH elements from location ARRAY to process DESTINATION using message tag TAG. DATA_TYPE is the data type for ARRAY. (Pre-defined values for DATA_TYPE are MPI_INT, MPI_REAL, etc. DATA_TYPE is required for heterogeneous communication.) This is a "synchronous" send. MPI_SEND_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "blocking" send. MPI_SEND_NON_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "non-blocking" send. MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in array ARRAY. Received data type must be DATA_TYPE. Data length must be LENGTH. This is a "blocking" receive. Error conditions will be returned if LENGTH or DATA_TYPE are not consistent with the content of the received message. MPI_RECV_NON_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Same as MPI_RECV_BLOCKING_SIMPLE() except that this is a "non-blocking" receive. 5 Code Fragment Examples in "C" Here are some code fragment examples of how programs using message capsules might look. Lots of stuff has been left out (some variable declarations and initializations, error control using return values, etc.) for brevity. Code segments for sending and receiving processes are shown separately for clarity. "Matching" criteria for receives is left undefined. Process ID is assumed to be an integer. 5.1 Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.2 Blocking Send -> Receive Vector Data Item: Receiver Does Not Know Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int tag, receiver_pid, vec_length, num_items, item, data_item; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_PROBE_WAIT(capsule, "MATCHING CRITERIA"); /* probe until message arrives*/ /* Get data description of arrived message using "info" routines and */ /* initialize message capsule. */ num_items = MPI_INFO_NUM_DATA_ITEMS(capsule); /* get number of data items */ /* (Resetting capsule is optional if this is the only place this occurs in */ /* the code. Resetting is necessary if this code fragment (excluding */ /* capsule creation and destruction) is in a loop. Resetting does not */ /* affect information contained in the capsule about the arrived message.) */ MPI_ATTACH_RESET(capsule); if (num_items == 1) /* Deal with single data item message. */ { /* What kind of data item is in the message? */ data_item = MPI_INFO_DATA_ITEM_DESCRIPTION(capsule, 0); if (data_item == REAL_VECTOR_ITEM) /* Deal with real vector data item. */ { /* Get length of vector item and store in "vec_length". */ MPI_INFO_VECTOR_ITEM (capsule, 0, &vec_length); if (vec_length <= 1000) /* Data fits in existing user array. */ { /* Initialize first data item to be "vec_length" floats stored */ /* at location "my_vector". */ MPI_ATTACH_REAL_VECTOR(capsule, vec_length, my_vector); } /* end of vec_length if */ else { ... /* allocate a larger buffer to hold message data */ } /* end of vec_length else */ } /* end of data_item if */ else { ... /* deal with other possibilities for first data item */ } /* end of data_item else */ } /* end of num_items if */ else { ... /* deal with multiple data items */ } /* end of num_items if */ MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.3 Blocking Send -> Non-Blocking Receive Complicated Message With Context: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid, my_context; float my_vector[1000], my_array[2000]; /* user data */ int my_indices[200]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* my_context must be set up earlier */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* must be same as sender's context */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_RECV_NON_BLOCKING(capsule, "MATCHING CRITERIA"); /* post receive */ ... /* "do something" */ MPI_WAIT (capsule); /* wait for message when done */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.4 Blocking Send -> Receive With "Simplified" Routines DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* send msg to receiver */ MPI_SEND_BLOCKING_SIMPLE (receiver_pid, tag, my_vector, 1000, MPI_REAL); ... } /* end of i for loop */ ... RECEIVING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* receive message */ MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", my_vector, 1000, MPI_REAL); mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... 6 Advantages and Disadvantages of "Message Capsule" Approach 6.1 Advantages - The number of separate send and receive routines is greatly reduced without sacrificing functionality. - A user who is used to "common practice" can use the simplified versions of the routines. - A user who wants more flexibility only needs to learn about the features required for his/her specific application. (For example, if I only need contiguous messages, then I don't need to know anything about strided data items. If the receiving process always knows the data description of received messages, then I don't need to know about the probe and "info" routines.) - "Hidden states" are removed so multi-threaded applications won't get confused. - Encapsulation of features in message capsules allows new features to be added later without modifying syntax of existing routines. A new feature would require addition of one or more new routines to modify and examine message capsules. 6.2 Disadvantages - The use of message capsules does not conform exactly to "common practice". It's fairly close to PVM though. - There is no way to prevent a user from messing with a message capsule directly. This is the same problem encountered in C with (FILE *) structures. The problem can be partially alleviated by including "magic numbers" in message capsules. - Possible slight performance degradation due to "casing" in receive routines. - Data description must be sent along with each message. This is only a significant problem if lots of short messages are sent by a program running on a machine with low communication latency. Sending data description information could be avoided if we do not allow a receiving process to be ignorant of the data description of incoming messages. 7 Unresolved Issues - Can the message capsule concept work well with collective communication routines? - Is it really necessary to support multiple data items in a message capsule? It is if we want to allow mixed-type messages. 8 Code Fragment Example With Even More Information Hidden In the Message Capsule The code fragment presented in Section 5.1 has been modified slightly to indicate how the capsule could possibly contain additional information to further reduce the number of send/receive calls and the number of parameters. Message tag and "matching" criteria for receive operations are now "hidden" in the message capsule. Communication "mode" (synchronous, blocking, or non-blocking) is used to reduce the number of send and receive routines. Communication "mode" is also "hidden" in the message capsule. Four new routines have been added. Routine MPI_ASSIGN_TAG() sets message tag. Routines MPI_SET_SEND_MODE() and MPI_SET_RECV_MODE() set communication "mode" (synchronous, blocking, or non-blocking for send, blocking or non-blocking for receive). Routine MPI_SET_MATCH() sets matching characteristics for receive operations. The send routines (MPI_SEND_SYNCH(), MPI_SEND_BLOCKING(), and MPI_SEND_BLOCKING()) have been combined into one routine (MPI_SEND()). The receive routines (MPI_RECV_BLOCKING() and MPI_RECV_NON_BLOCKING()) have been combined into one routine (MPI_RECV()). Both MPI_SEND() and MPI_RECV() have a reduced number of parameters. In the example below, lines containing "###" indicate changes from the example in Section 5.1. (Example from 5.1) Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 4 lines added ### */ MPI_ASSIGN_TAG (capsule, tag); /* Assign the tag to the capsule */ /* Set send mode to "blocking". Valid values for send mode are */ /* MPI_MODE_SEND_NON_BLOCK, MPI_MODE_SEND_BLOCK, and MPI_MODE_SEND_SYNC. */ MPI_SET_SEND_MODE (capsule, MPI_MODE_SEND_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* ### tag parameter disappears from parameter list ### */ MPI_SEND(capsule, receiver_pid); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 5 lines added ### */ /* Set matching criteria in capsule */ MPI_ASSIGN_MATCH(capsule, "MATCHING CRITERIA"); /* Set the receive mode to "blocking". Valid values for receive mode are */ /* MPI_MODE_RECV_NONBLOCK and MPI_MODE_RECV_BLOCK. */ MPI_SET_RECV_MODE (capsule, MPI_MODE_RECV_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* ### tag parameter disappears from parameter list ### */ MPI_RECV(capsule); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 05:16:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00760; Sat, 16 Jan 93 05:16:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24719; Sat, 16 Jan 93 05:16:13 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 05:16:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24593; Sat, 16 Jan 93 05:15:56 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21412; Sat, 16 Jan 93 02:15:53 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01226; Sat, 16 Jan 93 02:21:14 PST Message-Id: <9301161021.AA01226@jewel.us.oracle.com> Date: Sat, 16 Jan 93 02:21:14 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: mpi Dear Sirs, Over the past 4 years, some of us at Oracle have been addressing many of the same problems that you are addressing in your MPI effort. We have approached the problem from a C/Unix/Systems Software point of view instead of a Fortran/Scientific Software point of view. We learned of your effort from Sandia, whom we are hoping to work with in developing systems software for MP machines, and the communications interface will be a substantial part of this effort. I sent the following comments about MPI to Jack Dongarra. He suggested dropping the note on the 'pt2pt' mailing list. My primary hope is that my radically different background and point of view will stimulate new ideas. My expectations, however, are that you will find me obnoxious and ask me to go away. :-) Cheers, Chuck Simmons csimmons@oracle.com General Comments ---------------- The proposed interface specification is at best disappointing. From our point of view, it essentially standardizes and extends the existing nCUBE model of a message passing interface. We explicitly rejected this model two years ago. In this document, I'll describe why I think this interface model is inadequate. The most important aspect of the proposed standard is its lack of reference to prior related work. Communications interfaces have been standardized before. Thus, the most important question that the proposed standard must address is: Why does no existing interface meet the requirements of our applications? Associated with this are the questions: What concepts can we steal from existing interfaces? What can we learn from past mistakes? The current work on the MPI interface appears to be occuring in a vaccuum, ignoring prior work. The introduction to the draft standard states: "The main advantages of establishing a message passing standard are portability and ease-of-use." If that is the case, then the MPI effort is seriously misguided. For example, the existing TCP/IP standard is far more portable and easier to use than MPI will ever be. However, I would agree that TCP/IP is not adequate as a communications interface for an MP machine. The reason for this is that MP machines provide relatively high-bandwidth low-latency communications. To take advantage of this, a very lightweight communications protocol is required. We first need a highly efficient communications interface; we then need a portable and easy-to-use interface. An efficient communications interface needs to provide the following properties: (1) We require the ability to copy messages from a user address space directly onto a wire without an intervening memcpy() of the message into a system buffer. Similarily, we require that received messages be able to move from a wire into a user address space without an intervening memcpy(). (2) We require that messages be packetized. That is, in TCP/IP, individual message packets are concatenated together at the sender and sent out as a stream of data. This requires the receiver to parse the incoming data stream to recover the individual packets. In an efficient interface, we want the packet boundaries to be preserved. Once we have outlined our requirements from the communications interface, the next question to ask is whether we are ready to standardize on an approach. Standards bodies work best when they standardize existing work, and work worst when they attempt to design. Standardizing too early on an inferior interface will impede research efforts to improve the interface, and additional work will be needed when it becomes time to standardize the desired interface. If it is decided that a standard is desired, then the scope of the standard needs to be carefully delineated. A reasonable scope is that of defining a communications interface at about the ISO network or transport layer. Unfortunately, some of the text in the draft standard suggest that some of the researchers want to address all of the issues associated with building an MP operating system. For example, section 5 refers to creating and destroying processes and to addressing distributed computing in a heterogeneous environment. Appendix A contains routines that are completely unrelated to communications, and which may well conflict with other existing standards, or which are already well defined in other existing standards. (Consider, for example, the date and time routines. Compare and constrast these to functionality provided by ANSI C and/or Posix.) In general, most of the draft standard stays within a reasonable scope. Attempts to extend the standard beyond that reasonable scope should be brutally (:-) suppressed. The core issues of the standard are: 1) Point-to-point communications. This is the heart and soul of the standard. 2) Communications amongst multiple instantiations of a single program (a group or collection), and communications between these collections. This is a reasonable topic to explore since these collections will be common entities on MP machines. However, this is far less important than point-to-point communications. Given a good portable interface for point-to-point communications, an application can fairly easily implement completely portable and reasonably efficient inter- and intra- collection communications. 3) Reliable messaging and sequencing of messages. This is, for the most part, explicitly ignored by the draft standard. However, this is far more important than (2). As mentioned before, applications don't need (2) to be standardized in order to be portable. However, it is extremely difficult to write the code needed to provide robust, efficient reliable messaging. 4) Threads (aka "Communications Contexts"). For the most part, this is orthogonal to communications. Threads need to be addressed to the extent that the communications interface model must be capable of coexisting with a reasonable threads implementation. In summary, the draft standard should contain a section describing the scope of the standard. Issues such as heterogeneity, performance tracing, parallel I/O, process creation, load balancing, and environmental queries should be placed explicitly outside the scope of the standard, except to the extent that the standard needs to coexist with these features. [Note: On page 3, a reference is made to dynamic load balancing. The interesting problem of dynamic load balancing from the communications interface point of view is that some addresses known to some processes may become invalid at arbitrary points in time. The operating system may need to be able to forward messages that were sent to an obsolete address.] Specific Comments ----------------- My specific objections to details of the MPI standard fall into the following areas: 1) Reliability and sequencing. I feel that the standard should pay far more attention to these subjects. Reliability and sequencing of messages are extremely important in industrial strength applications. Because of the importance of this area and its difficulty of implementation, the standard should pressure hardware vendors and operating system vendors to provide this functionality. On page 6 of the draft standard, for example, it is explicitly allowed that an operating system may discard a message if that message would overflow system buffers, and the operating system need not inform the sender that the message has been discarded. This immediately requires most applications to make use of inefficient algorithms to detect when a message may have been discarded and retry the send of the message as well as to detect when a message has been successfully received but also retried. This is so onerous, that we have asked nCUBE to implement hardware for their next generation system so that when an OS receives a message, it can immediately return one of the following stati to the sending OS: message accepted; message tossed due to parity error; message tossed due to lack of space. Allowing unreliable messages to be implemented encourages the implementation of unreliable applications. 2) The mechanisms used to receive a message are inefficient. In particular, many applications make use of variable length messages. The current proposal only works well for fixed length messages. In the face of variable length messages, the current proposal requires that an application either over allocate storage; or it requires that the application allow the OS to buffer the message, and then the application will probe for the message to determine the length, allocate a correctly sized buffer, and copy the message into the new buffer. Essentially, we want the OS to be able to copy messages off of a wire directly into user address space, so that it is then easy for the application to access variable length packets. We are willing to accept interfaces that require the hardware to implement paging for efficient implementation. 3) The 'mode' argument on the send and receive functions is inefficient. We strongly suspect that this argument will virtually always be a constant. If that is the case, it is more appropriate to make this constant part of the function name instead of an argument. For example, the obvious implementation of the mpi_csend() routine would look like: int mpi_csend (mode, buf, dest, type, len) ... { switch (mode) { case NON_BLOCKING: return mpi_csend_blocking (buf, dest, type, len); case BLOCKING: return mpi_csend_nonblocking (buf, dest, type, len); case SYNCHRONOUS: return mpi_csend_synchronous (buf, dest, type, len); default: return (save_errno, -1); } } static int mpi_csend_nonblocking (buf, dest, type, len) ... { trap to os; save any error; return value; } static int mpi_csend_blocking (buf, dest, type, len) ... { int msgid; msgid = mpi_csend_nonblocking (buf, dest, type, len); mpi_wait (msgid); return status/length sent; } ... The costs of this approach are: (A) An extraneous error condition when the value of 'mode' is out of bounds. Making the mode part of the function name allows this error to be detected at compile time. (B) Additional cpu cycles wasted passing an additional argument to various routines. (C) Cpu cycles as we decide which piece of code to execute in response to the 'mode'. Further, we take the point of view that the non-blocking operations are of fundamental importance, and that the other modes of operation are relatively minor, uninteresting envelopes around this fundamental operation. 4) The send and receive routines for scatter-gather messages ignore prior work. Consider, for example, the closely related Unix readv() and writev() operations. These are specified as: int readv (int fd, struct iovec *iov, int iovcnt); Since an 'iovec' structure is already defined, this structure should be used instead of inventing a new paradigm that separates this into two distinct arrays. In particular, the draft standard pretty much requires system software to perform an expensive translation to convert iovec's into separate arrays. [Note also that we consider the sending of a contiguous message as a relatively uninteresting envelope around the more fundamental transmission of a scatter-gather message.] 5) The send and receive routines specify the use of a pair of arguments. At a conceptual level, these arguments are a single concept: the address to which a message should be sent (or from which the message should be received). This approach has numerous problems: A) Passing two arguments to specify an address is slower than passing a single argument. B) This implies an implementation whereby each process has one message receive queue. To pull messages off this queue in an order other than that in which the message was received requires a potentially time-consuming queue scan to search for a desired message. A much better approach would allow the OS to take a received message and quickly hash it to one of a few receive queues for a process. Since the code that sends a message is generally closely related to the code that receives a message, the sender can make sure that it puts the message on a queue in such a fashion that the receiver will never be interested in anything other than the head of one of its queues. C) Allocation of message type constants by library code in a robust fashion is exceedingly difficult. If message types are to be required, the standard must address the mechanisms that are to be used to allocate message types so that separately developed libraries are guaranteed to have no conflicts. We prefer the concept of a "port". Essentially, this allows a process to ask the OS to create another receive queue for the process. Sending processes then send messages to a specific queue. A receiving process can retrieve a message from the head of a specific queue, or it can use a routine similar to the Unix select() system call to provide a bitmap of interesting queues and retrieve a message from the head of one of these queues. Taking the above into account, we would then specify the interface as: /* non-blocking scatter/gather recv */ int mpi_recv (port_t local_port, struct iovec *iovecp, int iovec_len); /* blocking version of the above */ int mpi_recvb (port_t local_port, struct iovec *iovecp, int iovec_len); /* synchronous version of the above */ int mpi_recvs (port_t local_port, struct iovec *iovecp, int iovec_len); /* vector versions of the above */ int mpi_recvv (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvb (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvs (port_t local_port, void *bufp, int blklen, int stride, int nblks); /* contiguous versions of the above */ int mpi_recvc (port_t local_port, void *bufp, int buflen); int mpi_recvcb (port_t local_port, void *bufp, int buflen); int mpi_recvcs (port_t local_port, void *bufp, int buflen); Also, in the above, the vector versions of the above routines may need additional work. We envision an rpc protocol whereby an rpc header is prepended to a vector that is to be scattered. The application would like to receive the rpc header into one buffer and then scatter the vector into numerous locations. The most general approach would be for the primitive operation to specify a list of arrays into which the received buffer is to be vector scattered. For mpi_infos() and mpi_infot()... Here you are addressing the problem of how to handle an OS header that is prepended to every message. We agree that the general problem is somewhat tricky. Which is probably why the proposed implementation has some constraints on it that are less than trivially easy to use. Note that in a C program running on Unix, the application may need to disable signals before it calls a recv routine, and re-enable the signals after calling a routine to retrieve header information. One question that needs to be answered is the purpose of this header information. If the purpose is application specific, then perhaps the application should have embedded this information in its RPC header. (Our most generally used RPC protocol does this. This approach also allows a thread context pointer to be easily embedded in the message and appropriately retrieved.) If the purpose of this header is for security and verification that a sender has permission to perform an operation, then the simple source/type information available here is probably inadequate. (The existing Unix paradigm is to provide a cred_t structure in this header that specifies the credentials of the sender. This header information is sufficiently burdensome that an application may want to tell the operating system whether or not to attach credentials information to each message.) For mpi_probe(), we strongly desired both blocking and non-blocking versions of this. The non-blocking version simply tests for the presence of a message and returns. The blocking version blocks until a desirable message is present. Consider again the select() system call which provides for both of these functionalities as well as the ability to specify a timeout on the blocking version of the operation. I'm not qualified to comment on groups since we don't use those much, and when we do, we completely ignore little things like topology and locality. However, lack of qualification never stopped me before... We've been toying with the idea of an "array port". The concept is not well defined. For certain types of applications, we may be able to create a port in a number of processes such that the ports have the property that the bit pattern of one port is easily computable from the bit pattern of another port. The primary goal here is to use a very small amount of storage in each process to specify the addresses of all processes. I would appreciate a lot more information about the intended use of groups. Certainly the proposal is far more complicated than anything I would ever use. Also, it wouldn't work for things that I would use. In particular, when I use a group, the first thing that I am doing is creating a list of addresses for all processes in the group. I cannot implement this using MPI_DEFRG. In particular, the processes in the group don't know the addresses of the processes in the group until after all processes have become part of the group. Typically, what we do is have one process in the group store its address in a well known location (a nameserver). Other processes in the group wait for the first process to appear, and then they send their addresses to the first process. The first process generates the list of addresses and broadcasts it back to all of the processes (now that all the processes are known). Thus, the list of addresses is our canonical data structure. Our canonical operations are the routines that help create this data structure, the routines that help us broadcast the data structure, and the routines that allow us to communicate pt-to-pt to an indexed address in the data structure. This approach is easily extended to allow processes outside the group to communicate with processes inside the group. The mpi_*unpack routines specify that the 'msg' argument is an output argument. It should really be an input argument. For the general pack/unpack routines; again, using a 'struct iovec *' instead of would prevent conflicts with existing paradigms. For section A.4, this section is completely outside the scope of the document. The communications contexts essentially implement threads. The communications routines should not specify how you get information about threads, nor how you go about creating a new thread. Also, the PUSH and POP approach to communications and group contexts looks like a design flaw waiting to bite. In section A.5, the cpu,date,machine,infmn,time, and wall routines are outside the scope of the document. The date and time functions are particularly suspect. Compare then to the ANSI C and Posix routines for manipulating the date and time. The mpi_etext and mpi_error routines are reasonable. Note that in our heavily layered applications, we will constantly need to convert mpi_error codes into Unix error codes, and it might be nice if the MPI error handling paradigm was better integrated into Posix error handling. (All we care is that in addition to implementing mpi_error and mpi_etext, the implementation set 'errno' to a reasonable value.) Note that it is not at all clear that communications contexts will work except for in highly restricted circumstances. They are less general than either threads or ports. In fact, I suspect that ports can provide all the needed functionality of communications contexts, and message types, and pids with less hassle. From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 06:04:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11435; Sat, 16 Jan 93 06:04:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00620; Sat, 16 Jan 93 06:03:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 06:03:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00612; Sat, 16 Jan 93 06:03:37 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21711; Sat, 16 Jan 93 03:03:35 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01231; Sat, 16 Jan 93 03:08:56 PST Message-Id: <9301161108.AA01231@jewel.us.oracle.com> Date: Sat, 16 Jan 93 03:08:56 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: RE: Communication contexts > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather Allow me to clarify my sentiments. The example of context usage given in the MPI draft documentation appears to explicitly assume the existance of threads, and here contexts appear to be being used as threads. > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. I could strongly support almost all of the above. [The part I wouldn't support is a "tag" as I will explain below.] We do our work on the nCUBE, which, as you may know, has a message passing interface essentially equivalent to that proposed by MPI. We ran into the exact problem quoted above: it was difficult to write libraries that we're guaranteed not to conflict in their usage of message types. The usage of the word "contexts" in the above is equivalent to the usage of the word "ports" amongst operating systems programmers. Because "port" is slightly less overloaded than "context", I prefer the word "port". Note that ports are sufficiently powerful that they subsume the concepts of "contexts", "pids", and "message types" or "tags" as used by MPI. >The only expression stated so far about the difference between tags and >contexts I had gleaned was that context should now be wildcardable (IE no >-1 for All contexts) while a receive ALL or some other MASK variant would >be allowed on tags. I agree that there is little difference between tags and contexts. This helps explain why ports are so good at subsuming both contexts and tags. > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) I don't particularly understand the above, probably because I haven't been listening in on enough of the discussion. I think I understand what you mean by stack-based management, and we do use the word "registration"... When using ports, stack-based management isn't an issue for the same reasons you don't manage pids and message types using stack based management. For example, when sending a message, instead of using the MPI push_context (contextp); send (dest, type, buf, buflen); pop_context (contextp); you use send (port, buf, buflen); In our usage, a "port" is simply a queue to which messages can be sent. The receiver can remove a message from the head of the queue. Wildcards are neither needed nor implemented, making ports more efficient. The efficiency arises from the fact that a process can have multiple ports. In MPI, you essentially implement one receive queue for each pid. If you want to use a wildcard to access messages out-of-order, you need to waste time scanning the queue. With multiple ports, however, you can set up multiple queues so that you never need to access anything other than the head of a queue. Sometimes ports do need to be "registered". In our terminology, registration is the process of publishing in a well-known location sufficient information about a "port" created by one process that another process can send messages to that port. This is a very high level action that is almost outside the scope of MPI. We use a nameserver for this purpose. (The nameserver is accessed via a well-known port, i.e. a port whose bit-pattern is a well-known constant.) [Do note that we require that the bit-pattern representing a port be allowed to be transmitted between to processes without the operating system needing to know that the bit-pattern represents a port. This means that accessing the nameserver is not required in order to use a port.] For groups, we have thought about extending the concept of a port to an array-port. The simple implementation of this allows each process in a group to create a port. These ports are all sent to a central location which creates an array of ports and broadcasts the array to all ports in the array. The complex memory efficient implementation may require processes of a group to be created in a special fashion. Basically, it allows the address of a port for a specific process in a group to be easily computed by another process given the port of the first process in the group and the index of the target process in the group. -- Chuck From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 11:38:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 19 Jan 93 11:38:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21908; Tue, 19 Jan 93 11:38:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 11:37:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21879; Tue, 19 Jan 93 11:37:21 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13277 (5.65c/IDA-1.4.4); Tue, 19 Jan 1993 11:36:49 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA27411; Tue, 19 Jan 93 16:36:38 GMT Date: Tue, 19 Jan 93 16:36:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301191636.AA27411@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05440; Tue, 19 Jan 93 16:35:18 GMT To: csimmons@us.oracle.com Cc: mpi-pt2pt@cs.utk.edu, mpi-context@cs.utk.edu In-Reply-To: Charles Simmons's message of Sat, 16 Jan 93 02:21:14 PST <9301161021.AA01226@jewel.us.oracle.com> Subject: mpi Content-Length: 3447 To: mpi-pt2pt@meiko.co.uk, mpi-context.@meiko.co.uk (Apologies to those who get it twice) Gentlepeople, Chuck Simmons has raised some interesting issues, with which I have a great deal of sympathy. The original messaging model which we implemented on our machines (and which we still support, and expect to continue to support) is in exactly the vein which Chuck is asking for (and has been used for an Oracle port !). In particular CSN supports 1) multiple end points for communication in a single process (know as "transports") 2) No tagging of messages 3) No system buffering, but the ability for users to queue multiple non-blocking receives on a transport, thus providing the buffering they require. 4) Send and receive by struct iovec. (As Chuck observes, the implementation ends up building an iovec (on the stack) for the simpler forms) 5) Both blocking and non-blocking tests for I/O completion. (our test actually has a timeout value). 6) The ability to pass transport addresses around the machine without the system being involved. 7) A standard name server service to associate textual names with transport addresses. If people are really interested then I can probably send the man pages. (Though this is not the main point of this note). HOWEVER (and this is one of the points) although this system had clean, specified semantics and a fast implementation (on our hardware), it hasn't helped to sell machines, and we have now produced an NX style interface as an alternative. I think that the problem with popularising such an interface among users is that it appears to do less for them than a model with implicit buffering, and is therefore harder to start to use. The fact that it allows them greater control and can ultimately produce higher performance is not their immediate concern and does not therefore count for much. Many FORTRAN programmers in particular to do not wish to be concerned with "system programming issues" like buffer management. I would certainly like MPI to be able to support such an interface (so that we can achieve the higher performance it offers, and also make MPI applicable in non-scientific application areas). (Though those who were present in Dallas will have noted that I wasn't trying to push such an approach there, mainly because I think it's a lost cause. NX style is what we have to live with...) (Second point coming up...) However it crosses my mind that there may be some potential for embedding such an interface within the MPI model. (This is a vague thought, but I'm throwing it out so others can think about it as well). Observe that 1) Marc's persistent communication descriptors seem to remove the tag matching requirement (though they're still rather vague !) 2) One way of viewing contexts would be to implement a context as a particular queue of messages (or in other words an entirely separate communication end point) within a process. (The implications of this for contexts would be a) contexts must be declared before use b) their number may be limited c) they should be freed after use ) The combination of the two things could then come somewhere near to what Chuck is asking for... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:04:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21335; Tue, 19 Jan 93 13:04:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25741; Tue, 19 Jan 93 13:04:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:04:20 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25733; Tue, 19 Jan 93 13:04:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26371; Tue, 19 Jan 93 18:04:14 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19328; Tue, 19 Jan 93 11:03:13 MST Date: Tue, 19 Jan 93 11:03:13 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191803.AA19328@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal Tony Skjellum pointed out that Zipcode has been successfully using "invoices" that work in the same way as "capsules" for several years. Tony's comments were not sent to the entire mailing list and are repeated below, with his permission: > Please read about our message packing in the following uuencoded, > compressed postscript file. We already have created this technology. > (we've omitted the postscript file) > > We are able to deal effectively with fortran packing and unpacking > of messages, and 'pack+send' and 'unpack+receive' are inherently > more optimizable on some systems. > > - Tony Chuck Simmons points out that: > ... > Standards bodies work best when they standardize existing work, and work > worst when they attempt to design. > ... > > Cheers, Chuck Simmons We think that the success of "invoices" in Zipcode supports the argument that the "Message Capsule" proposal is an attempt to standardize existing work. (BTW, we aren't really that satisfied with the term "capsule". Another term may be better.) Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:13:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21564; Tue, 19 Jan 93 13:13:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26145; Tue, 19 Jan 93 13:12:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:12:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26131; Tue, 19 Jan 93 13:12:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26408; Tue, 19 Jan 93 18:12:52 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19336; Tue, 19 Jan 93 11:11:50 MST Date: Tue, 19 Jan 93 11:11:50 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191811.AA19336@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal The following is excerpts from an email discussion we have been having with Tony Skjellum regarding message "capsules" or "invoices". We have forwarded it with Tony's permission. Our comments are marked with ">>" and Tony's are marked with ">". Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov > > Please pass along my comments. Invoices are important, in that they > permit additional possible optimizations. The more the user says > "what" and not "how," to accomplish transfer, the more opportunity for > faster performance. Invoices make Fortran-based message passing much > nicer, as well as providing more sophisticated packing/unpacking for C > people. > >> Several people (including us) were uneasy with the "hidden state" associated >> with stack-based context management. Any particular reason for using stacks >> in Zipcode? > Stacks are a detail of conext handling. They are not essential. > - Tony From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 18:04:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23239; Tue, 26 Jan 93 18:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26207; Tue, 26 Jan 93 18:02:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 18:02:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26199; Tue, 26 Jan 93 18:02:53 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.9) id AA16157; Tue, 26 Jan 93 17:02:49 CST Received: by teak18.cray.com id AA29628; 4.1/CRI-5.6; Tue, 26 Jan 93 17:02:48 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301262302.AA29628@teak18.cray.com> Subject: MPI task identifiers To: mpi-pt2pt@cs.utk.edu Date: Tue, 26 Jan 93 17:02:44 CST X-Mailer: ELM [version 2.3 PL11b-CRI] At the meeting in Dallas, I asked a question about how tasks were identified, referring (for example) to the argument that one would pass to one of the variants of send. I was surprised to hear that the October draft specified this, and that it was defined as sequential integers starting at 0. I've scanned through the document since then, and didn't notice where this was stated. (The only indication I saw to such identifiers was an example included as "Table 1" on page 9, which listed "PID"s of 0-7.) Each place I'd expect to see a definition, says only "the ID number of the process..." and leaves it as that. But I'm willing to accept that it is there and that I just missed it (or the authors forgot to print it ;-). In any case, I'd like to propose that the final spec should *not* define the internals of task identifiers. Please consider this as such a proposal. I'd appreciate hearing any comments. Assuming the October spec does describe what a regular task identifier is, I'd like someone to point out where and to hear reasons why it is important to dictate a 0..n ranges for task identifiers. Thanks for your time. - Peter Rigsbee par@cray.com PROPOSAL -- OPAQUE TASK IDENTIFIERS 1. Proposal TASK IDENTIFIER All tasks in MPI are identified by unique integer identifiers that are assigned by the system upon which MPI is implemented. The MPI spec does not define the contents of these identifiers and users writing portable applications should not make any decisions about nor perform any operations based on their contents. 2. Discussion a. The major benefit of a 0..n range of identifiers is that users on many systems (most notably those supporting one process per CPU) can write efficient applications without having to deal with groups or virtual topologies. If the relationship between process identifier and process location is well-defined for a particular system, codes can operate directly on identifiers (send to "myid()+1", for example, and have the data sent to a nearest neighbor CPU). b. The drawback is that the above advantage leads programmers away from writing efficient, portable code. A code written using the convenient method above may work great on a few systems, acceptably on another set of systems, and horribly on a third set. It should get the correct answers in each case, but, for most people writing MPI codes, horrible performance would be considered a bug. (This of course assumes that the MPI specs for groups and virtual topologies will allow for efficient, portable applications. If not, we need to fix those specs, and not simply introduce a faster alternative.) c. For an MPP system with one process per CPU, a 0..n assignment is easy and natural. For almost any other system, it will be more complex. For a network implementation, it may require the use of a "process-identifier server" or some sort of globally shared "next identifier" variable. These can be very costly to implement. It also raises ugly questions about where and how to map these identifiers to the information actually locating the task. d. In contrast, the proposal allows an implementation to use the most appropriate and efficient method. It may simply be a Unix PID. It could be a range of 0..n integers. Or it could be a complex encoding of host/PID information that would allow a network implementation to get better performance. It's up to the implementation. e. The MPI spec does not address the fact that some implementations may be provided on top of systems allowing dynamic creation and termination of tasks. A 0..n requirement assumes a static system. What happens when task goes away? There is no longer a sequential list from 0..n, so does this implemenation still adhere to the MPI spec? (If so, one could argue that the 0..n spec is then just a red herring, since one could assign any numbers at all, and claim simply that all missing numbers represented tasks that never started.) With the proposal, the answer is simple -- that identifier no longer represents a task, and the implementation still adheres to the spec. f. The above proposal does not prohibit implementations from assigning 0..n identifiers, or for users of such implementations relying on this assignment. But it makes it clear that such reliance is non-standard and non-portable. 3. Summary The above proposal has a number of benefits, particularly when compared to the 0..n alternative: - allows implementations to use the most appropriate and/or efficient method for assigning task identifiers (including 0..n) - encourages portable programming that does not rely on operations on task identifiers - no confusing semantics for MPI implementations built on top of dynamic systems From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 19:25:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23808; Tue, 26 Jan 93 19:25:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29304; Tue, 26 Jan 93 19:24:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 19:24:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29296; Tue, 26 Jan 93 19:24:13 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05866; Tue, 26 Jan 93 18:24:11 CST Date: Tue, 26 Jan 93 18:24:11 CST From: Tony Skjellum Message-Id: <9301270024.AA05866@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: MPI task identifiers Concerning Peter's proposal... It may have been a mistake to simplify the naming of processes to a single integer when MPI was defined. Commonly, systems like RK, and so on had used pairs of integers on multicomputers and networks before that; the two numbers did have semantic content: the first number, a node number, connoted the location of the process. The second, a PID, could provide an implementation-dependent way to name a process on a processor. Hence, at the low level, specification of WHERE processes ran was permitted, but it was not necessary to restrict how they were named (beyond it being an integer quantity) on a processor. At a low level, it is convenient to be able to control process placement. When processes are created, the user could specify the node and PID on some systems, with either or both picked by the system on some implementations. Such composite information could then be used as part of message passing. nCUBE, for instance, specifies the PID's but does not restrict the locations (nodes are selected by user). The Mosaic system will generally place processes at will, and a message-passing interface for it will have neither option. For scalable spawning of large numbers of identical processes, some mechanism of this form is needed (user specifies collection of process locations, gets back identifier describing these as a group; eg, a common PID). In early systems, and arguably still, it is important to provide low-level control over process placement. Arguably, this need not be done in the message passing interface, but it must be done portably somewhere in the system. Otherwise, it will be difficult to provide guarantees of performance behavior (ie, force one process per processor, if desired; clump processes together to get overlapping capability when desired, etc). Our Zipcode system uses {node,pid} in addressee lists, but these are not used by the programmer directly, in most message classes (eg, 3D logical process grids) but are hidden with further mappings to application-relevant names for processes. In fact, we would like to generalize these addressee lists to include other information besides this naming, but that remains a future-release feature. It does not matter so much to us what convention names processes, but it is interesting to have access to semantic information about processes named by the system (eg, accessors to location, etc). Such information would be used when providing services, like Zipcode, that are higher level. The user might not want them at all for direct use. Forbidding a user code or application layer like Zipcode to take advantage of architecture (albeit transparently to the user code) is a disadvantage. Typically, process placement remains an aspect of architecture we need to control. Just some thoughts... - Tony Skjellum From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 08:37:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02588; Wed, 27 Jan 93 08:37:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01030; Wed, 27 Jan 93 08:37:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:37:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01022; Wed, 27 Jan 93 08:37:16 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17101; Wed, 27 Jan 1993 08:37:15 -0500 Date: Wed, 27 Jan 1993 08:37:15 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301271337.AA17101@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Task Indentifers I assume that "task" in Peter's proposal are the same as "processes" elsewhere in the MPI discussion. I think there is some confusion over the terms "PID" and "rank". If PIDS do not convey any information to the application then as Peter says: "users writing portable applications should not make any decisions about nor perform any operations based on [task identifiers]." In fact there's almost no point in having task identifiers accessible to an application. Instead applications typically use what was referred to at the Dallas meeting as the "rank", which is a one-to-one mapping of PIDs to the integers 0,1,2,...n-1. The user needs ranks so s/he can say things like, if ( myID() .EQ. 0 ) output(data) PID (or task ID if you like) and rank are synonymous in MPI1 since we could see no reason for distinguishing them. Note that "rank" is essentially an arbitrary enumeration of processes, and contains no information about relative placement of processes on physical processors. A ranking is the same as a 1-D topology. MPI does not currently address how a topology is mapped to physical processors, in fact the whole topic of process creation has so far been avoided. I guess my main question with Peter's proposal is : "If an application shouldn't make any decisions about nor perform any operations based on task IDs, then why does an application even need to know about task IDs?" From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 09:00:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03235; Wed, 27 Jan 93 09:00:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01779; Wed, 27 Jan 93 08:59:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:59:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01771; Wed, 27 Jan 93 08:59:40 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00815 (5.65c/IDA-1.4.4 for ); Wed, 27 Jan 1993 08:59:36 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA13095; Wed, 27 Jan 93 13:59:32 GMT Date: Wed, 27 Jan 93 13:59:32 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301271359.AA13095@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02254; Wed, 27 Jan 93 13:58:07 GMT To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: David Walker's message of Wed, 27 Jan 1993 08:37:15 -0500 <9301271337.AA17101@rios2.epm.ornl.gov> Subject: Opaque Task Identifers Content-Length: 1945 I tend to agree with David here. It is clear that the underlying system level identification of a process can potentially be arbitrarily complex, and will almost certainly require at least a (node,process within node) tuple. However, this is not the way the user wants to think about her program. She wants to think about a set of processors with an obvious mapping onto the positive integers. (Fortran programmers would probably prefer that to read "strictly positive integers" ...). At the user level we shouldn't be exposing the internals of the system's process mapping scheme, since it will likely be different on each machine. (On a net doubtless there will be whole ip addresses kicking around, and so on). I would suggest that we let the user think about a simple enumeration of processors, but stipulate that this is a random enumeration (in other words there is no guarantee whatever that an N to N+1 communication is faster than an N to N+x communication). The system can then use a simple table lookup to translate from the user's process number into the system address. Most of the time this will be necessary anyway (on all systems which need more than this simple numbering scheme), and in any case it's only a single (unsigned) compare followed by one store access. If the user needs to have guarantees of "nearness", then she should use the virtual topologies mechanism to obtain a good process enumeration. If we were feeling mean, then we'd add an implementation recommendation that the intial mapping should be random, so that people get an incentive to use the virtual topology mechanisms even to implement a straight line of processes. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 10:29:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05985; Wed, 27 Jan 93 10:29:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05149; Wed, 27 Jan 93 10:28:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 10:28:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05141; Wed, 27 Jan 93 10:28:50 -0500 Message-Id: <9301271528.AA05141@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 7863; Wed, 27 Jan 93 10:28:42 EST Date: Wed, 27 Jan 93 09:50:23 EST From: "Shlomo Kipnis ((914) 945-1281)" To: mpi-pt2pt@cs.utk.edu Subject: Process enumeration The main goal of the MPI standard is the portability of applications between different machines. Linear ordering of process (task) ids in the range 0..n is the simplest and most portable numbering scheme I can think of. As David and Jim state, this numbering scheme is used by many applications, and most machines provide a virtual domain of processes in the range 0..n to be used. Any implementation of MPI, obviously, need to translate this numbering scheme to the physical process/task/thread ids used in the underlying system. However, a portable user's application cannot rely on such a physical numbering scheme. The performance of an application may be very different on different machines, but at least an application that uses MPI is guaranteed to run on different machines (unlike the state of the art today). An interesting feature of the "process groups" idea is that pids in a process group can always be addressed in the range of 0..n using the ranks of processes within the group. By using process groups, one can avoid the issue of noncontiguous pids range which may be caused by particular partitions of the machine, processes leaving or joining an application, running several application codes concurrently under the same job, etc... As currently defined, process groups do not address the issue of an application's topology. However, topologies can be defined on top of process groups. (I hope the topology subcommittee discusses these issues.) To summarize, I think that process ids in the range 0..n are the right approach for ensuring portability of applications. This scheme also addresses the virtual organization of processes in a process group. The virtual-to-physical translation of pids is done in many systems and is typically very fast. Shlomo Kipnis IBM T.J Watson Research Center Yorktown Heights, NY 10598 (914) 945-1281 kipnis@watson.ibm.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:12:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08710; Thu, 28 Jan 93 09:12:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02034; Thu, 28 Jan 93 09:11:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:11:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02026; Thu, 28 Jan 93 09:11:42 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA03474; Thu, 28 Jan 93 09:11:39 -0500 Date: Thu, 28 Jan 93 09:11:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301281411.AA03474@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: My two cents worth on process numbering. >To summarize, I think that process ids in the range 0..n are the right >approach for ensuring portability of applications. With the ability to have multiple groups each with processes numbered 0...p then no process can be uniquely identified by a single integer [0,1,...n]. The above process naming scheme will require some group context information down in the point to point routines. It can be explicit as in a (group_id, rank) pair which I prefer because it makes codes more readable and easier to debug, or it can be implicit as in the original MPI draft. Peter Rigsbee is correct in noting that on large MPP systems looking up this mapping of 0-p to real process location and id can be expensive in both time and memory. If the high priests want to be efficient and go fast then Peter's suggestion of having a more sophisticated task naming should be considered seriously. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:18:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08787; Thu, 28 Jan 93 09:18:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02247; Thu, 28 Jan 93 09:18:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:18:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02239; Thu, 28 Jan 93 09:17:56 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA24455 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 15:16:56 +0100 Received: by f1neuman.gmd.de id AA15219; Thu, 28 Jan 1993 15:16:01 +0100 Date: Thu, 28 Jan 1993 15:16:01 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9301281416.AA15219@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: MPI task identifiers Cc: gmap10@f1neuman.gmd.de During the last two days there was an interesting discussion about the naming of processes in MPI. I just want to throw some thoughts into the discussion. If we number the processes in a group from 0 to n-1, I see two benefits: 1. The applications programmer immediately knows how to address the other processes in the same group, instead of first having to look up some strange id numbers. Sure, the translation to the underlying (h/w) or network addresses has to be done somewhere, but it's not necessarily the applications programmer who wants to do this. 2. The 0,...,n-1 ranking of processes provides a natural order for collective communications, like gather. Without nicely ordered id numbers, almost every reasonable MPI program would have to use the virtual topologies. If you look at the proposal of the PTOP subcommittee, you will find no assumption about the definition of the process identifiers in a group. All ordering and addressing of processes is provided by the topologies. So, as a proponent of virtual topologies I could be happy with that solution. At the Dallas meeting somebody already suggested that there be always a topology associated with a process group. At group creation this would be just a linear ordering. Since topologies can be overwritten, this default could be changed by the user at any time. Note that this would not mean that the pid numbers are arranged as 0,..,n-1. A process would use the topology look-up functions to get the pids of its neighbors, whereas the pid numbers could have any value (as Peter Ringsbee proposes). I think we have to choose between: a. We force programmers to use the topology functions in virtually all applications. The pid numbers can then have any values. The default topology of a group is a linear ordering. The mapping of processes to processors causes some (usually small) overhead at group creation. b. The pid numbers are numbered as 0,..,n-1. This does not mean the processes are mapped in a kind of nearest-neighbor fashion to the processors. It is rather a (random) enumeration of the processes in a group, in order to get benefits 1. and 2. (see above). If one wants to have a good mapping, one should use the virtual topology functions (which I recommend, of course). On the other hand, this introduces another level of address indirection. I tend towards alternative b., although there are some tricky problems. For example: when a topology is assigned to a group, does this then change the pid numbers? If not, then the buffer structure in collective operations will not be aligned with the topology. On the other hand, changing the pid number of a process looks ugly to me. Perhaps it helps if we avoid the word "pid" alltogether, and only refer to the "rank" of a process, as David Walker suggests. I suggest that at the next Dallas meeting we have a session in which the subcommittees for collective communication and process topologies meet together. To me this seems to be the ideal forum where we can try to solve the problem. Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:57:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09245; Thu, 28 Jan 93 09:57:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04162; Thu, 28 Jan 93 09:57:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:56:59 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04154; Thu, 28 Jan 93 09:56:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA11324 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 09:56:52 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA25589; Thu, 28 Jan 93 14:56:47 GMT Date: Thu, 28 Jan 93 14:56:47 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301281456.AA25589@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02620; Thu, 28 Jan 93 14:55:18 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 28 Jan 93 09:11:39 -0500 <9301281411.AA03474@msr.EPM.ORNL.GOV> Subject: Re Al's two cents worth on process numbering. Content-Length: 2327 (Come on Al give us the whole $1 worth !) Al asserts that > Peter Rigsbee is correct in noting that on large MPP systems looking > up this mapping of 0-p to real process location and id can be > expensive in both time and memory. I disagree that this is necessarily expensive, for the following reasons :- 1) At some level the system has to know the lowest level address, otherwise it cannot send the data. (I know this could be a route, rather than an absolute address, but it comes to the same thing) So it's a cost you have to pay somewhere. 2) In the worst case the cost is constant in time and linear in number of nodes in memory. (Just index into an array of low level addresses). How bad is this really ? Suppose you have a 16K node machine with 128 bit low level addresses (4 32bit words) this occupies 256K bytes, which doesn't seem so bad. 3) You can trade off the store requirement against a larger time cost. Rather than holding all of the destination addresses you implement an address cache (using standard symbol table technology), and fill in the addresses as you use them. (Of course you need a distributed scalable lookup to find them the first time). This then requires store which depends on the working set of targets for communication, and whatever CPU time is taken in the lookup. (Should be logarithmic in number of processors if you use a tree). The issue I'd like to clarify here is what is actually being asked for by Peter ? Is he asking that the user deal with genuine low level addresses (which are potentially of arbitrary size, and certainly different on different vendors machines) ? or Is he asking that users deal with integer identifiers, but these are opaque in the sense that modifying then makes no sense ? The former seems extremely unattractive (especially in Fortran !), and if it's the latter, then you still have to do a translation somewhere to get to the real low level address, so why not let the user have what they really wanted (i.e. 0..N) ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29228; Fri, 5 Feb 93 00:33:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27483; Fri, 5 Feb 93 00:31:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:31:54 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27473; Fri, 5 Feb 93 00:31:52 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA09343; Fri, 5 Feb 93 00:31:59 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002919.13027; Fri, 5 Feb 1993 00:29:19 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09357; Thu, 4 Feb 1993 19:11:45 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05200; Thu, 4 Feb 93 19:11:44 -0600 Date: Thu, 4 Feb 93 19:11:44 -0600 Message-Id: <9302050111.AA05200@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Composition. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 What is the meaning of these sequential compositions? 1) two non-blocking sends to the same destination? 2) a non-blocking send and a blocking send to the same destination? 3) a non-blocking send and a broadcast? 4) two non-blocking receives? 5) a non-blocking receive and a blocking receive? and so on ... I suspect different machine vendors may have different views. Steven ps. mpi-formal members are refered to mpi-pt2pt. From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29230; Fri, 5 Feb 93 00:33:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27500; Fri, 5 Feb 93 00:32:03 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:32:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27492; Fri, 5 Feb 93 00:31:59 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AB09405; Fri, 5 Feb 93 00:32:06 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002916.13009; Fri, 5 Feb 1993 00:29:16 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09184; Thu, 4 Feb 1993 18:56:01 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05164; Thu, 4 Feb 93 18:56:00 -0600 Date: Thu, 4 Feb 93 18:56:00 -0600 Message-Id: <9302050056.AA05164@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Concerns. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I have to agree with Steve Otto and confess more than a little anxiety at this end. It seems to me that we are much too concerned with being all things to all men and I am absolutely convinced that a mix and match philosophy is a recipe for disaster: not that it can't be implemented or specified but purely from the usability point of view. We should be able to provide more wizardly and wise power by providing less not more! We would do better to ensure that the issues we address are those that we all currently understand well or that we have time to defer to assure ourselves that new forms or compositions are at least well concieved. The latter is impossible in the six months we have allotted ourselves. If we really want people to use MPI it has to be straight forward and at least no more complex than existing systems. If we can't decide on a *concise* set of wisely restricted forms for send and recieve, if you really believe we have to provide all forms, then I suggest this effort may be premature. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 05:48:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13673; Fri, 5 Feb 93 05:48:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18966; Fri, 5 Feb 93 05:44:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 05:44:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18958; Fri, 5 Feb 93 05:44:20 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24268; Fri, 5 Feb 93 05:44:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 054326.7477; Fri, 5 Feb 1993 05:43:26 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA16580; Thu, 4 Feb 1993 23:45:25 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06410; Thu, 4 Feb 93 23:45:23 -0600 Date: Thu, 4 Feb 93 23:45:23 -0600 Message-Id: <9302050545.AA06410@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 It will probably come as no surprise that I, for one, have no fear of being politically incorrect in these matters ;-) I am not afraid to say that I think at this stage it would be a very good idea to define only process composition and blocking send/recv. The rest comes - more or less - as a construction of the other two. Process ids are simply a many to one channel descriptor - implementable by a single process, non-synchronized communication is simply a process creation, broadcast is a composition of N sends with N receives, and such an interface can be implemented by all the available technologies. In addition this simplification would greatly aid the task of the poor programmer. Where is the innovation in this forum? It seems that we are pandering to the lowest common denominator. Has noone learnt anything in the past 7/8 years? Otherwise we should simply take an existing system (PVM for example, though there are many well designed MP systems) and adopt it as the accepted standard, work from that point and be done with it. The interface efforts of the Unix community (e.g., DCE and sockets) may well out run this one. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 06:05:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17447; Fri, 5 Feb 93 06:05:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20412; Fri, 5 Feb 93 06:04:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 06:04:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20403; Fri, 5 Feb 93 06:04:45 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA27476 (5.65c/IDA-1.4.4); Fri, 5 Feb 1993 06:04:40 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16231; Fri, 5 Feb 93 11:04:37 GMT Date: Fri, 5 Feb 93 11:04:36 GMT From: jim@meiko.co.uk (Paul Kelly) Message-Id: <9302051104.AA16231@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA06680; Fri, 5 Feb 93 11:02:53 GMT To: zenith@kai.com Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Steven Ericsson Zenith's message of Thu, 4 Feb 93 23:45:23 -0600 <9302050545.AA06410@brisk.kai.com> Subject: Be brave. Be sure. Content-Length: 896 Steve, > I am not afraid to say that I think at this stage it would be a very > good idea to define only process composition and blocking send/recv. > The rest comes - more or less - as a construction of the other two. > Process ids are simply a many to one channel descriptor - > implementable by a single process, non-synchronized communication is > simply a process creation, broadcast is a composition of N sends with > N receives, and such an interface can be implemented by all the > available technologies. You just described occam. One thing I have learnt from the last 10 years is that that's NOT ENOUGH. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 08:24:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02543; Fri, 5 Feb 93 08:24:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27610; Fri, 5 Feb 93 08:23:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 08:23:33 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27602; Fri, 5 Feb 93 08:23:31 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA19627; Fri, 5 Feb 93 08:17:03 EST Date: Fri, 5 Feb 93 08:17:03 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302051317.AA19627@cs.wmich.edu> To: mpi-pt2pt@CS.UTK.EDU hi; A good offline note to me from > From: Alan Mainwaring together with some other recent posts, indicates that we are using at least two valid but very different meanings for the term "overlappping communication and computation". -------- 1) Overlapping IO and computation means transfers from the communication processor to and from the node's computation processor memory are carried out by the node's memory system system while a computation is actively running on the node. 2) Overlapping IO and computation means receives and sends requests in a process may be pending while a computation is actively running in a process. -------- Note 2) does not require 1) , some ways 2) could (has) been provided: 2) could be cone by 1) 2) could be done by the OS while the unknowing computation process is interupted and placed in a wait state by the OS when it wants to prefrom a transfer. 2) could be done when the computation process makes any system call. -------- I believe MPI must have 2), that is non-blocking sends and receives. I also think it would pass any straw vote by the widest of margins, I do not have any real feeling about how important 1) currently is. I would expect 1) could become an issue if communications become fast enough to really support distributed virtual memory (that is, commmunication speed approaches computation speed). Then much finer grained parallelism can be applied. It would seem that most interfaces that support direct transfers into memory, rather than a buffer (iovec, stride etc.), could logically also well support 2), if the hardware provides it. It would be best if the interface specifed could use 2) if available, without extra complication. cheers - john From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 13:13:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09267; Fri, 5 Feb 93 13:13:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16600; Fri, 5 Feb 93 13:09:51 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 13:09:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16588; Fri, 5 Feb 93 13:09:48 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA25537; Fri, 5 Feb 93 13:09:50 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 130847.19757; Fri, 5 Feb 1993 13:08:47 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA04173; Fri, 5 Feb 1993 10:16:24 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA07622; Fri, 5 Feb 93 10:16:22 -0600 Date: Fri, 5 Feb 93 10:16:22 -0600 Message-Id: <9302051616.AA07622@brisk.kai.com> To: jim@meiko.co.uk Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Paul Kelly's message of Fri, 5 Feb 93 13:20:43 GMT <9302051320.AA16757@hub.meiko.co.uk> Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Thanks Jim (or is it Paul) for the name correction (via private mail)... I know what I'm suggesting - and yes, though I hate to admit it, it is the Occam model more or less. Though you shouldn't allow your distaste for the language to deflect you from the qualities of it's communication model. The inadequacies of that language are many but when it comes to the communication model (CSP) it is fundamental. All I'm suggesting here is that we do not make the other mistake, that of providing so much that it is neither managable by us or our intended audience. Either we are prepared to provide something better than currently exists based on a good understanding of our collective experience with mature concepts or we should start from an existing system, with all its flaws, and work from there. This is not a forum for invention. I should qualify my comments to reiterate that in fact I am looking to this forum to provide a consistent interface for low level implementation in operating systems and code generation. I never want any of my general purpose users using message passing as a means to program machines. And if you gave me just processes and blocking send and receive that *would* be enough. As someone who has written many message passing programs (using Express and other libraries, as well as Occam and CSP), and made it my business over the years to talk to others who have, I have to tell you that the fewer well conceived choices you give me the happier I am and, in my experience, the user is. I just don't believe those people who say we have to provide every form - it is counter to all my experience with users (no doubt, there are users here who will insist on being provided with every variant ;-) - but you will have seen that manifestation before. ). One way to look for evidence to support or denounce this is to undertake an investigation. Find an array of *applications*, not systems tools etc. but "real" applications, and identify the subset of functionality used. My bet is that you will discover the subset is small, the place where non-blocking communication is used is on machines where the process model implementation is inadequate, and that code that does not fit into my simple model is not readily portable. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 14:07:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10311; Fri, 5 Feb 93 14:07:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20802; Fri, 5 Feb 93 14:07:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 14:07:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20794; Fri, 5 Feb 93 14:07:03 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA11334; Fri, 5 Feb 93 12:06:57 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nKYO9-0016ZKC; Fri, 5 Feb 93 12:06 MST Message-Id: Date: Fri, 5 Feb 93 12:06 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Be brave. Be sure. Steven: While I am impressed that you have enough influence over your users to convince them that they need not see the messaging environment themselves, I must confess that I am not so fortunate. My users are developing their codes in C, F77, and C++. The current trend here at Sandia is towards C++. Our REAL codes (REAL code is one that we make money off of, hence very important :-) are done by those that are still waiting for compilers to do everything. But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. As for overlapped messaging, I dare say that EVERY code here at Sandia makes use of overlapped messaging. At the very least, the programmers want to have multiple I/O channels active simultaneously, which is at least possible on our two 1024-processor nCUBE 2 hypercubes. Furthermore, it is clear that my users are already digging into what possibilities for performance enhancements await them in our new Paragon. These are general users! Not just the wizards. Again, the "food chain" is quite clear. We do want simply understood features for the "novice" or "don't care" people. However, we must have what the experienced general users want. Furthermore, we must have the "buy-in" of the wizards. To make this more clear, I use the term wizard to refer to application types doing real, dollar/pound income-oriented work. That is, they are not of the CS lunatic fringe type, but are real-world applications people. All of this is just to state that the standard should be primarily for human use, not for compilers only. As has been voiced by others, I strongly suspect that overlapped I/O and computation is a feature that general users will (AND DO) want. Furthermore, I believe that "our collective experience with mature concepts" can tackle this problem in a very successful manner. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 17:00:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13516; Fri, 5 Feb 93 17:00:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01026; Fri, 5 Feb 93 16:59:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 16:59:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01011; Fri, 5 Feb 93 16:59:03 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00846; Fri, 5 Feb 93 15:58:52 CST Date: Fri, 5 Feb 93 15:58:52 CST From: Tony Skjellum Message-Id: <9302052158.AA00846@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, srwheat@cs.sandia.gov Subject: Re: Be brave. Be sure. To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 18:13:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14881; Fri, 5 Feb 93 18:13:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04250; Fri, 5 Feb 93 18:13:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 18:13:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04238; Fri, 5 Feb 93 18:13:15 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24078; Fri, 5 Feb 93 18:13:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 181248.24121; Fri, 5 Feb 1993 18:12:48 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA01338; Fri, 5 Feb 1993 17:10:16 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08601; Fri, 5 Feb 93 17:10:15 -0600 Date: Fri, 5 Feb 93 17:10:15 -0600 Message-Id: <9302052310.AA08601@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: In-Reply-To: Steven Ericsson Zenith's message of Fri, 5 Feb 93 10:16:22 EST Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 From: srwheat@cs.sandia.gov (Stephen R. Wheat) ... But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. The fact that your users are programming using message passing suggests that they are certainly also prepared to use assembler when necessary. The reason they use MP is the same reason they use assembler at times: there is no higher level way currently to achieve what they want with the performance they require on the machines they are using. ;-) Evenso, when I've asked people (physicists and so on) with some experience programming such machines this question they have confirmed that they use MP as reluctantly as they use assembler - it's just that sometimes it's unavoidable. Some of those people are at Sandia. I certainly would not want to deprive them of their lively hood and there are exceptions - of course. Some people really enjoy the challenge of writing MP code, just as they enjoy the challenge of writing assembler - me too at times ;-) But the pro's and con's of message passing as a programming model, and the alternatives, are not the issues here. While we may have different motivations we do all understand the importance of MP at some level. Certainly it should be usable by "users" in the absence of anything better. In fact I'd contend that my recent calls serve those that do have to use it well - wizard or not. I am not calling for us to take power away from users - on the contrary - I'm calling for greater utility and usability through simplicity. I am calling for us to focus on things we understand well and to avoid invention. I do not feel that such simplicity will fail to serve wizards - your implication is that wizards are only attracted to complexity: this surely is not true. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. No argument here - but then this isn't an objective of MPI. As for overlapped messaging, I dare say that EVERY code here ... I am not saying that non-synchronizing send is a "bad thing" - I am saying there is a simpler way of looking at these things. Define the fundamentals and work from there. We have been arguing about the nuances of buffer behavior without getting a real feel for our base objectives. You can build all you want - if you really want it - with fundamental and well understood components - it makes a great opportunity for third party software vendors like Parasoft and DSL. Buffering characteristics are likely to remain a choice based on hardware design and we should leave such things alone in our interface design. Now I understand that most machine vendors have a variety of approaches - and that is exactly why we should should be careful. But I cannot believe the solution for MPI is the union of those approaches - though it may be some component of the intersection. Isn't our objective to provide a Message Passing Interface that will allow people who want to write portable (or transportable) MP code to do so? And to provide for compiler writers and architects - people like me - who are just looking for a consistent interface on MP machines? Doesn't usability have a right to be at the fore of our concerns? If this is the case then we do need some well conceived abstraction or we must adopt an existing system and refine it. When it comes right down to it, I really don't mind which. I would like though to take a program written in MPI and simply transform it into another MPI program for execution on a machine with different characteristics. What I've seen so far is going to make it very very hard to do that. I'm only likely to want to do literal translation on a machine identical to the original target - it would be nice to permit simple transformations and checking (for deadlock for example) ... wouldn't it? Incidently, in addition to my architects point of view I do still design message passing systems and program them - so I hope I qualify as a user too. ;-) Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 22:04:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Fri, 5 Feb 93 22:04:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13646; Fri, 5 Feb 93 22:04:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 22:04:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13638; Fri, 5 Feb 93 22:04:19 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Fri, 5 Feb 1993 22:04:17 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA05848; Fri, 5 Feb 1993 22:04:16 -0500 Date: Fri, 5 Feb 1993 22:04:16 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302060304.AA05848@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu Subject: compiler target (was Be brave. Be sure.) Tony says: To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. There are some compiler folks who might disagree with you. The group at Rice is working on generating message-passing, as well as the group at Syracuse, U. Maryland, and a few other places. Perhaps you could be more specific as to why you think that compilers cannot make good use of this standard? -scott berryman From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 23:21:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17474; Fri, 5 Feb 93 23:21:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15951; Fri, 5 Feb 93 23:20:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 23:20:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15940; Fri, 5 Feb 93 23:20:42 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01319; Fri, 5 Feb 93 22:20:29 CST Date: Fri, 5 Feb 93 22:20:29 CST From: Tony Skjellum Message-Id: <9302060420.AA01319@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: compiler target (was Be brave. Be sure.) Cc: bert@athena.llnl.gov, tsw@mpci.llnl.gov Yes Scott, I can be more specific. I think that MPI[1] is not interesting for compiler writers. Active messages are finer grain message passing, which have lower overhead. There is naturally a larger overhead in a full system call like a send()/recv() pair. If the compiler knows it wants to drop a single real number on a process, it could use a finer grain mechanism. Compilers would be good at cacheing limited open channel resources, such as supported at the low-level by CMMD. Thus, they could take advantage of not having to build up and tear down connections between processes, as is needed for send()/recv(). A good example of this is neighbor communication for boundary sharing in domain decomposition. They could take advantage of special properties of the network(s) of a system, that are lost when a robust send()/recv() mechanism is provided, each of which implies a library [if not system] call. The overhead of a subroutine call send()/recv() implies a minimal granularity, because stack frames must be assembled, etc. The compiler itself could be responsible for the non-deadlock properties of the communication, rather than insisting that the messaging calls provide this capability. Furthermore, compilers could make use of the "force semantics" per Intel, but still provide the user with a guarantee of correctness (ie, use the cheapest calls, but build their own appropriate protocols, but as weak as possible). Mixing communication and computation is another example of where the messaging libraries with send/recv semantics are too heavy duty, compared to active messages. A compiler might want to add two values and store them remotely. It does not need a full stack of messages, with rendezvous to accomplish this. I think the people who are building compilers to generate straight message- passing code should read the work from Berkeley and TMC on active messages (Culler et al). and build compilers that generate code using these excellent ideas. Performance will be superior. Message-passing libraries are for people who write explicit message-passing code without the benefit of compiler support. They need guarantees like non deadlock of messages, they need queues so messages are there when you look for them, and so on. Compilers can manage these things, and avoid the necessary compromises made to present robust message passing to users through send/recv. It is difficult to cope with the many sequential calls that occur in message passing code, without the need to build things like send+recv, in order to reduce overhead. Compilers could optimize away such overheads if they controlled the messaging resource directly (ie, stamp out unneeded overheads). Also, they could inline conversions for heterogeneity, and reduce the number of copies of messages by driving the network as they convert data (or just drive the network as they build parts of a message). Furthermore, they could balance the work done by sender and receiver based on the relative power of the sender and receiver to do conversions. This gets quite complex for hand-written message passing code. Active messaging can also be standardized (eg, create MPI-AM) to provide portable code to machines which are message passing, and which are hierarchically shared memory. So, one could build a model that unifies machines like the Paragon, CM-5, and Cray Tera, for the purpose of compiler code generation, though these are different machines, to be sure. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 08:54:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26068; Mon, 8 Feb 93 08:54:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20830; Mon, 8 Feb 93 08:52:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 08:52:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:25:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27616; Mon, 8 Feb 93 09:25:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Mon, 8 Feb 93 09:24:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:24:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22575; Mon, 8 Feb 93 09:24:40 -0500 Message-Id: <9302081424.AA22575@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9817; Mon, 08 Feb 93 09:24:38 EST Date: Mon, 8 Feb 93 09:24:37 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 7422; Mon, 8 Feb 1993 09:19:14 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R2) with TCP; Mon, 08 Feb 93 09:19:11 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930113) id AA24152; Mon, 8 Feb 1993 09:19:15 -0500 Date: Mon, 8 Feb 1993 09:19:15 -0500 From: snir@snir.watson.ibm.com (Marc Snir) Message-Id: <9302081419.AA24152@snir.watson.ibm.com> To: snir@yktvmv.vnet.ibm.com \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:37:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27880; Mon, 8 Feb 93 09:37:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23139; Mon, 8 Feb 93 09:37:24 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:37:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23131; Mon, 8 Feb 93 09:37:18 -0500 Message-Id: <9302081437.AA23131@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9955; Mon, 08 Feb 93 09:37:17 EST Date: Mon, 8 Feb 93 09:37:16 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Some (all?) of you may have received a truncated version of this doc at my previous attempt. Trying again. \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:12:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28591; Mon, 8 Feb 93 10:12:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24749; Mon, 8 Feb 93 10:11:20 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:11:19 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24741; Mon, 8 Feb 93 10:11:17 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25516 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:11:15 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06314; Mon, 8 Feb 93 07:06:07 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16745; Mon, 8 Feb 93 07:10:39 PST Date: Mon, 8 Feb 93 07:10:39 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081510.AA16745@lion.parasoft> To: mpi-pt2pt@cs.utk.edu To: mpi-pt2pt@cs.utk.edu Re: Buffering requirements I agree with Jim Cownie's comment that trying to figure out buffering requirements in an application is difficult for the receiver and hence the problems in creating REALLY reliable code with buffered systems. A potentially easier option is the one that Rik Littlefield mentioned - buffering at the sender instead. For most applications it is MUCH easier to compute how much is going to be sent than how much will be received. However, buffering at the sender seems to impose an implementation feature, i.e., that there be an intial message describing the actual transaction followed by a data request and finally the data itself. Is there a way around this? If so, it seems like a great idea to me. If not, is it going to be acceptable to the High Priests to have this three-hop version? Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29236; Mon, 8 Feb 93 10:36:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25820; Mon, 8 Feb 93 10:35:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:35:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25812; Mon, 8 Feb 93 10:35:43 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25693 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:35:40 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06436; Mon, 8 Feb 93 07:30:33 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16763; Mon, 8 Feb 93 07:35:05 PST Date: Mon, 8 Feb 93 07:35:05 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081535.AA16763@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: /dev/null, /dev/full? To: mpi-pt2pt@cs.utk.edu Re: /dev/null, /dev/full? Despite the fact that we still don't seem to have settled on a model for specifying process identifiers I would like to raise a different issue on this topic ..... the idea of having a special identifier for a node that doesn't actually exist. Sending a message to this destination would always succeed and return as though the data were transmitted, and reading from this node would also return immediately, without modifying the user buffer, but with a return code indicating that no error occurred. I think this offers users great simplifications in their coding. A classic example is the standard domain decomposed PDE solution. In one dimension you have to write code for the guard strip exchange such as if(I_am_on_left_edge) { send_to_right(); read_from_right(); } else if(I_am_on_right_edge) { send_to_left(); read_from_left(); } else if(there_is_more_than_one_node) { send_to_right(); read_from_right(); send_to_left(); read_from_left(); } Note that this code assumes buffered communication. If you have to code unbuffered messages then the logic becomes considerably worse since you have to deal with even and odd parity and the case of an odd number of nodes, etc..... If we have the "no node" concept then the above becomes: send_to_right(); read_from_right(); send_to_left(); read_from_left(); in all cases. Given an "exchange" or "sendrecv" function this code becomes even simpler. When dealing with virtual topologies this is invaluable because the virtual topology code can return this special value as the process I.D. and it can then be passed directly to the communication routines without having to be interpreted at all. I think this can be implemented without cost to the higher level software by making the "no node" identifier illegal for normal communication. Then the basic communication functions which, I imagine, do checking such as if( ! destination_is_legal_node ) { return ILLEGAL_NODE; } can be simply changed to if( ! destination_is_legal_node ) { if( destination_is_no_node ) { /* Pretend this was OK */ return OK; } return ILLEGAL_NODE; } without any impact on the regular code. EXTENSION: Marc Snir raised an interesting point in connection with this idea. Since it is normally used to deal with boundary conditions in the physical world why not extend the idea by allowing a read from the "no node" source to actually implement the user boundary condition either by filling an array with predefined values, or even calling a user-defined function? Again I think this could be implemented at no cost to the higher levels by simply invoking some code in the error handling for normal communication, and extending the "no node" identifier to a range of identifiers. I think in this case you might also be able to run user-supplied code, even when you have a communication co-processor, assuming that the error checking shown above occurs early enough in the sequence that it would still be in the main processor(?) I don't have any experience with this extension but it certainly seems "cute". One possible objection might be that the boundary case code is almost certainly already coded in any existing application and pushing it down into the message passing layer might be a step backwards. For applications written from scratch, however, it is a nice abstraction. Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29242; Mon, 8 Feb 93 10:36:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25849; Mon, 8 Feb 93 10:36:36 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:36:34 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25840; Mon, 8 Feb 93 10:36:33 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA20721; Mon, 8 Feb 93 08:36:26 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nLaX2-0016ZKC; Mon, 8 Feb 93 08:36 MST Message-Id: Date: Mon, 8 Feb 93 08:36 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Buffering requirements Jon asks: >> Is there a way around this? If so, it seems like a great idea >> to me. If not, is it going to be acceptable to the High Priests >> to have this three-hop version? This brings us back to the question of the "Ready Receiver". That is, should one KNOW that a matched receive has already been posted, then one should be able perform the send without having to go through the heavy duty "RTS/CTS/Data" sequence. One other point I would like to raise is that the phrase "system buffering" implies an implementation where the system provides the buffering rather than the application. I believe that one could argue that "application buffering" would be more portable, in that the application/library would depend on more basic services, such as application memory allocation to provide the buffering. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:54:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29472; Mon, 8 Feb 93 10:54:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26520; Mon, 8 Feb 93 10:53:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:53:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26512; Mon, 8 Feb 93 10:53:49 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25873 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:53:47 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06638; Mon, 8 Feb 93 07:48:40 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16875; Mon, 8 Feb 93 07:53:12 PST Date: Mon, 8 Feb 93 07:53:12 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081553.AA16875@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: System vs. Application buffers Having application level buffers certainly seems to give the user more control over the amount of memory they hve to play with but is it consistent with layered (library) packages? Assume that we have a simple three-tier library calling sequence with buffer requirements as follows User-application (Needs buffer space X) Library-1 (Needs buffer space Y) Library-2 (Needs buffer space Z) Are we then going to ask for memory (X+Y+Z) or simply max(X,Y,Z)? I suspect that if the system is buffering we'll probably get the latter. In the application buffered case you could probably have either if buffers are allocated with malloc/realloc. But if buffers are declared explicitly then X+Y+Z is probably going to be required. One option would be to have one of the arguments to the library package be its message buffer space in the same way that IMSL routines have you pass in workspace as arguments. Then you could potentially re-use an outer layer buffer. This seems ugly but would probably work. Jon From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 12:19:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01665; Mon, 8 Feb 93 12:19:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00736; Mon, 8 Feb 93 12:18:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 12:18:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00728; Mon, 8 Feb 93 12:18:11 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Mon, 8 Feb 93 09:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23874; Mon, 8 Feb 93 09:03:34 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA18935; Mon, 8 Feb 93 09:03:26 PST Date: Mon, 8 Feb 93 09:03:26 PST From: d39135@sodium.pnl.gov Subject: Re: Buffering requirements To: jwf@lion.Parasoft.COM, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9302081703.AA18935@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Jon Flower says: > I agree with Jim Cownie's comment that trying to figure out buffering > requirements in an application is difficult for the receiver and hence > the problems in creating REALLY reliable code with buffered systems. > A potentially easier option is the one that Rik Littlefield mentioned - > buffering at the sender instead. For most applications it is MUCH > easier to compute how much is going to be sent than how much will > be received. > > However, buffering at the sender seems to impose an implementation > feature, i.e., that there be an intial message describing the actual > transaction followed by a data request and finally the data itself. > > Is there a way around this? If so, it seems like a great idea > to me. If not, is it going to be acceptable to the High Priests > to have this three-hop version? Let's see if I can remember what it was I proposed. I think it went like this: 1. Provide an MPI hook that an application could call to declare the maximum amount of data that might be sent to unready receivers. 2. Require that an application calling the hook also provide buffer space for that much data. 3. Require that if the hook is called, MPI then must *act as if* it were working in "sender-buffers" mode. 4. One possible implementation of this approach is for MPI to simply transform all blocking send calls into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) This implementation does not seem to impose any extra costs at the protocol level. In particular, if non-blocking send can be accomplished without an "RTS/CTS/data" sequence, say by assuming some amount of buffering at the receiver as in the Intel iPSC short-message and DELTA protocols, then the sender-buffers implementation proposed here should be able to do the same. No doubt many optimizations within MPI are possible -- the proposal just requires that MPI act as described here. I am not sure how the sender-buffers declaration should look in order to best support libraries. One possibility would be to allow the declarations to be nested, so that each library could declare its own needs at the point where those needs are determined. I'm not convinced that this is much easier than just (re)coding the library to use non-blocking sends directly, but it seems to fit with the basic idea of portable support for non-blocking csend's. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Tue Feb 9 02:04:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18444; Tue, 9 Feb 93 02:04:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06135; Tue, 9 Feb 93 02:03:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 9 Feb 1993 02:03:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06127; Tue, 9 Feb 93 02:03:03 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20719; Tue, 9 Feb 93 01:02:59 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04746; Tue, 9 Feb 93 01:02:55 CST Message-Id: <9302090702.AA04746@donner.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Tue, 09 Feb 93 01:02:54 CST From: Rusty Lusk Last week Tom Henderson asked for more detail about the multiple-level proposal, so we wrote up a slightly more detailed explanation of the idea, with some examples, which is appended at the end of this note. It is partly subsumed by Marc Snir's posting of a thoroughly worked out point-to-point proposal, which has a lot in common with what we described as Levels 1 and 2. A few notes: In Level 1, we use a very minimal set of required parameters in the "init" calls, so that more of the parameters can be reset inside a loop without acquiring a new handle. We really think that Levels 3 and 4 (the simpler levels, where there are fewer options and parameters) are necessary in order to provide a simple interface for basic users. We didn't change the way heterogeneity is handled, but Tom Henderson's suggestion (that the datatypes *always* be specified, perhaps with a BYTES type for the "raw" datatype) is a quite sensible one. Here is a medium-sized description of what we mean by a "multi-level" standard. The specific routines and their parameters are not as important as the idea that complexity can be managed in this way. ---------------------------------------------------------------------- \documentstyle[11pt]{article} \textwidth=6.0in \textheight=9.0in \hoffset=-0.5in \voffset=-1.0in \parskip=.1in \def\explicitspace{\hbox{\vrule height 2pt width .5pt \vbox to 8pt{\vfil\hrule height .5pt width 6pt}% \vrule height 2pt width .5pt}} \begin{document} \title{\Large \bf A Proposal for a Multi-Level Message-Passing Interface Standard} \author{ {\it William D. Gropp and Ewing L. Lusk} \\ Argonne National Laboratory \\ Argonne, IL 60439 \\ gropp@mcs.anl.gov \\ lusk@mcs.anl.gov } \date{} \maketitle \begin{center} {\large \it DRAFT} \end{center} \section{Introduction} \label{sec:introduction} In \cite{gropp-lusk:mpi1} we described the organization of a library of routines that could be used to implement a standard set of message-passing functions. We used the library to rapidly implement most of the routines described in \cite{dongarra-hempel-hey-walker:october-draft}. The implementation testbed was not originally offered as a counterproposal to the routines described in \cite{dongarra-hempel-hey-walker:october-draft} since no real discussion had taken place in the MPI Committee yet, but it did contain many capabilities we believed would ultimately be desired. It was constructed in a way that offered a wide range of capabilities with a minimum of complexity in syntax. Basically, it consisted of a large number of separate routines whose names were orthogonally organized along a set of axes corresponding to synchronization type, buffer structure, high-performance mode, and heterogeneity specification. See \cite{gropp-lusk:mpi1} for details. At the MPI meeting in January, it became apparent that \begin{itemize} \item Many of the committee members were uncomfortable with having a large number of routines. \item A very wide range of capabilities and options was nonetheless desired. \item A very wide range of users was to be targeted, from experienced users requiring access to high-performance operations to beginning users requiring simplicity in both syntax and semantics. \item While discussion of precise syntax was deferred, it might be useful to suggest one, in order that everyone could see what the ultimate standard might look like. \end{itemize} We added a further consideration of our own: \begin{itemize} \item The standard needs a way to expand in order to add functionality or options that have not been envisioned yet, without becoming incompatible with itself. \item An implementation would have to be extremely adaptable if it was expected to be able to track the discussions of the MPI Committee. \end{itemize} The present draft is a reflection of these considerations. It is still incomplete (as are the committee's discussions) but is offered in the hope that it might be useful. It focuses primarily on the point-to-point operations. \section{The Basic Idea: Orthogonality of Function} \label{sec:orthogonality} The MPI committee has discussed a large number of options for point-to-point message passing. The fact that there could be as many as a thousand different ``send'' routines has become a joke, and some believe that the scope of MPI way have to restricted in order that it not become unwieldy. This problem has at least two solutions, and we explore each of them here. What makes the porblem tractable is that the options that are being discussed are {\it orthogonal\/}; that is, choice of one value of an option does not determine what value another opetion must have. For example, whether a send is blocking or nonblocking is completely independent of whether the message consists of contiguous memory or not. This orthogonality can be exploited to make the interface simple to specify, implement, and use, while at the same time not sacrificing flexibility. The first way is to set the options with a ``universal'' {\tt set\_option(handle,option,value)} routine. There will be a small number of allowed values for {\tt option}, and for most of these, a small number of values for {\tt value}. (This is the approach taken in Level I, below.) The second way is to absorb the options and their values into the routine names. This leads to a large {\em number\/} of routines, but their names can be made predictable and descriptive, so that they are easy to understand and use. (This is the approach taken in Level II, below.) The principal organizing idea of this library is that it proposes multiple sets of routines, roughly grouped by complexity level, {\it all of which are proposed for simultaneous inclusion as part of the MPI Standard}. That is, although it is possible to implement some of the higher-level routines in terms of the lower ones, either by combining short sequences of low-level calls or by supplying specific values for parameters, it is strongly suggested here that all levels be adopted together. This is the way to attain a set of routines that, taken altogether, offers complete flexibility for the advanced user while not imposing any unnecessary complexity in either syntax or semantics on the beginner. The relationships among the levels remain extremely useful for defining the semantics of the higher-level routines in terms of the lower level ones, which have simpler semantics. We now describe the four ``levels'' of MPI routines. For expository purposes, we describe them in what might be considered an unnatural order: \begin{description} \item[Level IV] The highest level, consisting of only four routines, suitable for beginning users. \item[Level I] The lowest level, giving complete access to all of the facilities offered by the MPI Standard. The actual number of routines is small. \item[Level III] A middle level, required in order to write programs that are more efficient and portable than those expressible with Level IV. This is the approximate level of most current message-passing systems. \item[Level II] A level between I and III, chosen for convenience of use without giving up the features of Level I and possible added efficiency in the implementation. It corresponds roughly to the set of routines described in \cite{gropp-lusk:mpi1}. \end{description} It is to be emphasized that this layered structure is for ease of definition and convenience of use, and does not imply a layered structure for the implementation, which could result in suboptimal efficiency. Likewise, it is not suggested here that a user would have to pick one of the layers and stick with it. In all cases where it makes sense, the layers should be interoperable; that is, routines from all layers may appear in the same program. \section{The Point-to-point MPI routines} \label{sec:pt-to-pt} Note on syntax: we have to pick {\em some\/} syntax in order to write these down clearly. We leave the selection of an elegant, expressive, complete, intuitive syntax and appropriate language bindings to the further discussions of the MPI Committee. \subsection{Level IV: The Four-function Interface} \label{sec:four-func} A large number of parallel algorithms can be expressed in terms of four simple functions. In the quest for simplicity, it is useful to see how far the standard might be able to go. It appears to us that the following set of routines represents a minimum. (Strictly speaking, only two of these belong in this point-to-point section, but it is not worth separating this tiny set of routines into parts.) \begin{verbatim} MPI_numpids() returns the number of active process ID's MPI_mypid() returns the id of the calling process MPI_send(dest,tag,buffer,length) sends the contiguous set of length bytes addressed by buffer to process dest and tags the message with tag. The buffer is available for reuse when this returns. MPI_recv(dest,tag,buffer,length) waits for a message from process dest with tag tag and puts in buffer, truncating it to length if necessary for it to fit into buffer. \end{verbatim} More precise definitions of these can be made once we have defined the lower levels. Basically, the point-to-point component consists of an asynchronous send that blocks only until the buffer can be reused, and a blocking receive that waits for the arrival of a message. \subsection{Level I: Access to complete functionality in a small number of routines} \label{sec:level1} The above trivial interface is enough to specify a parallel program, and in many cases, is actually sufficient. The only reason for adding more point-to-point routines is to improve efficiency, portability, and convenience. Before defining what perhaps will be the most commonly-used layer for most programmers (Level III), we go down to Level I, where all of the MPI capabilities will be exposed. We can then describe the intermediate layers more easily, in terms of the lowest one. The idea here is to fully exploit for the lowest level two ideas that were discussed favorably at the January meeting: \begin{itemize} \item separation of the ``setup'' part of an operation from the ``initiate'' part. This idea is described in detail in \cite{snir:notes}. \item the use of a control structure (either in system space or user space) to hold the parameters for an operation that has been ``set up'', but is not complete. (It may or may not have been initiated.) This structure serves as the ``handle'' for the operation. Its precise format depends on the MPI implementation, and it is accessed by the user only through MPI routines. \end{itemize} \begin{verbatim} handle = MPI_init_send(dest,tag) handle = MPI_init_recv() MPI_mod_send(handle,option,value) MPI_mod_recv(handle,option,value) MPI_do_send(handle) MPI_do_recv(handle) MPI_free(handle) \end{verbatim} The idea is that the ``init'' routines return a handle that is usable {\em as is\/}, because it selects a reasonable set of defaults, but that can be modified before the operations is actually initiated. If the {\tt do\_send} or {\tt do\_recv} operations are issued without any intervening calls to {\tt mod} routines, then the default behavior occurs, which we will have to decide on. However a wide range of options can be set by the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines. Although we are not intending to specify an implementation, it will perhaps clarify things to envision how this might work. The {\tt init} routines will allocate a block of storage (or, alternatively, deal with a block a storage allocated by the user). The ``do'' routines will take their parameters from this control block. The control block will be immediately usable, since the minimal parameters will be specified on the ``init'' call. (A send doesn't make sense without a destination and a tag, so they are its parameters. A receive need specify no filtering by tag or sender, and could receive a message of 0 bytes, requiring no buffer, hence the {\tt recv\_init} has no parameters.) The {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} are used to modify the default parameters set up by the {\tt MPI\_init\_send} and {\tt MPI\_init\_recv} routines. The parameters to the {\tt MPI\_mod} routines are as follows: \begin{description} \item[handle] the value returned by a previous {\tt MPI\_init\_send} or {\tt MPI\_init\_recv}. \item[option] one of a collection of symbolic constants specifying the parameter being set. \item[value] one of a collection of symbolic constants or integers or addresses, depending on the option. \end{description} One suggestion for a set of options and their allowed values might be: \begin{verbatim} option value ------- ------- BUFTYPE - one of CONTIG, STRIDED, VEC, HVEC BLOCKING - one of NONE, LOCAL, GLOBAL TRANSLAT - one of NONE, XDR PROTOCOL - one of PLAIN, RCVRRDY BUFADD - address of buffer, or NULL BUFLEN - integer lengh of buffer DATATYPE - one of INT, FLOAT, DOUBLE, LONGINT, BYTES, CHAR TAG - integer or (on receive) ANY BUFVEC - address of structure describing the buffer data, or NULL SOURCE - (on receive) integer process id, or ANY \end{verbatim} The {\tt MPI\_init\_send(dest,tag)} could reasonably set the following defaults, in addition to setting the {\tt dest} and {\tt tag} fields in the control block: \begin{verbatim} BUFTYPE = CONTIG BLOCKING = LOCAL TRANSLAT = NONE PROTOCOL = PLAIN BUFADD = NULL BUFLEN = 0 DATATYPE = BYTES BUFVEC = NULL \end{verbatim} The {\tt MPI\_init\_recv()} could set the same default values, together with {\tt SOURCE = ANY} and {\tt TAG = ANY}. (In a real proposed standard, these symbolic constants should have an MPI prefix, in order to prevent the name-space pollution problem, but here we leave them off for simplicity.) The large number of routines listed in \cite{gropp-lusk:mpi1}, some of which will reappear in Level II, have in Level IV been absorbed into parameter values of the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines, in particular the first four values of {\tt option}. The advantages of this approach are: \begin{itemize} \item a small number of routines at the lowest level \item extensibility: new options, even implementation-specific ones, can be handled by adding more options without adding more routines. \end{itemize} The disadvantage is that to send a message requires a sequence of subroutine calls, adding both program complexity and a moderate amount of overhead to each operation. It is this problem that is addressed by level II. \subsection{Level III: A Medium-Level Interface} \label{sec:level3} In this section we describe a level that is designed to provide access to a sufficient set of MPI routines that efficient, portable programs may be written in it, without accessing all of the flexibility of the complete MPI interface. This level will, however, support both blocking and non-blocking sends and receives as well as messages between machines from different vendors. By imposing a few restrictions on what can be done at this level, however, we can reduce the number of routines and shorten parameter lists, thus reducing complexity. A possible set of restrictions for this level might be: \begin{itemize} \item No noncontiguous buffer structures \item No mixed datatypes in messages destined for heterogeneous processors \item No ``rendezvous'' send \item No ``receiver-ready'' messages \end{itemize} Programmers who need more functionality than is offered at this level can use the routines in levels I or II. The set of Level III routines, given the above restrictions, looks like \[ \left[ \begin{array}{c} n \\ b \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right], \] \noindent together with the {\tt wait} and {\tt status} routines. The first column specifies whether the opertion is to block or not, and the last column specifies whether the message is to be translated for transmission to a machine with a different data representation. That is, the above diagram ellustrates the way the names are to made up. Since in this case the list is relatively short, we can also list them: \begin{verbatim} bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) \end{verbatim} \subsection{Level II: Flexibility and Efficiency in a large number of routines} \label{sec:level2} For Level II, we use only the naming schematic, since the list becomes rather long. The first column represents the choice of buffer structure ({\tt BUFTYPE} in Level I). The second column specifies the blocking option ({\tt BLOCKING}). Then whether it is a send or receive, then the {\tt TRANSLAT} option, and finally the receiver-ready protocol ({\tt PROTOCOL = RCVRRDY}). \[ \left[ \begin{array}{c} c \\ s \\ g \end{array} \right] \left[ \begin{array}{c} n \\ b \\ s \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ rr \end{array} \right], \] \noindent See \cite{gropp-lusk:mpi1} for details. \section{Examples of the Multi-Level Approach} \label{sec:examples} Before giving the details, we give three examples of how certain operations would be expressed at teach of the levels. Note that some operations are not available at the higher levels. The point is that simple operations should be simple to express while more sophisticated ones may be more complicated. \paragraph{Example 1.} A blocking send of a contiguous stream of uninterpreted bytes. At Level IV (the four-function level) nothing but the basic parameters needs to specified, since this is the only type of send operation. \begin{verbatim} MPI_send(dest,tag,buf,len) \end{verbatim} At Level III, where both blocking and non-blocking sends are possible, we need to specify that this is a blocking send. We choose to incorporate the specification into the routine name. \begin{verbatim} MPI_sendb(dest,tag,buf,len) \end{verbatim} At Level II, noncontiguous buffers are allowed, so we need to specify that the buffer is contiguous, which we again do with the name. \begin{verbatim} MPI_csendb(dest,tag,buf,len) \end{verbatim} At Level I, we need to set up the operation first, then modify the default parameters of the operation before actually initiating it. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_mod_send(handle,BLOCKING,LOCAL) MPI_do_send(handle) MPI_send_wait(handle) MPI_send_free(handle) \end{verbatim} \paragraph{Example 2.} A non-blocking receive of data that is all of the same type into a contiguous buffer, with translation for receipt from a process on a machine with a possibly different data representation. At Level IV, this operation is impossible to express. At Level III, the specification that is it is non-blocking and heterogeneous is encoded in the name, which specifies a routine that expects a datatype and number of items, rather than a length. \begin{verbatim} handle = MPI_nrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level II, we must also specify that the buffer is contiguous. \begin{verbatim} handle = MPI_cnrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level I, we set up the operation and then modify its parameters. \begin{verbatim} handle = MPI_init_recv() MPI_mod_recv(handle,SOURCE,source) MPI_mod_recv(handle,TAG,tag) MPI_mod_recv(handle,BUFADD,buf) MPI_mod_recv(handle,BUFTYPE,CONTIG) MPI_mod_recv(handle,TRANSLATE,XDR) MPI_mod_recv(handle,DATATYPE,datatype) MPI_mod_recv(handle,NUMVALS,numvals) MPI_do_recv(handle) MPI_recv_status(handle) \end{verbatim} \paragraph{Example 3.} A non-blocking send of non-contiguous, mixed heterogeneous data, with the assumption that the receiver has already issued his receive. At Levels IV and III, this is impossible to express. At Level II, most of the options are encoded in the name. \begin{verbatim} MPI_gsendhrr(dest,tag,...) \end{verbatim} At Level I, we use the same basic routines to set the options. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod(handle,BUFTYPE,HVEC) (Build vector describing data locations, types, and numbers of items in each clump.) MPI_mod_send(handle,HVECADD,vec) MPI_mod_send(handle,TRANSLATE,XDR) MPI_mod_send(handle,RECVRRDY,TRUE) MPI_do_send(handle) \end{verbatim} \paragraph{Example 4.} Set up a channel from one process to another, and then use it for repeated send operations of different data. This operation cannot be specified at Levels II, III, or IV. At Level I, it can be specified in the following way. \begin{verbatim} handle = MPI_init_send(dest,tag) begin loop MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_do_send(handle) end loop \end{verbatim} \section{Collective Communication} \label{sec:collective} The idea of a separate setup routine, together with multiple levels to accomodate both and basic operations as well as more elaborate ones, should prove useful for collective operations as well. We will try to extend the point-to-point operations to collective operations in as natural a way as possible. \bibliographystyle{plain} \begin{thebibliography}{1} \bibitem{dongarra-hempel-hey-walker:october-draft} J.~Dongarra, R.~Hempel, T.~Hey, and D.~Walker. \newblock A proposal for a message passing interface standard. \newblock (circulated before November meeting). \bibitem{gropp-lusk:mpi1} Bill Gropp and Rusty Lusk. \newblock A test implementation of the {MPI} draft message-passing interface standard. \newblock Technical Report ANL--92--47, Argonne National Laboratory, December 1992. \newblock (circulated at January meeting). \bibitem{snir:notes} Marc Snir. \newblock Message-passing interface--outline. \newblock (circulated at January meeting). \end{thebibliography} \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 06:33:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22237; Mon, 15 Feb 93 06:33:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11881; Mon, 15 Feb 93 06:32:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11873; Mon, 15 Feb 93 06:32:34 -0500 Date: Mon, 15 Feb 93 11:32:26 GMT Message-Id: <21543.9302151132@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: various To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear colleagues I circulated the documents prepared by Marc, and Bill & Rusty, around interested people locally, for discussion. Here are some resulting considerations of the point-to-point component of MPI. a) Synchronous messages and blocking/nonblocking communication calls: The properties of end to end synchronisation, and blocking/nonblocking communication calls are "orthogonal" in the sense described by Bill and Rusty. Thus, another "axis" in the procedure naming (section 3.3) should be added, which perhaps is either nothing or 's'. The approach taken by Marc is a little, different and appears to implicitly recognise this. This doesnt seem (to me) to fit in with the framework which Marc uses, but it might be appropriate to consider synchronous and ready-receiver as points on the same axis. They both refer to the knowledge which the sender has about receipt of the message. In the synchronous case, the sender absolutely knows that the receiver has got the message; in the ready-receiver case the sender (presumably) has no knowledge of whether the receiver has or has not got the message. This approach would also remove the curious "synchronous ready receiver" from Marc's document. b) Cancellation of nonblocking communications: This subject has appeared before. We strongly support the points made by Jim Cownie regarding the requirement for cancellation of nonblocking communications. An example of its usefullness arises when writing libraries encapsulating common communication patterns, we view it as desirable that the communication call should be able to return with an error code if something is wrong. A routine XXX may not be able to detect this until after some nonblocking communications have been started, and some of those which have been started have also been completed and the information therein examined. At this point it becomes known that the operation cannot succeed, however there are these nonblockinf communications outstanding. It may not be possible to wait for them to complete, as they may never complete since the matching operation may never be started, so th natural thing to do is cancel the outstanding communications and return cleanly with an error code. The importance of the clean return is apparent when one considers the possibility that the memory into which pending nonblocking receives may write (or may not, unknown) is a local variable of the caller of XXX, which will itself then wish to return to its caller, which will in turn call some user error procedure. In this case the stack can become corrupted by data from the uncancelled nonblocking receive started in XXX and the whole program barfs up. Not nice. c) Matching of buffer structures: The proposals contain three provisions for buffer structures: contiguous, strided and iovec (ignoring heterogeneous issues for the moment, see below). Each of these should be defined to provide a byte stream (which may not map to a contiguous set of bytes in memory), and the programmer should thus be allowed to mix'n'match sends and receives of different types freely. This provides a handy way of performing scatter and gather type operations. d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. e) Completion/checking functions: In sections 1.5.5 and 1.5.6 of Marc's document, the START and COMPLETE operations are described. I cannot agree that there should be seperate MPI_SEND_XXX and MPI_RECV_XXX routines (XXX is one of {START,COMPLETE,CHECK}). The handle contains all of the information regarding the kind of operation which has been started, so it is not necessary to have different routines. It can be most convenient to be able to wait for a mixture of sends and receives to complete, more specifically to wait for any of such a mixture. In order to do this, you must have an interface which allows send and receive handles to be passed to the same wait routine. Another advatage, perhaps, is that htis wait function does not have to raise an error if the wrong kind of handle is supplied, although this advantage is minimal since it will have to check validity of handles in a more general sense. There has been an argument of readability for having seperate routines for send and receive. We could resolve this by defining MPI_SEND_XXX and MPI_RECV_XXX to be wrappers around MPI_XXX, which check that all handles are of the appropriate nature. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:09:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04339; Mon, 15 Feb 93 10:09:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20259; Mon, 15 Feb 93 10:08:42 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:08:41 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20251; Mon, 15 Feb 93 10:08:40 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15846; Mon, 15 Feb 93 09:08:27 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27168; Mon, 15 Feb 93 09:08:24 CST Date: Mon, 15 Feb 93 09:08:24 CST Message-Id: <9302151508.AA27168@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 11:32:26 GMT <21543.9302151132@subnode.epcc.ed.ac.uk> Subject: various X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Date: Mon, 15 Feb 93 11:32:26 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk ... Just one clarification: ... d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. Actually, our proposal is somewhat stronger than this, at least at the "expert" level (level 1): The user can provide an array of structures, each of which is essentially an "iovec" (that is, pointer and length) with an additional field for the datatype. Thus, a message consisting of a pair of integers and an array of doubles could be sent as (please excuse the syntax; this can be cleaned up in any number of ways): mvec[0].len = 2 * sizeof(int); mvec[0].ptr = &i[0]; mvec[0].datatype = MPI_INT; mvec[1].len = 347 * sizeof(double); mvec[1].ptr = &array[0]; mvec[1].datatype = MPI_DOUBLE; Alternate forms of this structure can basically define a "channel program" that allows for things like non-unit-stride access and scatter/gathers. Thus, just as with PVM, a single message may portably contain multiple data types; the difference is in when the data is presented to the interface (one datatype at a time, as in PVM, or all at once, as in our proposal). This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. This is our intent as well. Note that it would be easy to write a tool that would take a structure/derived type definition and generate the appropriate setup/calls in either the PVM case or our proposal (A compiler could do it to, but I suspect that initially many would be happy not to have the compiler tinkered with, particularly since such enhancements would not be portable). Bill and Rusty From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:21:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04647; Mon, 15 Feb 93 10:21:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20871; Mon, 15 Feb 93 10:21:27 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:21:25 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20820; Mon, 15 Feb 93 10:20:39 -0500 Date: Mon, 15 Feb 93 15:19:51 GMT Message-Id: <21840.9302151519@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: various To: gropp@antares.mcs.anl.gov (William Gropp) In-Reply-To: William Gropp's message of Mon, 15 Feb 93 09:08:24 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Hi Bill and Rusty Thanks for the clarification. I hadn't picked up the 'datatype' field in the descriptor structure for iovec messages - I can't find it in the proposal document. What you propose here for level 1 is indeed as strong as PVM - it may be more convenient also. Of course the iovec is then a bit less like the kind of iovec that Un*x programmers are familiar with, and used in this way it makes sense to check that the iovec structures at sender and receiver have some kind of consonance. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 15:26:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08992; Wed, 3 Mar 93 15:26:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03642; Wed, 3 Mar 93 15:25:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 15:25:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03627; Wed, 3 Mar 93 15:25:50 -0500 Date: Wed, 3 Mar 93 20:25:31 GMT Message-Id: <6861.9303032025@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: safe, unsafe and erroneous MPI programs To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk MPI, more specifically the point-to-point component, describes three classes of programs: safe, unsafe and erroneous. I'm thinking about buffer resource exhaustion here. It is possible to write safe programs in MPI. It is not possible to identify a subset of MPI which is guaranteed to provide either safe or erroneous programs, and cannot produce unsafe programs. We think that MPI should contain such an identifiable subset. This subset was provided by the SYNCHRONOUS completion mode proposed by Marc Snir, which was rejected at the last meeting. I could not support the proposal because I do not believe that the completion mode proposed is actually what users want, and did not reflect common practice. Primary proposal ---------------- I propose that a further completion mode should be added to MPI, in addition to the surviving REGULAR completion mode, which reflects common practice and creates the identifiable subset of MPI which. I propose that a SECURE (for want of a better name) completion mode be added, which has the following semantics: (a) The COMPLETE operation will wait until the user buffer for send is clear for use. In send the data has been copied from the user buffer (without specification of where it has been copied to), and in receive the data has been copied to the user buffer. AND (b) The COMPLETE operation will wait until the sender (receiver) has performed the START operation which matches the receiver (sender). Note that (a) is just the REGULAR completion mode, and (b) has no additional meaning for a receiver. It really just applies to senders. With this addition, the identifiable subset of MPI which produces either safe or erroneous programs is that which contains the SECURE complete mode and does not contain the READY start mode. The program Process 1 Process 2 --------- --------- secure send to 2 secure send to 1 blah blah blah secure recv from 2 secure recv from 1 is erroneous (by definition?). It will deadlock with every correct implementation of MPI. The program Process 1 Process 2 --------- --------- nonblocking secure send to 2 nonblocking secure send to 1 nonblocking secure recv from 2 nonblocking secure revc from 1 blah blah blah wait for send to 2 wait for send to 1 wait for recv from 2 wait for recv from 1 is safe (by definition?). It will not deadlock with any correct implementation of MPI. Secondary proposal ------------------ MPI should not contain a combination of SECURE complete mode and READY start mode. I can see no sense in such a combination, and leaving it out decreases the overlap between the safe subset of MPI and the potentially unsafe subset of MPI. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 16:54:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11917; Wed, 3 Mar 93 16:54:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08053; Wed, 3 Mar 93 16:53:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 16:53:07 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08044; Wed, 3 Mar 93 16:53:05 -0500 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA20435; Wed, 3 Mar 93 13:52:55 PST Message-Id: <9303032152.AA20435@SSD.intel.com> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: safe, unsafe and erroneous MPI programs In-Reply-To: Your message of "Wed, 03 Mar 93 20:25:31 GMT." <6861.9303032025@subnode.epcc.ed.ac.uk> Date: Wed, 03 Mar 93 13:52:31 -0800 From: prp@SSD.intel.com I agree that there should be a way to produce programs which are known to be safe. However, I also think that once a program is proven safe, it should be easy to allow it to take advantage of buffering when available. To make it easy, I recommend a global safeness mode, rather that one that must be selected on each send and receive. I would suggest that the environmental management section contain a call in which the user can specify that all operations are to have SECURE semantics if coded as normal semantics, and those coded with READY semantics would either be proved correct (might be hard) or flagged as erroneous. This way, a program can be proved safe and then released from SECURE semantics with a single line change, instead of going through all the sends and receives and changing them from SECURE to normal. This is particularly important if we decide to distinguish modes by syntax rather than by a mode parameter. (I think we should use syntax because it can be more efficient.) So, I would propose a call like this: MPI_safe() which, if present, will cause the implementation to flag an unsafe program as erroneous. Alternatively, it could be a link option (that is, the MPI standard would require that the implementor provide a means of specifying that a program is to be created or executed in such a way as to report an error if any unsafe operation is attempted.) This would be analogous to lint for C programs. You use it when you need it, and the rest of the time it is not in the way. Paul From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 06:57:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13100; Thu, 4 Mar 93 06:57:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19151; Thu, 4 Mar 93 06:55:09 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 06:55:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19137; Thu, 4 Mar 93 06:55:05 -0500 Date: Thu, 4 Mar 93 11:55:00 GMT Message-Id: <7359.9303041155@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: safe, unsafe and erroneous MPI programs To: prp@SSD.intel.com In-Reply-To: prp@SSD.intel.com's message of Wed, 03 Mar 93 13:52:31 -0800 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu I'm afraid that I must disagree with the notion of a global SECURE mode as Paul has suggested, which can be switched on or off as the programmer/user desires. In the first case, we really must think about programs being written to include compositions of parallel libraries which have already been written and determined to be correct and secure. So the library writer should have control of how the library communications are performed, and this kind of "global change the semantics of everything" should be avoided and discouraged. In the second case, once a module has correctly and safely been written using secure communications I really don't think there will be so much to "take advatage of" in use of system buffering. All of my colleagues at EPCC program in this way, and all of our library work is done in this way. Using the secure communications and blocking/nonblocking forms one can write, without difficulty, secure and efficient software. In fact, more efficiently, in principle, since less memory copies of message data are needed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 07:17:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20778; Thu, 4 Mar 93 07:17:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19895; Thu, 4 Mar 93 07:16:45 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 07:16:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19885; Thu, 4 Mar 93 07:16:36 -0500 Date: Thu, 4 Mar 93 12:16:27 GMT Message-Id: <7397.9303041216@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: mpi-pt2pt@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 3 Mar 93 16:01:25 CST Reply-To: lyndon@epcc.ed.ac.uk > I think that we agreed to defer dynamic issues to a future MPI standard > level, but I do agree that this is never guaranteed to happen, as Rusty > has said. Unless of course we all want to and can afford from our funding more beano trips to Dallas ;-) (joking, honest guv!) > The lack of control for processes, and the lack of dynamic > group syntax/semantics worries me. They will almost certainly appear as > non-portable extensions. Yup, probably, over what timescale do we think? > Perhaps the original N-month limit was too short to permit the best > possible standard. > Perhaps, but on the other hand dynamic groups (i.e., those where the membership changes in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)) really do seem to be at an experimental stage in our worlds. We have dynamic groups in the system we have implemented (CHIMP), but to be quite honest hardly any users have really used them to implement dynamic groups, rather they have implemented static groups (i.e., those where the membership does not change in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)). Dynamic groups have been put into PVM3, as experimental functions which may or may not survive in the current or some other form. In short, it seems to be just too early to try to define a standard for dynamic groups (glossary), which is of course unfortunate. I'd suggest strongly that we must have an open mind toward - or even plan for - a later invocation of MPI, at which point subjects such as dynamic group and dynamic process models are dealt with. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 09:48:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28327; Thu, 4 Mar 93 09:48:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27405; Thu, 4 Mar 93 09:47:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 09:47:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27396; Thu, 4 Mar 93 09:47:38 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17713; Thu, 4 Mar 1993 09:47:36 -0500 Date: Thu, 4 Mar 1993 09:47:36 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041447.AA17713@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: So many proposals, so few standards... I just wanted to put in my endorsement for Bill and Rusty's proposal to have three levels. The lowest having Marc's init, start, wait, free, etc. constructs. The next level having send and receives using buffer descriptors. The highest level send and receives specify the buffer and the datatype and send only contiguous bytes. I also like the idea of having opaque task id. This has been brought up by Peter, Tony, and now Bill and Rusty. There are nice features that arise from this approach that makes computing in a dynamic, heterogeneous environment much easier and efficient. PVM has now gone to this method of task naming because of the advantages. There is still a need for users to think in terms of 0-[p-1] and this is logically provided by having a rank in a group. I also agree with Bill and Rusty that to write portable code we have to have a standard way to start up one or more tasks. The existing interfaces all provide some mechanism for doing this. Are we going to propose a standard that doesn't even provide some of the basic features of the exiting interfaces? With regard to opaque task id, spawning tasks, and dynamic groups, I see a double standard being applied that I do not like. When we discuss communication routines, many people vote for new totally untried (maybe unworkable) concepts because "they could lead to more efficient programs in the future." Fine. But when we talk about things like dynamic groups, people start saying "gosh, never been tried before. We better not do it." Some of our standard effort is future looking and other parts are backwards looking. I feel it is time to try to be consistent. One final thought. If we do not design an interface that is as flexible, intuitive, and easy to use as existing interfaces, then we will have simply created Yet Another Message Passing interface (YAMP) and not a standard. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:17:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00544; Thu, 4 Mar 93 11:17:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01967; Thu, 4 Mar 93 11:14:31 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:14:29 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01959; Thu, 4 Mar 93 11:14:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05686; Thu, 4 Mar 93 10:11:41 CST Date: Thu, 4 Mar 93 10:11:41 CST From: Tony Skjellum Message-Id: <9303041611.AA05686@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, geist@gstws.epm.ornl.gov Subject: Re: So many proposals, so few standards... I concur with Al. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:21:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00629; Thu, 4 Mar 93 11:21:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02226; Thu, 4 Mar 93 11:19:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:19:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02173; Thu, 4 Mar 93 11:19:42 -0500 Date: Thu, 4 Mar 93 16:19:30 GMT Message-Id: <7649.9303041619@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: So many proposals, so few standards... To: geist@gstws.epm.ornl.gov (Al Geist), mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 4 Mar 1993 09:47:36 -0500 Reply-To: lyndon@epcc.ed.ac.uk Okay, this is mainly about task identifiers again. [stuff deleted] > I also like the idea of having opaque task id. This has been brought up > by Peter, Tony, and now Bill and Rusty. There are nice features that > arise from this approach that makes computing in a dynamic, heterogeneous > environment much easier and efficient. PVM has now gone to this > method of task naming because of the advantages. > There is still a need for users to think in terms of 0-[p-1] > and this is logically provided by having a rank in a group. if (MPI will provide process groups) then One thing processes do really find useful - from hard experience - is to be able to receive a message from any member of the process group. If we frame point-to-point in terms of some opaque (global) task identifier type then how can we provide the very expressive receive selection above? If we can't, then opaque types do not seem capable of providing expressive power for process groups. fi > I also agree with Bill and Rusty that to write portable code > we have to have a standard way to start up one or more tasks. > The existing interfaces all provide some mechanism for doing this. > Are we going to propose a standard that doesn't even provide > some of the basic features of the exiting interfaces? Either one includes the task creation in the source code of the program in which case MPI somehow ought to look at a mechanism compatible with the overall conceptual content of MPI Or one puts the task creation outside the program and outside the domain of MPI. This also seems to boil down to whether we have a dynamic process model or a static process model. When we decide this then we can look at the details, but I really do think MPI should agree which the goal is before trying to reach consensus on the attack plan. Why don't we email vote on this? My vote: static model. > With regard to opaque task id, spawning tasks, and dynamic groups, > I see a double standard being applied that I do not like. > When we discuss communication routines, many people vote for > new totally untried (maybe unworkable) concepts because > "they could lead to more efficient programs in the future." Can we find examples of this? Let me see off the top of my head ... a) Communication handles, unless I missed somthing. b) Ready sender communication (symmetry partner of ready receiver) c) Buffer descriptiors with data types > Fine. But when we talk about things like dynamic groups, > people start saying "gosh, never been tried before. We better not do it." > Some of our standard effort is future looking and other parts > are backwards looking. I feel it is time to try to be consistent. Unfortunately it is the burden of a standards committee that it should standardise only agreed existing practice, in such a manner that imminent developments are anticipated and a route for extension of the standard at such a time that such developments become common practice is practical, insofar as the standards committee is able to so do. My phrasing may not be the best, but the principle should be perfectly clear. > One final thought. If we do not design an interface that is > as flexible, intuitive, and easy to use as existing interfaces, > then we will have simply created Yet Another Message Passing interface (YAMP) > and not a standard. i.e., it will not be acceptable to a sufficiently large portion of the market and therefore will not be accepted by the market as a 'de facto' industry standard. How about some serious suggestions regarding how MPI can ensure to avoid this apocalypse? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 12:53:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03951; Thu, 4 Mar 93 12:53:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07822; Thu, 4 Mar 93 12:50:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 12:50:06 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07811; Thu, 4 Mar 93 12:50:05 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17511; Thu, 4 Mar 1993 12:50:03 -0500 Date: Thu, 4 Mar 1993 12:50:03 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041750.AA17511@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Reply to Lyndon >is to be able to receive a message from any member of the process group. >If we frame point-to-point in terms of some opaque (global) task >identifier type then how can we provide the very expressive >receive selection above? The receive side is easy, and if you REALLY mean process group above then the send side is also. But if you refuse to use the rank in a group then knowing who to send to is the trick. >Either one includes the task creation in the source code of the program >Or one puts the task creation outside the program... Putting it outside only works for multprocessors. What if you have multiple multiprocessors or just a bunch of workstations on a network? Users would have to write a separate (non-portable) program just to start up their applications. The argument still boils down to portability. >Why don't we email vote on this? My vote is dynamic model >Unfortunately it is the burden of a standards committee that it should >standardise only agreed existing practice, This is what I read as one of our original GOALS in MPI. But that is not what I see happening. We either need to stick to the original goal and drop all the "fringe" stuff out of the communication specification, or we take a consistently forward-looking approach (which can't possibly be agreed on in two more meetings) but may be the only way that we >can ensure to avoid this apocalypse? Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 03:26:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23608; Fri, 5 Mar 93 03:26:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22531; Fri, 5 Mar 93 03:25:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 03:25:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22521; Fri, 5 Mar 93 03:25:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 4 Mar 93 13:22 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA11676; Thu, 4 Mar 93 13:21:48 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA10514; Thu, 4 Mar 93 13:21:44 PST Date: Thu, 4 Mar 93 13:21:44 PST From: d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303042121.AA10514@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Al Geist writes: > Putting it outside only works for multprocessors. What if you have > multiple multiprocessors or just a bunch of workstations on a network? > Users would have to write a separate (non-portable) program just > to start up their applications. I am puzzled by these words. There is at least one package (TCGMSG, by Robert Harrison) that uses external invocation and works quite nicely on workstation networks as well as multiprocessors. In TCGMSG, the programming interface is that the application just calls PBEGIN_(argc,argv) to initialize, then finds out the number of participating processes by calling NNODES(). (This is a static process model.) For workstations, TCGMSG supplies a general purpose program to start up application processes. It works by passing them command-line flags that allow them to find each other. The application does not need to worry about these flags, because they are handled by the PBEGIN_ . On multiprocessors, other procedures may be used. For example, on the DELTA, the application processes are initiated by the Intel utility 'mexec', and PBEGIN_ works by calling the Intel intrinsic numnodes(). > The argument still boils down to portability. Exactly. As discussed above, the start-em-up-externally approach used by TCGMSG allows a program to execute without change across widely varying environments, with no need for the application programmer to write non-portable code. It would have been extremely difficult, verging on impossible, to support an interface that explicitly allows an application to create new processes on the fly. (That capability is not provided by the DELTA operating system. But they do support sockets, so I guess you could have a demon on some other system that would rsh back to the delta and do another mexec, then use some undocumented and risky system interfaces to pass messages between the nodes of the two mexec's, and then... But that doesn't seem very robust.) > >Why don't we email vote on this? I propose that MPI simply mandate the existence of a capability to start up processes externally. Those processes find themselves executing spontaneously and then find out about the others by making MPI calls. The syntax and semantics of the MPI calls should be specified completely, but we should not say anything about the external capability except that it exists. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 09:45:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27927; Fri, 5 Mar 93 09:45:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14009; Fri, 5 Mar 93 09:43:32 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 09:43:30 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14000; Fri, 5 Mar 93 09:43:26 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA22099 (5.65c/IDA-1.4.4 for ); Fri, 5 Mar 1993 09:43:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA05607; Fri, 5 Mar 93 14:43:17 GMT Date: Fri, 5 Mar 93 14:43:16 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303051443.AA05607@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02088; Fri, 5 Mar 93 14:40:29 GMT To: mpi-pt2pt@cs.utk.edu Subject: Cancel Content-Length: 4971 People, I believe that any of the non-blocking operations which MPI mandates should be cancellable. Why have cancel ? ================= There are various reasons for this (in random order) 1) Resource liberation Some (? ALL ?) of these operations commit system resources. If the user knows that the operation can never complete, then she should be able to release those resources. 2) Access revocation Some of these operations give the MPI system authority to modify areas of the user's store. (non blocking receive in particular). The user should be able to revoke this permission. [See examples with receive buffers on the stack for the true potential horror here !] 3) Cleanliness In some of Marc's early documents (and maybe still) there is an implication that all messages MUST have been received at program exit, otherwise the program is erroneous. It seems reasonable also to require all non-blocking operations to have completed (even MPI -1.1's non blocking exit call completes at this point !), if this is so then we need to be able to force the completion of these operations. (This is particularly the case for server type objects which may wish to queue many non-blocking receives [so as to avoid system buffering and additional data copying], but then wish to exit, possibly returning to the rest of the code...). What does cancel mean ? ======================= Cancelling an operation forces its completion with an error status of CANCELLED. One can reasonably require that the cancelled operation then be tested for completion in the normal way that any such non-blocking operation would be. Cancelling an operation can FAIL. This occurs if the operation to be cancelled has already successfully completed (Entered the state in which it could be successfully found as complete) or the system can guarantee its completion in a bounded time (in which case the cancel call should block until the operation completes). After an operation has been cancelled, I know that either 1) The operation completed normally or 2) The operation will never now proceed (it has been cancelled) The two cases can be separated by testing the completion return from the normal WAIT or STATUS call. (This is preferable to returning this result as the status of the cancel call as it makes the case of one thread cancelling a call on which another thread is already waiting resolve correctly). Consider for example cancelling a non-blocking send operation, there are three possible cases :- 1) data transfer has not actually started => it never starts, the operation is really cancelled, returning a CANCELLED kind of result. 2) data transfer has already completed => cancel failed and returns a COMPLETED kind of result 3) data transfer has started, so it is allowed to complete then reduces to case 2) Note that cancellation is an operation which happens to a REQUEST, it is not an operation which is applied to a MESSAGE. Therefore once a particular receive has been (successfully) cancelled the system state is as if that receive had never existed. There is NO implication that the cancel has an effect on any future incoming message which would have matched the, now cancelled, receive. Such a message can still be received in the normal way by any suitable matching receive call. Cancelling Group operations =========================== If we have non-blocking group operations we should be able to cancel them. It seems entirely reasonable to require that all the participants who have started the operation have also to cancel it. Issues ====== People are nervous about cancel because there is a race condition involved. This race is between the cancel operation itself, and the external event which signals completion of the operation being cancelled. (e.g. the cancel and the incoming message on a non-blocking receive). However this race is no worse than the existing one between a receive and an incoming message. (Does the data go straight into the user buffer, or require system buffering ??) Implementing a full cancel as described above is non-trivial, since it may be necessary to send out a cancel message after a non-blocking send. (This is the case if one uses receiver buffering). If this is viewed as too complex, (however note that various people already have implementations of cancel which implement semantics similar to those described above), then at the very least we should still seriously consider including the ability to cancel operations which gave the system permission to write into our address space. (Thanks to Lyndon for some of the examples and other discussion). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 10:05:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28310; Fri, 5 Mar 93 10:05:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15328; Fri, 5 Mar 93 10:04:29 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 10:04:26 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15320; Fri, 5 Mar 93 10:04:25 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06600; Fri, 5 Mar 93 09:01:26 CST Date: Fri, 5 Mar 93 09:01:26 CST From: Tony Skjellum Message-Id: <9303051501.AA06600@Aurora.CS.MsState.Edu> To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon Rik, Using the Delta's lack of an elegant process model to defend your point is not really valid. They could have supported dynamic process creation, but just were too busy with other things, and did not see how useful the previously supported feature(s) were from iPSC960 and iPSC/1&2. The real issue is whether anything is lost/gained by separating spawning and MPI messaging. Why might there be? 1) ability to spawn processes in a way to take advantage of special group properties (hierarchy in the memory/network access). Oh well, the user can still do this him/herself, and then tell MPI that the group has a special property. 2) ability to support dynamic process model to allow MPI to serve as a base layer for additional services. For true portability, I would like to avoid having MEXEC on Delta, and spawn() functions on iPSC860 and CE/RK, and so on. The lack of a well-understood need for group, so that group-scope operations, can be attributed to a group, have not been recognized in MPI. As each group might want to implement a different broadcast or combine or other operations, the lack of a well-formed group concept is the bottleneck to this discussion. I am sleepy, I can't think of more now. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 11:07:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00459; Fri, 5 Mar 93 11:07:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20433; Fri, 5 Mar 93 11:06:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 11:06:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20425; Fri, 5 Mar 93 11:06:37 -0500 Date: Fri, 5 Mar 93 16:06:03 GMT Message-Id: <8555.9303051606@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Cancel To: jim@meiko.co.uk (James Cownie), mpi-pt2pt@cs.utk.edu In-Reply-To: James Cownie's message of Fri, 5 Mar 93 14:43:16 GMT Reply-To: lyndon@epcc.ed.ac.uk Hello world :-) I just like to say that I generally support Jim's proposal. I see three sensible options, at this point: (1) Cancel for nonblocking receive operations (2) Cancel for nonblocking point-to-point operations (i.e., send and receive) (3) Cancel for nonblocking operations (i.e., point-to-point and collective) [How about cancellable nonblocking group partitions? :-)] I will certainly vote for (2) - I don't suppose there are any surprises there. I expect that I will find in favour of (3), but I really have to think through implementation first, so I'm reserving judgement. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 13:07:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03718; Fri, 5 Mar 93 13:07:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27705; Fri, 5 Mar 93 13:06:17 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 13:06:15 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27686; Fri, 5 Mar 93 13:06:11 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 5 Mar 93 10:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA13638; Fri, 5 Mar 93 10:00:28 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA11163; Fri, 5 Mar 93 10:00:25 PST Date: Fri, 5 Mar 93 10:00:25 PST From: d39135@sodium.pnl.gov Subject: static process creation To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303051800.AA11163@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Tony Skjellum writes: > Rik, > > Using the Delta's lack of an elegant process model to defend your > point is not really valid. They could have supported dynamic process > creation, but just were too busy with other things, and did not see > how useful the previously supported feature(s) were from iPSC960 and > iPSC/1&2. I was addressing the question of whether it is OK for MPI to REQUIRE dynamic process creation, or whether it should include a capability that requires only static processes. My position is that a static process model should be supported. I have no problem with MPI also providing a standard interface to dynamic processes in environments where that can be done. But I strongly feel that a dynamic process interface should not be the only one. In the note that Tony refers to, I presented existence arguments that: a) The static process model can be supported cleanly on systems that also support dynamic processes, such as workstation networks. (Someone else had asserted the contrary.) b) There exists at least one system (the Delta) that MPI seems to target, and that does not support dynamic process creation. This still seems like a valid argument for supporting static processes. I agree that the argument would be stronger with a longer list of systems that do not adequately support dynamic process creation. I used the Delta because that's one example that I'm sure of. I am also sure that the iPSC/860 under NX does not have dynamic process creation. I believe that the iPSC/1 and /2 did have it, but I am not so sure that their model/implementation allowed a process running on a node to create a process on another node. That capability seems to be assumed by the arguments about why we want dynamic process creation. If more examples are really needed, perhaps they can be provided by people who have the facts. I am not sure, as examples only, that the CM-5, NCube-2, or Meiko systems provide all the dynamic process support that's required. I'd be pretty surprised if Intel was the only vendor with current offerings that don't. But my guess is that this is one of those issues where we have an extended discussion and then all vote one way. If we're going to vote, then I propose that the issue be phrased "Should MPI support a static process model, or should it support ONLY a dynamic process model?" Is there actually debate on this issue? Can anyone argue why a static process model should NOT be supported? --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 13:21:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04185; Fri, 5 Mar 93 13:21:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28589; Fri, 5 Mar 93 13:21:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 13:21:04 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28519; Fri, 5 Mar 93 13:20:55 -0500 Date: Fri, 5 Mar 93 18:20:50 GMT Message-Id: <8662.9303051820@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: d39135@sodium.pnl.gov's message of Fri, 5 Mar 93 10:00:25 PST Reply-To: lyndon@epcc.ed.ac.uk Rik writes: > I used the Delta because that's one example that I'm sure of. I am > also sure that the iPSC/860 under NX does not have dynamic process > creation. I believe that the iPSC/1 and /2 did have it, but I am not > so sure that their model/implementation allowed a process running on a > node to create a process on another node. That capability seems to be > assumed by the arguments about why we want dynamic process creation. > > If more examples are really needed, perhaps they can be provided by > people who have the facts. I am not sure, as examples only, that the > CM-5, NCube-2, or Meiko systems provide all the dynamic process > support that's required. I'd be pretty surprised if Intel was the > only vendor with current offerings that don't. > FYI: Meiko Computing Surface - i.e., the machine they can deliver today as opposed to the "new" machine - don't allow dynamic process creation. I believe that CM-5 only allows you to kick off an SPMD program, not very dynamic really. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 17:47:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11144; Fri, 5 Mar 93 17:47:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12931; Fri, 5 Mar 93 17:45:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 17:45:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12922; Fri, 5 Mar 93 17:45:51 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA07602; Fri, 5 Mar 93 17:45:31 -0500 Date: Fri, 5 Mar 93 17:45:31 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303052245.AA07602@msr.EPM.ORNL.GOV> To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Subject: Re: static process creation Hi Rik, Just a point of fact: we just got through testing a fully dynamic model on the Intel i860. It is a part of the PVM 3.0 software. Maybe the delta is the ONLY machine. Good thing only one was built. I think your phrasing is ment to sway the reader to vote a particular way (your way of course) We shouldn't loose track of the fact that a static model is just a subset of the dynamic model. There is no such thing as ONLY a dynamic model. I can not argue that a static model should not be supported. This is not the debate. The issue is should we limit MPI to a static model (We haven't even voted to have process creation of any kind yet) given that a more flexible model interface already exists (PVM and others) but maybe a better argument is the ability to write more fault tolerant applications with a dynamic model. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 20:48:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14007; Fri, 5 Mar 93 20:48:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20019; Fri, 5 Mar 93 20:46:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 20:46:54 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20011; Fri, 5 Mar 93 20:46:51 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 5 Mar 93 17:38 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA14836; Fri, 5 Mar 93 17:36:52 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA12820; Fri, 5 Mar 93 17:36:39 PST Date: Fri, 5 Mar 93 17:36:39 PST From: d39135@sodium.pnl.gov Subject: static process subset? To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, geist@msr.EPM.ORNL.GOV, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303060136.AA12820@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Al Geist writes: > Just a point of fact: we just got through testing a fully dynamic model > on the Intel i860. It is a part of the PVM 3.0 software. Great! Can you share with us how you did it, and what is the performance of the communications between dynamically allocated processes on two i860 nodes? I have visions of layers of daemons (like my tongue-in-cheek description of how to do dynamic processes on the Delta), but then I'm a notorious skeptic. > I think your phrasing is ment to sway the reader to vote a particular way > (your way of course) This strikes me as a personal attack, although of course I have no idea how it was intended. I will resist responding other than to ask, we can focus on the technical issues? > We shouldn't loose track of the fact that > a static model is just a subset of the dynamic model. > There is no such thing as ONLY a dynamic model. Here's an example of what I mean by "supporting ONLY a dynamic model". Suppose we define an interface that says the only way to get P processes running is to write code that says for (i=1; i I can not argue that a static model should not be supported. > This is not the debate. The issue is should we limit MPI > to a static model (We haven't even voted to have process creation > of any kind yet) given that a more flexible model interface > already exists (PVM and others) Perhaps this is an area where MPI needs to have a defined subset. I think it is important that implementations can call themselves "MPI-compliant" without having dynamic process creation. I also think that the MPI interfaces should be defined such that static process creation is cleanly supported. At the same time, I agree that dynamic process models are important. I would NOT vote to restrict the design of MPI to a static process model, in the sense of defining fundamental features that are not compatible with dynamic processes. For this reason, I prefer process ID's to be opaque types rather than sequential integers. I might however vote to do only static now because of time constraints. This is a question of priority. My personal top priority is getting pt-pt and group/context/topology definitions such that rock-solid libraries can be written that will run fast at least in static process environments. Even that much seems to be taking a gosh-awful long time. Dynamic processes would probably be my second priority. Given the current schedule and rate of progress, I don't think I need a third (low) priority. However, (to switch topics a bit) if I did need to name low priority stuff, then certainly figuring out how to implement multiple non-blocking scatter-gather operations on overlapping groups would fall someplace below third. That stuff sounds great for an application, but it's the devil's own time to implement correctly, even given a good definition. So that stuff scares me -- I'm afraid that if we require it, NOBODY will have a compliant implementation. Does anybody else worry about this? > Al Geist --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 22:26:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14706; Fri, 5 Mar 93 22:26:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23643; Fri, 5 Mar 93 22:26:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 22:25:59 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23635; Fri, 5 Mar 93 22:25:56 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA22624 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Fri, 5 Mar 1993 19:25:53 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA19714; Fri, 5 Mar 93 19:18:53 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA09061; Fri, 5 Mar 93 19:25:16 PST Date: Fri, 5 Mar 93 19:25:16 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303060325.AA09061@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Agreeing with Rik I fully agree with Rik Littlefield's comments and am very worried that we are running out of time. My personal preferences at this point would be to prioritize various things as: a) Get collective communication down properly. Include all the (blocking) routines that applications could benefit from - even obscure ones. b) Try to agree on some minimal way of creating a program and identifying a process in it. c) Define point-to-point message passing. d) Worry about "other things" like dynamic process models and non-blocking concatenate and its friends. I put collective communication ahead of point-to-point in my list because I am EXTREMELY concerned about the complexity of the current point-to-point design. My feeling is that a complex point-to-point layer is only important if it allows applications to be written that execute faster than they would with a simpler layer. In my experience, especially with number-crunching programs, users most benefit from highly optimized collective communication routines, not a vast array of point-to-point routines. One might argue, of course, that the sophistication in the point-to-point layer is exactly so that implementors can make highly optimized collective comm. layers. I don't agree with this for the simple reason that I don't think it's obvious that any particular layer in the point-to-point spec. is faster than any other. If this is so I would imagine that collective communication calls will be optimized independently for each machine (as they are now) rather than being built "portably" from the point-to-point layers. Thus the sophistication in the low level is probably wasted. Of course, many people will want to build their codes from simple point-to-point routines, but I wonder if this doesn't just mean that we're missing some important collective ones, at least in some cases. To respond to one of Al Geist's comments it bothers me too that we seem to be accepting vast quantites of point-to-point stuff that certainly isn't current practice (some of it doesn't even exist in any other system) and yet are wary of adding things like smart collective routines and toplogy support that are in day-to-day use in a lot of applications. Jon Flower p.s. By the way, my vote would be for opaque task ID's for hypothetical forward-compatibility, ..... but I think they're a royal pain to use!! In every case where I've had to do this I've ended up making an array in each node, collecting the opaque ID's and converting them to consecutive integers. From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 13:58:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02798; Sat, 6 Mar 93 13:58:21 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08785; Sat, 6 Mar 93 13:57:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 13:57:13 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08762; Sat, 6 Mar 93 13:57:10 -0500 Date: Sat, 6 Mar 93 18:57:01 GMT Message-Id: <328.9303061857@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: d39135@sodium.pnl.gov's message of Fri, 5 Mar 93 10:00:25 PST Reply-To: lyndon@epcc.ed.ac.uk I sent this yesterday, but it seems to have dissapaered into the great bit bucket in the sky (nice phrase, Jim :-). Rik writes: > If more examples are really needed, perhaps they can be provided by > people who have the facts. I am not sure, as examples only, that the > CM-5, NCube-2, or Meiko systems provide all the dynamic process > support that's required. I'd be pretty surprised if Intel was the > only vendor with current offerings that don't. FYI: Meiko Computing Surface - i.e., the machine they can deliver today as opposed to the "new" machine - don't allow dynamic process creation. I believe that CM-5 only allows you to kick off an SPMD program, not very dynamic really. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 14:17:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03026; Sat, 6 Mar 93 14:17:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09402; Sat, 6 Mar 93 14:17:20 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 14:17:19 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09388; Sat, 6 Mar 93 14:17:17 -0500 Date: Sat, 6 Mar 93 19:17:14 GMT Message-Id: <346.9303061917@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Fri, 5 Mar 93 17:45:31 -0500 Reply-To: lyndon@epcc.ed.ac.uk Hi Al, Rik, mpi-pt2pt world ... I mailed out two other peices of kit which I understand don't allow dynamic processes. I should have pointed out, I guess, that to the best of my knowledge this a restriction placed on the user by the "system" software, and not by the hardware. If my guess is correct The system architects within the respective companies will have made a decision that this would be the interface which they present to the user, and they *could* have made a different decision. So I hardly think that the argument that there are existing systems which do not provide dynamic process creation really is an argument that those systems should not be asked to provide dynamic process creation within MPI, unless the vendors among us advise that if MPI contains dynamic process creation they will not implement that part of MPI for those systems *and* the existence of such systems which do not implement MPI, or part thereof, will have a significant affect on the market impact and penetration of MPI. I'm hardly a person who is able to say much of value on the last point. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 15:37:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03409; Sat, 6 Mar 93 15:37:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12480; Sat, 6 Mar 93 15:36:46 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 15:36:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12470; Sat, 6 Mar 93 15:36:41 -0500 Date: Sat, 6 Mar 93 20:36:37 GMT Message-Id: <390.9303062036@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: dynamic processes and dynamic groups To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Hi again Well, we certainly are having some fun now in this discussion of static vs dynamic processes in MPI. This is as much about dynamic groups as dynamic processes, and the latter part is really about MPI as a standards committee. I have a lot of sympathy with John Flower's concerns. There appears to be reasonable consensus among those contributing to the discussion that in a world process model where processes dynamically appear and dissapear in an asynchronous fashion an enumeration of processes is not a useful way of identifying processes - what sense could we make of the enumeration? An opaque type process (or task) identifier is better. The same considerations surely lead us to the observation that in a world group model where processes dynamically join and leave groups in an asynchronous fashion an enumeration of members of a group is not a useful way of identifying members - what sense could we make of the enumeration? Rolf previously pointed this out on mpi-comm - March 4, subject "Process ids". What are asynchronously created (terminated) processes to do? I guess they either join (leave) an existing group, or create (destroy) a group for their own purposes. The primitives for these two are really process group resizing (or destruction/recreation) and process group creation (termination). I guess their must be heavyweight PVM users who have other experiences of usage of dynamic process creation/termination - Al?. These thoughts/questions, however interesting and discussion provoking they may or may not be, do not take us any further on answering some of the BIG questions about what MPI will be, regarding for example: a) Process model; static v dynamic; ... b) Group model; static v dynamic; ... I've previously stated that I'd vote for static process model in (a). This "seemed to be" the intent of MPI (although I couldn't find it written down and recorded as agreed), would be useful, and could be agreed upon within the MPI time schedule. This is not to say that I think limitation to a static process model would be the most useful thing upon which to agree to describe a 'de facto' standard. I have pretty well the same opinions regarding process groups. I'd like us all agree on how we're going to move forward and get the details of the standard described, as I'm concerned that at the current rate we may not. I'll broadly support John Flower's suggestions, primarily because I don't see any others, and I'll do my best to implement whatever strategy we can agree on. Final thought, perhaps we should move the discussion of MPI strategy to mpi-comm. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 11 18:14:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10133; Thu, 11 Mar 93 18:14:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23324; Thu, 11 Mar 93 18:13:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 11 Mar 1993 18:13:13 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23312; Thu, 11 Mar 93 18:13:05 -0500 Via: uk.ac.southampton.ecs; Thu, 11 Mar 1993 18:45:16 +0000 Via: brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:38:09 GMT From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:46:23 GMT Date: Thu, 11 Mar 93 18:46:26 GMT Message-Id: <19625.9303111846@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: safe, unsafe and erroneous MPI programs L J Clarke writes: > This subset was provided by the SYNCHRONOUS completion mode proposed by > Marc Snir, which was rejected at the last meeting. I could not support > the proposal because I do not believe that the completion mode proposed > is actually what users want, and did not reflect common practice. It was with some dismay that I learned from the above message that the SYNCHRONOUS completion mode had been dropped, having been unfortunately unable to attend the last meeting, for which minutes have not yet been posted. Was this decision due to some awkward aspect of the precise semantics in Marc's proposal, or was it just that the committee does not see the need for any form of end-to-end synchronization mode at all? If the latter is the case then I am astounded. In any case, I would strongly advocate the inclusion of a synchronous, mode of communication, with both blocking and non-blocking variants, for the same reasons elucidated by Lyndon in a message posted before the February meeting: > Given the above, I come to the conclusions: > > i) MPI must contain unbuffered communications with blocking/nonblocking > (irecv/isend/msgwait kinda thing) calls, for reliability and > portability. > > ii) If a goal of MPI is that existing applications using message passing > interfaces (eg Express, PVM) should easily port to MPI, then MPI must > also contain buffered comms. This seems to be a matter for the full > committee, hence I have crossposted. What went wrong at the February meeting? I've not been very vocal on the mpi mailing lists of late, having got behind in reading the mound of messages arriving daily, but I'm back on top of it and hope to be contributing more from now on. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Sun Mar 14 22:10:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12927; Sun, 14 Mar 93 22:10:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02224; Sun, 14 Mar 93 22:09:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 14 Mar 1993 22:09:42 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02216; Sun, 14 Mar 93 22:09:37 -0500 Message-Id: <9303150309.AA02216@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8667; Sun, 14 Mar 93 22:09:35 EST Date: Sun, 14 Mar 93 22:09:34 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir \\ William Gropp and Ewing Lusk} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following subsections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. } \subsection{Data Types} \subsubsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. Handles are of type {\tt void *} in C and of type {\tt integer} in Fortran. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). An opaque object is created by a call to {\tt MPI\_CREATE}, and destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to create, access and update specific opaque objects. {\bf \ \\ MPI\_CREATE(handle, type, persistence)} \begin{description} \item[OUT handle] handle to object \item[IN type] state value that identifies the type of object to be created (e.g., {\tt MPI\_COMMUNICATION, MPI\_BUFFER, MPI\_CONTEXT}, etc.). \item[IN persistence] state value; either {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}. \end{description} {\bf \ \\ MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} An object can be destroyed only if there is no pending operation that is using this object; after successful return of the routine, the handle is undefined. {\bf \ \\ MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \paragraph*{List of handles} An MPI call may need a parameter that is a {\em list of handles}. In C, such list will be a record with one component being the length of the list, the other components being an array of pointers. In Fortran, the list will be an array of integers, the first one of which is the length of the list. \discuss{ The mechanism for opaque objects used here follows the POSIX Fortran binding standard. An alternative choice is to have different type declarations for each type of opaque object. Then, opaque objects are created/destroyed like regular variables, rather than by MPI calls; they are still accessed and updated only via MPI functions. } \subsubsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_CREATE} routine has a state type parameter with values {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}. An {\tt enumeration} declared in an included MPI.h file will be used in C for state datatypes. The Fortran 77 mechanism needs to be decided. \discuss{ Named integer constants can be used in Fortran 90, using the {\tt PARAMETER} mechanism. The constant declarations can be made available via an {\tt INCLUDE} file. Fortran 77 does not seem to offer any convenient mechanism. One possibility is to specify explicit integer values, and allow the use of named constants with those Fortran 77 compilers that support them conveniently. Another possibility is to use character strings, rather than integers. } \subsubsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). The special values are provided by named constant, that are made available via an MPI.h include file in a C binding. \discuss{ Need to agree on a Fortran mechanism for named constants (see the discussion above). Implementers should try to detect illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent should be flagged as an error. } \subsubsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. In C, a formal parameter of type {\tt void *} will be used, with an actual pointer parameter. in Fortran, we shall cheat. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \subsection{Processes} An MPI program is executed by several autonomous processes that execute each their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow to specify the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution, although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. Absolute, system-wide unique process id's are (will be) needed only if dynamic process creation is to be supported (in such eventuality we propose to use handles to opaque {\em process structures} for that purpose). \subsection{Contexts} \discuss{ This section contains a proposal for use of contexts that will subsume groups. It borrows heavily on the current group proposal. This proposal has not yet been discussed in MPI meetings.} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \paragraph*{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \paragraph*{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \paragraph*{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \paragraph*{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \subsubsection{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \paragraph*{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } \subsubsection{Error Handling} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it is the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to store an error number in a global variable ({\em errno} in a C environment) and generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. } \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may mix values of different types. \discuss{ May also need {\tt DOUBLE COMPLEX} in Fortran.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the current context. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. {\tt Context} should be a context shared by both source and destination. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. A buffer consists of a sequence {\bf buffer components}. Each component consists of a sequence variables of the same basic type. There are three basic types of buffer components: \begin{description} \item[block] A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[datatype] Type of elements \end{description} \item[vector] A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk \le stride$) \item[datatype] Type of elements \end{description} Note that a constant stride becomes contiguous when {\tt stride = lenblk}. A vector buffer component can be an arbitrary submatrix of a two-dimensional matrix. \item[indexed] A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[list\_of\_indices] List of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} \end{description} For example, if a Fortran array is declared as {\tt double precision a(10)}, then the tuple {\tt $<$(a(3), (0,3,6,2), DOUBLE$>$} specifies a buffer component with entries {\tt a(3) a(6), a(9), a(5)}. \discuss{ Do we allow entries to be repeated in an indexed buffer component? Do we allow different buffer components to overlap? Do we require in an vector buffer component that {\tt len} be a multiple of {\tt lenblk}? } A buffer is described by an opaque object accessed via a {\bf buffer handle}. Such object is created and destroyed via calls to {\tt MPI\_CREATE} and {\tt MPI\_FREE}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_ADD\_BLOCK}, {\tt MPI\_ADD\_VECTOR} or {\tt MPI\_ADD\_INDEX}, in order to append a component to the buffer associated with a buffer handle. A buffer object can be destroyed only if there is no pending communication operation using it. After a buffer object is destroyed the associated buffer handle is undefined. {\bf \ \\ MPI\_ADD\_BLOCK( buffer, start, len, datatype)} \\ Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_VEC( buffer, len, stride, lenblk, datatype )} \\ Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_INDEX( buffer, start, list\_of\_indices, datatype)} \\ Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[start] initial position for indexing (choice) \item[list\_of\_indices] list of relative indices of entries (array of integers) \item[IN datatype] datatype identifier (status) \end{description} Consider, for example, the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, B, 1, MPI_INT) CALL MPI_ADD_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_ADD_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_ADD\_xxx} call). Such handle is associated with an empty buffer, and a message created from it contains no data. \discuss{ The main modifications w.r.t. the proposal of Gropp and Lusk is measuring length in elements, rather than bytes. Seems more natural, since type is known. Also, object creation uses a generic function, rather than a function specific to buffer descriptors. As result, I gave up on the size parameter. This may not be such a loss: it is not clear that specifying a maximum length at buffer object creation is useful, since indices of arbitrary size may need to be stored in the object. } \subsubsection{Data Conversion} The types and the locations of the entries used to create a message is solely determined from the information in the buffer descriptor, using the storage association rules specified by the host language and its implementation; the type and the locations of these entries do not depend on the declaration for the corresponding variables in the calling program. It is not required that the data types specified in a buffer descriptor match the data types of the corresponding elements in the host program. However, in case of mismatches, the correspondence between entries in the host program and entries in a message created with the buffer descriptor may be implementation dependent. No data conversion occurs when data is moved from a sender buffer into a message. Consider the following fragment of Fortran code \begin{verbatim} REAL A(100) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, A(1), 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, A(2), 1, MPI_INT) CALL MPI_ADD_BLOCK (BH, A(3), 1, MPI_LOGICAL) CALL MPI_ADD_BLOCK (BH, A(4), 1, MPI_COMPLEX) CALL MPI_ADD_BLOCK (BH, A(6), 1, MPI_DOUBLE) CALL MPI_ADD_BLOCK(BH, A(8), 4, MPI_CHAR) \end{verbatim} A message created from this buffer will consist of a sequence containing one real, one integer, one logical, one complex, one double, and four characters. No data conversion occurs when values are copied from the sender buffer to the message. Thus, the first entry in the message is a real number equal to {\tt A(1)}; the second entry is an integer number that happens to have the same binary representation as {\tt A(2)}; the third entry is a logical value that happens to have the same binary representation as {\tt A(3)}; the fourth entry in a complex number with a binary representation identical to {\tt A(4), A(5)}; the fifth entry is a double precision value with a binary representation identical to {\tt A(6), A(7)}; and if words have four bytes then the last four entries are bytes that make the binary representation of {\tt A(8)}, in the byte order of the executing machine. The correspondence between the first seven entries of the array {\tt A} and the first five entries of the message created from this buffer is determined by the rules of Fortran 77 on storage association: Each variable of type {\tt INTEGER}, {\tt REAL}, or {\tt LOGICAL} occupy one {\em numeric storage unit}; a variable of type {\tt DOUBLE} or {\tt COMPLEX} occupy two numeric storage units. Thus, the same correspondence will hold for any implementation. However, different implementations may have different binary encodings of integer, real and logical values, so that the actual values transferred by the message may differ. The correspondence between the entry {\tt A(8)} of the array, and the last four character entries in the message is implementation dependent, since the Fortran language does not specify a correspondence between character storage units and numeric storage units (an array of characters cannot be equivalenced with an array of integers). Different results may occur in big-endians or small-endians machines, or in 32 bit or 64 bit machines. The same holds, symmetrically, when a message is received. Entries are moved from the message into the receiver memory according to the information provided by the buffer descriptor, with no regard to the way the corresponding variables are declared in the receiving program. When data is moved in a homogeneous environment between nodes having the same architecture, then no data conversion occur at any point during data transfer. Assume, in the previous example, that an identically declared buffer descriptor is used to receive data into an identically declared array at the receiving process. Then, when these two nodes communicate, the values in the first eight entries of array {\tt A} of the sender will be copied into the first eight entries of array {\tt A} at the receiver (assuming that reals occupy the same storage as four bytes). In particular, in a homogeneous environment, it is possible to communicate using only character typed messages. When data is moved into a heterogeneous environment between nodes having distinct architectures, data conversion may occur during the transfer: Each entry is converted from the data representation used on the sending node to the data representation used in the receiving node. Consider the following fragment of Fortran code. \begin{verbatim} REAL X, Y CHARACTER (4) Z INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, X, 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, Y, 4, MPI_CHAR) CALL MPI_ADD_BLOCK (BH, Z, 4, MPI_CHAR) \end{verbatim} Assume that the same arrays {\tt X, Y, Z} and buffer handle {\tt BH} are created at two processes A and B; this handle is used by A to create and send a message for B, by B to receive this message. Further assume that both A and B run on distinct nodes, with possibly different 32 bit architectures. Then {\tt X} at process B is assigned the value of {\tt X} at process A (up to rounding errors that may occur during conversion); {\tt Z} on process B is assigned the character string value of {\tt Z} on process A; if both nodes use ASCII encoding, then no conversion is required for the characters. On the other hand, variable {\tt Y} on process B may be allocated a value that differs from the value of variable {\tt Y} on process A. This may occur if the two nodes use a different byte sequence (little-endian vs big-endian), or use a different binary representation for reals (IEEE vs HEX). Thus, in order to ensure correct execution in a heterogeneous environment, it is important that the types of values in a message match the types of the corresponding values in the sending and in the receiving program. \subsection{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \subsection{Communication Mode} A sending operation can occur in one of two modes: \begin{description} \item[REGULAR] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \end{description} A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required. \discuss{I deleted the symmetric ready receive. Will revive it if there is a requirement for it.} \subsection{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such object persists only until the communication operation is completed, at which point it is destroyed. Thus, correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Communication Object Creation} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. {\bf \ \\ MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} \\ Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] handle to context of messages sent with this handle \item[IN mode] send mode (state type, with two values: {\tt MPI\_REGULAR} and {\tt MPI\_READY}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} {\bf \ \\ MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, persistence)} \\ Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or DONTCARE (integer). \item[IN tag] user tag for messages received with this handle, or DONTCARE (integer). \item[IN context] handle to context of messages received with this handle. \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. } \subsubsection{Communication Start} {\bf \ \\ MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf \ \\ MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. {\bf \ \\ MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete, In such case, the return handle points to an opaque object that contains information on the completed information. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. Implementation notes: A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. {\bf \ \\ MPI\_RETURN\_STAT( handle, len, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT len] difference between length of receive buffer and length of received message, in bytes. Thus, the value returned is zero if the received message matches the the receive buffer, positive if it is shorter (integer). \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \discuss{ I put the difference between message buffer and message length as the value returned, rather than length of received message, so that it might be easy to test for exact match. The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming.} \subsubsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. {\bf \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_handle)} \\ Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} is equivalent to the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, see Section~\ref{subsec:correct}). {\tt \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_status\_handle)} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], return_handle); index = 0} || ... || {MPI_WAIT_RECV (handle[n], return_handle); index = n-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) {\bf \ \\ MPI\_STATUSANY ( list\_of\_handles, index, return\_handle)} \\ Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the later case, it returns a value of -1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} {\bf MPI\_WAITALL(list\_of\_handles, list\_of\_return\_handles)} \\ Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \subsection{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void or {\bf R}) indicates the start mode (regular or ready). {\bf \ \\ MPI\_SEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RSEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} {\bf Implementation note:} While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} indicates the {\tt READY} mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. {\bf \ \\ MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) \end{verbatim} \subsection{Block Sending Operations} The most frequent type of buffer used is a contiguous buffer of numeric storage units, i.e., a contiguous buffer of words that may contain either INTEGER, REAL or LOGICAL values (in FORTRAN). In a homogeneous environment such messages can be used to send arbitrary sequences of contiguous items, where each item occupies an integer number of words. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf B} in the function name, for {\tt BLOCK}. {\bf \ \\ MPI\_SENDB (start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_SEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_RSENDB (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RSEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf MPI\_RECVB(buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RECV(buffer_handle, source, tag, context, return_status_handle) \end{verbatim} {\bf \ \\ MPI\_ISENDB (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_ISEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRSENDB (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRSEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRECVB(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRECV(handle, buffer_handle, source, tag, context) \end{verbatim} \discuss{ I use word count, rather than byte count. I believe word messages are much more prevalent than byte messages, and it's a blessing not to have to multiply by 4 for each message. Byte messages are still available the hard way. Also, word messages fit well with the Fortran numeric storage unit concept. If we are to add more functions, my next addition would be a vector send operation. } \subsection{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVB(recvbuf, len, 1, tag, context) CALL MPI_SENDB(sendbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDB(sendbuf, len, 0, tag, context) CALL MPI_RECVB(recvbuf, len, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt len} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Sun Mar 14 22:16:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12993; Sun, 14 Mar 93 22:16:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02504; Sun, 14 Mar 93 22:16:17 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 14 Mar 1993 22:16:16 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02496; Sun, 14 Mar 93 22:16:15 -0500 Message-Id: <9303150316.AA02496@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8685; Sun, 14 Mar 93 22:16:15 EST Date: Sun, 14 Mar 93 22:10:08 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: draft of point-to-point section was sent Reply-To: SNIR@watson.ibm.com Main additions: 1. A more systematic treatment of the various datatypes used by MPI. 2. A proposal for context=group. 3. The incorporation of (a variant of) the Argonne approach to buffer handles. 4. The introduction of two levels of functios, for general buffers and for contiguous data. Comments are welcome. A much as possible, I suggest that we start arguing from the current draft, rather than ab initio. From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 11:33:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25262; Mon, 15 Mar 93 11:33:21 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03012; Mon, 15 Mar 93 11:31:24 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 11:31:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03004; Mon, 15 Mar 93 11:31:21 -0500 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA15829; Mon, 15 Mar 93 08:31:13 PST Message-Id: <9303151631.AA15829@ocfmail.ocf.llnl.gov> Date: Mon, 15 Mar 1993 08:39:09 -0800 To: "Marc Snir" From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: Pt2Pt document Cc: MPI-PT2PT@CS.UTK.EDU Marc, I tried to process your point to point document with LaTeX, but it blew up. Since I am not a LaTeX expert and since I believe LaTeX can output postscript (or at least it outputs something that can be turned into postscript), could you run your document through the necessary processing and send out a postscript document? I believe more of us have postscript printers than have LaTeX processors or LaTeX expertise. Thanks, Regards, Dan From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:06:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02388; Mon, 15 Mar 93 17:06:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14644; Mon, 15 Mar 93 17:05:50 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:05:49 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14636; Mon, 15 Mar 93 17:05:48 -0500 Message-Id: <9303152205.AA14636@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6357; Mon, 15 Mar 93 17:05:47 EST Date: Mon, 15 Mar 93 17:04:54 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: pt2pt draft Reply-To: SNIR@watson.ibm.com Just sent a postcript file, for the benefit of those that don't have access to latex. From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:06:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02386; Mon, 15 Mar 93 17:06:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14541; Mon, 15 Mar 93 17:04:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:04:49 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14533; Mon, 15 Mar 93 17:04:42 -0500 Message-Id: <9303152204.AA14533@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6333; Mon, 15 Mar 93 17:04:35 EST Date: Mon, 15 Mar 93 17:04:31 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT_1.DVI.* %%Pages: 35 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 2 36 df34 DI E /Fb 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E7E 8E12>3 D E /Fc 1 16 df15 D E /Fd 4 111 df<127012F8A3127005057C840E>58 D<15181578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F0 A2123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC007815181D1C7C99 26>60 D<12C012F0123C120FEA03C0EA00F0133EEB0F80EB01E0EB0078141EEC0780EC01E0EC00 78A2EC01E0EC0780EC1E001478EB01E0EB0F80013EC7FC13F0EA03C0000FC8FC123C12F012C01D 1C7C9926>62 D<381E03E0383F0FF038639838EBB01CEAC3E013C0138000075B1300A3000E5BA2 ECE08015C048EBE180EB01C1ECC300EB00E74813FE001813781A157F941D>110 D E /Fe 5 107 df0 D15 D<150C153C15F0EC03C0EC0F00143C14F0EB07 C0011FC7FC1378EA01E0EA0780001EC8FC127812E01278121EEA0780EA01E0EA0078131FEB07C0 EB00F0143C140FEC03C0EC00F0153C150C1500A8B612FCA21E277C9F27>20 D<12C012F0123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC0078151C 1578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F012C0C9 FCA8B612FCA21E277C9F27>I<12C0B3B3AD02317AA40E>106 D E /Ff 26 121 df<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C 01C0A2387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA8 7E007013E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>III73 D76 D<38FC01F8EAFE03A2383B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03 F8A3151C7F9B18>I<387E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A2 13311339A31319A2131D130DA3EA7F07EAFF87EA7F03151C7F9B18>III82 D<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>84 D<38FF83FEA3381C0070 B2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA007C171C809B18>I<387F8FE0139F 138F380E0700120FEA070E138EEA039C13DCEA01F8A26C5AA2137013F07F120113DCEA039E138E EA070F7F000E13801303001E13C0387F07F038FF8FF8387F07F0151C7F9B18>88 D97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3 E0EB80F0EB00701478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C 809B18>II101 D<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C13 1EEA0E0FEB0780130314C038FFC7F8A3151C7F9B18>107 DI110 DI<38 7F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 D<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E001319 7F9818>116 D<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>120 D E /Fg 72 126 df<127012F8B212701200A5127012F8A31270051E769D1A>33 D<1338137813F8EA01E0EA03C0EA0780EA0F00121E121C123C123812781270A312F05AA87E1270 A312781238123C121C121E7EEA0780EA03C0EA01E0EA00F8137813380D2878A21A>40 D<126012F012787E7E7EEA0780EA03C0120113E0120013F01370A313781338A813781370A313F0 13E0120113C01203EA0780EA0F00121E5A5A5A12600D287CA21A>I<13E0A40040134038F0E1E0 EAF8E3387EEFC0381FFF00EA07FC6C5A487EEA1FFF387EEFC038F8E3E0EAF0E13840E040000013 00A413157D991A>I<123C127E127FA3123F1207120F120E123E12FC12F812E0080D77851A>44 D<387FFFC0B512E0A26C13C013047D901A>I<127812FCA41278060676851A>I48 D<13C012011203A21207120F127F12FD 12791201B2EA7FFFA3101E7B9D1A>IIII<383FFFC05AA20070C7FCA8EA73F8EA7FFE7F387E0F80387803 C0EA3001000013E01300A3126012F01301EB03C038700780EA7C1F383FFF00EA1FFCEA07F0131E 7D9D1A>I<137F3801FF804813C03807C1E0EA0F01121E383C00C0003813001278127012F0EAE3 F8EAEFFEB5FC38FE0F8038F803C0EAF00114E01300A31270A2EA7801003813C0EA3C03381F0F80 380FFF006C5AEA01F8131E7D9D1A>I<12E0B512F8A338E001E014C0EA0003EB0780EB0F00130E 131E5B133813781370A213F05BA212015BA312035BA7151F7E9E1A>III<123C127EA4123C1200A9123C127C127EA3123E120E12 1E121C123C12F812F012E0071C77941A>59 D<387FFFF0B512F8A26C13F0C8FCA4387FFFF0B512 F8A26C13F0150C7E941A>61 D<1338137CA2136C13EEA313C6A2EA01C7A438038380A4380701C0 A213FFA24813E0EA0E00A4481370387F01FC38FF83FE387F01FC171E7F9D1A>65 DIIII<387FFFFCB5FC7E380E001CA41400A3EB0380A3EA0FFFA3EA0E03A390C7 FCA8EA7FE012FF127F161E7F9D1A>I<3801F8E0EA03FEEA07FFEA0F0FEA1E03EA3C011238EA78 001270A200F013005AA5EB0FF8A338F000E01270130112781238EA3C03121EEA0F0FEA07FFEA03 FEEA01F8151E7E9D1A>I<38FF83FEA3381C0070AA381FFFF0A3381C0070AB38FF83FEA3171E7F 9D1A>II<387F03F838FF87FC387F03F8381C01 E0EB03C014801307EB0F00131E131C133C5B5B7FEA1DFC121F139E130E130FEA1E07001C138013 0314C0EB01E0A2EB00F01470007F13FC38FF81FE387F00FC171E7F9D1A>75 DI<007E133FB4EB7F806C 1400381D80DCA313C1A2001C139CA213E3A2EB631C1377A21336A2133E131CA21300A7007F137F 39FF80FF80397F007F00191E809D1A>I<38FE03FE12FFA2381D8070A213C0121CA213E0A21360 1370A213301338A21318131CA2130C130EA21306A213071303A238FF81F0A21380171E7F9D1A> IIIII<3807F1C0EA1FFDEA3FFFEA7C1FEA7007EAF003EAE001A390C7FC7E1278123F EA1FF8EA0FFEEA01FF38000F80EB03C0130114E01300126012E0A2EAF001EB03C038FE0780B5FC EBFE00EAE3FC131E7D9D1A>I<387FFFFEB5FCA238E0380EA400001300B3A23803FF80A3171E7F 9D1A>I<38FF83FEA3381C0070B3A2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA00 7C171E7F9D1A>I<38FF01FEA3381C0070A3001E13F0000E13E0A3380701C0A438038380A43801 C700A4EA00C613EEA3136C137CA21338171E7F9D1A>I<00FE13FEEAFF01EAFE000070131C0078 133C00381338A7137C001C137013EEA513C6A2380DC760A31383A3000F13E0A2380701C0171E7F 9D1A>I<387F87F8A3380F01C03807038013833803870013C7EA01CE13EEEA00FCA21378A3137C A213FE13EEEA01CF13C7380387801383380703C01301380E00E0A2387F01FC38FF83FE387F01FC 171E7F9D1A>I<38FF01FEA3381C00706C13E0A2380701C0A213830003138013C700011300A2EA 00EEA2137CA21338AA48B4FCA3171E7F9D1A>I<383FFFF85AA23870007014F0EB01E014C0EA00 03EB0780EB0F00130E131E5B133813785B5B1201485A5B120748C7FC001E1338121C123C5A1270 B512F8A3151E7E9D1A>II93 D<387FFFC0B512E0A26C13C013047D7E1A>95 D97 D<12FEA3120EA6133FEBFFC0000F13E0EBE1 F0EB8070EB00781438000E133C141CA5000F133C14381478EB80F0EBC3E0EBFFC0000E13803806 7E00161E7F9D1A>I<3801FF80000713C04813E0EA1F01383C00C0481300127012F05AA57E1270 007813707E381F01F0380FFFE06C13C00001130014157D941A>II II<3801F8FC3807FFFE5A381F0F8C381C0380003C13C0EA3801A3EA3C03001C 1380EA1F0FEBFF00485AEA39F80038C7FC123C121C381FFF8014F04813F8387C00FC0070131C00 F0131E48130EA36C131E0078133C383F01F8381FFFF06C13E00001130017217F941A>I<12FEA3 120EA6133FEBFF80000F13C0EBE1E013801300A2120EAB38FFE3FE13E713E3171E7F9D1A>II<12FEA3120E A6EB0FFCEB1FFEEB0FFCEB03C0EB0780EB0F00131E5B5B13FC120F13DE138F380E0780EB03C0A2 EB01E0EB00F038FFE3FE14FF14FE181E7F9D1A>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAC387F1F1F39FF9F9F80397F1F1F 00191580941A>IIII<387F87F038FF9FFCEA7FBF3803FC3CEBF018EBE000A2 5BA25BA9EA7FFFB5FC7E16157E941A>114 D<380FFB80EA3FFF5AEAF80FEAE003A300F8C7FCEA 7FC0EA3FFCEA0FFF38007F80EB07C0EA600112E012F0130338FC0F80B512005BEAE7F812157C94 1A>I<13C01201A6387FFFE0B5FCA23801C000AA1470A314F0EBE1E0EA00FFEB7FC0EB3F00141C 7F9B1A>I<38FE0FE0A3EA0E00AC1301A2EA0F073807FFFE7EEA01FC17157F941A>I<387F83FC38 FFC7FE387F83FC380E00E0A3380701C0A338038380A33801C700A3EA00EEA3137CA2133817157F 941A>I<387FC7F8EBCFFCEBC7F8380703C038038380EBC700EA01EFEA00FE137C13781338137C 13EE120113C738038380000713C0EA0F01387FC7FC00FF13FE007F13FC17157F941A>120 D<387FC3FC38FFC7FE387FC3FC380E00E0A27EEB01C013811203EB838013C31201EBC700EA00E7 A213E61366136E133CA31338A35BA21230EA78E01271EA7FC06C5A001EC7FC17207F941A>I123 D<126012F0B3B31260042775A21A>I<127CB4FC7FEA07C0EA01E012 00AB7F137CEB3FE0131F133FEB7C0013F05BAB1201EA07C0B45A90C7FC127C13277DA21A>I E /Fh 22 121 df73 D77 D<90B512E015F890380F003C151E011E130FA21507A249130FA3150E49131E153C153815F09038 F003E090B51280ECFE0001F0C7FC485AA4485AA4485AA4EAFFF8A220227DA121>80 D97 D<137E48B4FC3803C380EA0703EA0E 07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<13F0EA0F E0A21200A2485AA4485AA448C7FC131FEB7F80EBE1C0380F80E0A21300120E381E01C0121CA338 380380A21484EB07060070130C130E1418EB063038E007E0386003C017237DA219>I<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>I108 D<391E07C07C393F1FE1FE3963B8738790 39E03E038012C3EBC03CEB803800079038780700EB0070A3000EEBE00EA21610ED1C18261C01C0 133015381660ED18C03A3803801F80D81801EB0F0025157C9428>I<381E0780383F1FE0EA63B8 EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140CEB031800 3813F0381801E018157C941B>I<137E48B4FC3803C380380701C0120E001C13E0123CA21278A3 38F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803C1F03807E3F8 380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EBE3C038077F80 EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863B0E013E0EAC3 C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380 EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA7007EAF00EA2485AEA7038 EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2EA0700120EA45AA45AA2 13101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F1330381F8070EA31C00061 13E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2130FEB1F183807F3F038 03E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC380A2000313C0EA0700A3 380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I<3803C1C0380FE7E0381C 7E3038303C70EB38F01260146038007000A45BA214200070133038F1C060A200E113C038E3E180 387E7F00EA3C3E14157D9416>120 D E /Fi 65 123 df<903807F83F017FB512C03A01FC0FE3 E03903F01FC7EA07E0D80FC01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A22323 7FA221>11 D13 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA31278 12F8AD1278127CA3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416> 40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A3 13E0A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I<1238127C12FE12FFA212 7F123B1203A212071206A2120C121C12181270122008117C8610>44 D I<1238127C12FEA3127C123807077C8610>I<13181378EA01F812FFA21201B3A7387FFFE0A213 207C9F1C>49 DII<14E013011303A213 07130F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA238 0007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018 C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F838 6007E0003013C0381C1F80380FFF00EA03F815207D9F1C>II57 D<1238127C12FEA3127C12381200A81238127C12FEA3127C12380716 7C9510>I63 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F 140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A >65 DII IIIIII75 DIIIII82 D<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400B4FC13E06C B4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA36C131EA26C 133C12FCB413F838C7FFE00080138018227DA11F>I<007FB61280A2397E03F80F007814070070 14030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127>IIII<3A7FFFC1FFF0A23A03FC00 0C006C6C5B000014386D5B90387F8060013F5B14C190381FE380010F90C7FC14F7EB07FE6D5AA2 6D7E13008081497F14BF9038031FE0496C7E130E90380C07F8496C7E133890383001FE496C7E13 E04848EB7F8049EB3FC03AFFFC03FFFEA227227FA12A>II<003FB512E0A29038801F C0383E003F003C14800038EB7F00485B5C1301386003FC5C130700005B495A131F5C133F495A91 C7FC5B491360485A12035B000714E0485A5B001FEB01C013C0383F8003007F1307EB003FB6FCA2 1B227DA122>I97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3 127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F 0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC 0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B38 1F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2 EAFFF0A20C247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC13 3E137EEA78FCEA7FF0EA1FC0102E83A311>III<3AFF87F00FE090399FFC3FF83A1F B87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I<38FF 87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FFC038 0F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E03807FF C03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2 EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A51207EB E300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01FC1A 167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C 6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<3AFFE3FF87F8A23A1F807C00C0 D80FC0EB0180147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB9038FF 07DC6C14F8EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>I<39FFF07FC0A2390F C01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F838 0381FC48C67E000E137E39FF81FFE0A21B167F951E>I<39FFF01FE0A2390FC00600A2EBE00E00 07130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA2130CA2 5B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>I<387FFFF0A2387C07E038700FC0 EA601F00E0138038C03F005B13FEC65A1201485AEBF0301207EA0FE0EBC070EA1F80003F1360EB 00E0EA7E03B5FCA214167E9519>I E /Fj 64 123 df<90380FC3E090387FEFF09038E07C7838 01C0F8D8038013303907007000A7B61280A23907007000B0387FE3FFA21D20809F1B>11 DII<90 380F80F890387FE7FE9038E06E063901C0FC0F380380F8380700F00270C7FCA6B7FCA239070070 07B03A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A2123812301260 1240060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A2 12E0AC1260A21270A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I< 12C012607E7E121C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A2 13005A12065A121C12185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A212381230 12601240060E7C840D>44 DI<127012F8A3127005057C840D>I<1303A2 13071306A2130E130CA2131C1318A213381330A213701360A213E013C0A212011380A312031300 A25A1206A2120E120CA2121C1218A212381230A212701260A212E05AA2102D7DA117>II<130EA2131E133EA2136E13EE13CEEA018E1203130E1206 120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>52 D<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A7 6CC7FC121F7D9D17>55 D57 D<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200 AA127012F8A312781218A41230A21260A21240051D7C930D>I63 D65 DI<90381FC04090387FF0C03801F8393803C00D380780 07380F0003121E003E1301123C127C1400127812F81500A8007814C0127CA2123C003EEB018012 1E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D9F21>IIII<90380FC02090387FF8603901F81CE03803E00638078003380F0001121E14005A127C156012 7812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E38078003EA03E03901F80E6039007FFC 2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90B5FCA2EB800FAD39FFF8FFF8A2 1D1F7E9E22>II76 DI<39FF807FF813C00007EB07809038E00300A2EA06F0A21378133CA2 131EA2130FA2EB078314C31303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0801D1F 7E9E22>III<3807E080EA0FF9EA1C1FEA300FEA70 07EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A213 0112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81F812217D9F19>83 D<007FB512E0A238780F010070130000601460A200E0147000C01430A400001400B23807FFFEA2 1C1F7E9E21>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800FC006A3913819E00ED8 03C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60A390393D801EC0A390 383F000F6D5CA3010E6DC7FCA32C207F9E2F>87 D<397FF83FF8A23907C00F800003EB06003801 E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C13078080130DEB1DF0EB18F8EB 3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FFE01FFEA21F1F7F9E 22>I92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00 E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>III I<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FC A216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I< 13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C 28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E 1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20 809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0 A223147F9326>IIII<3803E1 80EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA00 03A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318 A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01 F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA2 13FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14 E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E14 7F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA 01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E 13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3 007FC7FC123C151D7F9318>II E /Fk 37 121 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA25AA412F85A A97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01E0A2 13C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F123B12 03A21206A2120E120C12181270122008107C860F>44 D<1238127C12FEA3127C12381200A61238 127C12FEA3127C123807147C930F>58 D63 D<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F 48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E9E28>65 D<903807FC0290383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406 127EA200FE1400A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383F FF80903807FC001F1F7D9E26>67 DII73 D77 DIII 82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86CB4FC14C0 6C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE01E038E7 FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0140E00C0 1406A400001400B10007B512C0A21F1E7E9D24>I88 D97 D99 DII I104 D<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217EA00E>I107 DI<3AFE0FE03F80 90393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8FFEA227147D932C >I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D>I<48B4FC000713 C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F03807FFC0000113 0017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA6143E147E147CEB80 FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I<3AFFE7FE1FE0A2 3A1F00F007006E7ED80F801306A23907C1BC0CA214BE3903E31E18A23901F60F30A215B03900FC 07E0A290387803C0A3903830018023147F9326>119 D<38FFE1FFA2380F80706C6C5A6D5A3803 E180EA01F36CB4C7FC137E133E133F497E136FEBC7C0380183E0380381F0380701F8380E00FC39 FF81FF80A219147F931C>I E /Fl 73 124 df<90380F83E090387FE7F09038F07E783801C0F8 EA0380EC7000EA0700A8B612C0A23907007000B1397FE3FF80A21D2380A21C>11 DII<903807E03F90393FF0FF809039F03BC1C03A01C01F00E039 03803E01A23A07001C00C01600A7B712E0A23907001C011500B03A7FF1FFCFFEA2272380A229> I34 D<127012F812FCA2127C120CA41218A21230A212601240060F7CA20E>39 D<1330136013C0EA0180EA03005A1206120E120C121C12181238A212301270A3126012E0AE1260 1270A312301238A21218121C120C120E120612077EEA0180EA00C0136013300C327DA413>I<12 C012607E7E7E120E120612077E1380120113C0A2120013E0A313601370AE136013E0A313C01201 A21380120313005A1206120E120C5A5A5A5A0C327DA413>I<127012F812FCA2127C120CA41218 A21230A212601240060F7C840E>44 DI<127012F8A3127005057C840E> IIIIII<130EA2131EA2133EA2 136E13EE13CE1201138EEA030E12071206120E120C1218A212301270126012E0B512F8A238000E 00A73801FFF0A215217FA018>I<00101380EA1C07381FFF005B5B13F00018C7FCA613F8EA1BFE EA1F0F381C0780EA180314C0EA000114E0A4126012F0A214C0EAC0031260148038300700EA1C1E EA0FFCEA03F013227EA018>I<137E48B4FC3803C180380701C0EA0E03121CEB018048C7FCA212 7812701320EAF1FCEAF3FEEAF60738FC038000F813C0130112F014E0A51270A3003813C0130300 181380381C0700EA0E0EEA07FCEA01F013227EA018>I<12601270387FFFE0A214C0EA600038E0 018038C00300A21306C65AA25BA25BA25BA213E0A3485AA51203A86C5A13237DA118>III<127012F8A312701200AB127012F8A3127005157C940E>I< 127012F8A312701200AB127012F8A312781218A41230A3126012E01240051F7C940E>I61 D<497E497EA3497EA3497E130CA2EB1CF8EB1878A2EB383C 1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F0001F130139 FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003F0380780 0148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C1460123E12 1E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>IIII<903807F00890383FFC189038FC0E3838 01E0033903C001F83807800048C71278121E15385AA2007C14181278A212F81500A6EC1FFF1278 007CEB0078A2123CA27EA27E6C7E6C6C13F83801F0013900FC079890383FFE08903807F8002024 7DA226>I<39FFFC3FFFA239078001E0AD90B5FCA2EB8001AF39FFFC3FFFA220227EA125>II76 DI<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7F A2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000 EAFFF0156020227EA125>II< B512F014FC3807803FEC0F801407EC03C0A215E0A515C0A2EC0780140FEC3F00EBFFFC14F00180 C7FCADEAFFFCA21B227EA121>I82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA 3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF8 0138EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200 E0141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<39FFFC0FFFA239078001F8EC00 60B3A5000314406D13C0A20001EB01803900E00300EB7007EB3C1EEB1FF8EB07E020237EA125> I<3BFFF03FFC07FEA23B0F0007C001F00203EB00E01760D807806D13C0A33B03C007F001801406 A216032701E00C781300A33A00F0183C06A3903978383E0CEC301EA2161C90393C600F18A39039 1EC007B0A3010F14E0EC8003A36D486C5AA32F237FA132>87 D<397FF807FFA23907E003F00003 EB01C000015C3800F00301F890C7FCEB7806EB7C0EEB3E0CEB1E18EB1F38EB0FB0EB07E0A21303 1301497E80EB0778EB0E7CEB0C3EEB1C1EEB181F496C7EEB700701607F496C7E0001130101807F 00031300D80FC07F3AFFE007FFC0A222227FA125>I92 D 97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C141C141EA7141C 143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II<14 E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA38001278127012F0A712701278EA3801EA 3C03381E0EF0380FFCFEEA03F017237EA21B>II< 133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700B1EA7FF8A2102380A20F>I<14F03801F1 F83807FFB8380F1F38381E0F00EA1C07003C1380A5001C1300EA1E0FEA0F1EEA1FFCEA19F00018 C7FCA2121CEA1FFF6C13C04813E0383801F038700070481338A400701370007813F0381E03C038 0FFF803801FC0015217F9518>I<120E12FEA2121E120EAAEB1F80EB7FC0380FC1E0EB80F0EB00 70120EAE38FFE7FFA218237FA21B>I<121C121E123E121E121CC7FCA8120E12FEA2121E120EAF EAFFC0A20A227FA10E>II<120E12FEA2121E120EAAEB0FFCA2EB07E0EB0380EB0700 130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA217237FA21A >I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF809039C0F303 C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038FE7FC0 38FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II<380E1F8038FE7FE038FFC1F0380F0078120E143CA2141EA714 3CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FCA8EAFFE0A2171F7F941B>I<3801F060EA 07FCEA1F06381C03E0EA3C01EA7800A25AA712781301123C1303EA1F0EEA0FFCEA03F0C7FCA8EB 0FFEA2171F7E941A>I II<1206A5120EA3121E123EEAFFF8A2EA0E00AA13 0CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E1370AC14F0 1301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A26C13C0A338 038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2391E01C07C D81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D81980141DA2 EBF00F00001400A2497EEB600620157F9423>I<387FC1FFA2380780F8000313E03801C1C01480 3800E3001377133E133C131C133E13771367EBC3803801C1C0380380E0380700F0EA0F8038FFC1 FFA2181580941A>I<38FFC3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2 EA00E6A3137CA31338A21330A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>I< 383FFFC0A2383C038038380700EA300EEA701EEA603C13385BEA00F0485A3803C0C01380EA0700 5AEA1E01001C1380EA3803EA7007B5FCA212157F9416>II E /Fm 44 122 df<903901FF81FF011F01EF13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13 F8D807F013FCA2EE07E0020190C7FCA6B712F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7F A928>11 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FE A317277BA622>III<14075C5C5C5C5CA25B5B497E13 0F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F 13F8A31D277EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC038 0FFFF8EB80FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F 007814806CEBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C015 80A215003870001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201 A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F 90393FF007FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503 A3484891C7FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F8 90393FF007F0010FB512C0010391C7FC9038003FF829297CA832>II73 D77 DIII82 D<90387F80603903FFF0E0000F13FF381F807F383F001F003E1307007E1303127C00FC1301A214 007E7E6D130013F8EBFF806C13F814FE6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F 1407A200E013031401A37E15F06C13036C14E0B413079038E01FC090B5120000E05B38C01FF01D 297CA826>I<007FB712C0A39039807FC03FD87C00140700781503A20070150100F016E0A24815 00A5C71500B3A490B612E0A32B287EA730>I<48B47E000F13F0381F81FC486C7E147FA2EC3F80 A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F 83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<90 38FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80 FE001F5B380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14 F0121F397E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038 007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3 A3EAFFFEA30F2B7DAA14>105 DIII<3BFFC07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F90 3ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FF C07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B 7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A2 5DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26> I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C 13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FFC000C11300161B 7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438A73803F8700001 13F03800FFE0EB1F8015267FA51B>I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F0 3BFF803801FFF338003FC3211B7D9A26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEB FC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D 5A211B7F9A24>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3FF8 6D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01FFF 80A3211B7F9A24>120 D<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C 6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA38 07007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fn 11 118 df49 D<913A03FF800380023FEBF00749B5EAFC0F0107ECFF1F011F9038803FBF903A3F F80007FFD9FFE07F48497F48497F4890C8127F4848153F49151F121F49150F123F5B007F1607A3 4992C7FC12FFAB127F7FEF0780A2123F7F001F160F6D1600120F6D5D6C6C153E6C6D5C6C6D14FC 6C6D495AD93FF8495A903A1FFF801FC0010790B55A01014AC7FCD9003F13F80203138031337BB1 3C>67 D80 D97 D99 D105 D<2703F007F8EB0FF000FFD93FFFEB7FFE4A6DB5FC903CF1F03FC3E07F8090 3CF3C01FE7803FC0260FF780EBEF0000079026000FFEEB1FE001FE5C495CA2495CB2B500C1B500 83B5FCA440207D9F45>109 D<3903F007F800FFEB3FFF4A7F9039F1F03FC09039F3C01FE0380F F7800007496C7E13FE5BA25BB2B500C1B51280A429207D9F2E>II<1378A513F8A41201A212031207120F381FFFFEB5FCA33807F800AF140FA7141F 3803FC1EEBFE3E3801FFFC38007FF0EB1FC0182E7EAD20>116 DI E /Fo 27 120 df<1238127C12FEA212FF127F123B1203A41206A2120CA21218 12381270122008137B8611>44 D<1318133813F8120712FF12F81200B3AD487E387FFFF0A21428 7CA71E>49 D<137F3801FFC0380781F0380E00F80018137C121E003F137EEB803EA3381F007E00 0E137CC7FCA25C5C495AEB07C001FFC7FCA2EB01E06D7E147C80A280A21580123C127EB4FCA315 00485B007C133E00305B001C5B380F01F06CB45AC690C7FC19297EA71E>51 D<00181318001F13F8EBFFF014E014C01400EA1BF80018C7FCA8137E3819FF80381F83C0381E01 E0381C00F0001813F8C71278147CA2147EA31238127C12FEA25A147C12F0006013F81270383001 F0381803E0380E07C03807FF00EA01FC17297DA71E>53 D<137F3801FFC03807C1E0380F007000 1E7F001C133C003C131C48131EA200F87FA41580A41278141F127C003C133F121C001E136F6C13 CF3807FF8F0001130FD8001013001300A2141EA2121E003F5BA25C1470003E5B381801C0380E07 80D807FEC7FCEA01F819297EA71E>57 D69 D<02FF13100107EBE03090391FC0707090387E001C01F8EB0E F048481303485A4848130148481300A248C812705A123E1630127E127CA200FC1500A84AB5FC12 7C007E90380007F01503123EA2123F7E6C7EA26C7E6C7E6C6C13076C7E017E131C90391FC07870 903907FFE0100100EB8000282B7DA92F>71 D76 DI83 D87 D97 D99 D<140F49B4FCA2EB001F 80AC137E3801FFCF3807C0EF380F003F001E7F487FA2127C127812F8A81278127C123C5C6C5B6C 5B3907C1EF803903FF8FF83800FE0F1D2A7EA921>I<140F3901FC3F803907FF73C0380F07E338 1E03C3003EEBE180393C01E000007C7FA6003C5BEA3E03001E5B381F078001FFC7FCEA39FC0030 C8FCA21238123C383FFFC06C13F86C7F487F3838003F48130FEC0780481303A40070EB07000078 5B6C131E380F80F86CB45AC613801A287E9A1E>103 DI<120FEA1F80A213C0 1380A2EA0F00C7FCA8EA0780127FA2120F1207B3A2EAFFF8A20D297FA811>I107 DI<3A0783F801FC3AFF8FFE07FF903A9C 0F0E07803B0FB0079803C03B07E003F001E001C013E0A2018013C0B13BFFFC7FFE3FFFA2301A7F 9933>I<380783F838FF8FFCEB9C1E380FB00F3907E0078013C0A21380B139FFFCFFFCA21E1A7F 9921>I<137F3801FFC03807C1F0380F0078001E7F487FA2487FA200F81480A800781400007C5B 003C131EA26C5B6C5B3807C1F03801FFC06C6CC7FC191A7E991E>I<380783F838FF8FFEEBBC1F 390FE00780D807C013C090388003E0140115F0A2EC00F8A8EC01F0A215E0EBC003EC07C09038E0 0F809038B83F00EB8FFCEB83F00180C7FCAAEAFFFCA21D267F9921>I<380787C038FF9FE0EBB9 F0EA0FF1EA07E1EBC0E01400A25BAF7FEAFFFEA2141A7F9917>114 D<3807F840381FFFC0EA3C 07EA7003EA6001EAE000A36C1300127EEA7FF0EA3FFC6CB4FC0007138038003FC0130738C001E0 13007EA36C13C0EAF80138FE078038C7FF00EA83F8131A7E9918>I<390780078000FF13FFA200 0F130F00071307AF140FA2141F0003133F9038E077C03900FFC7FCEB7F071E1A7F9921>117 D<3AFFF1FFC3FFA23A0F803E00FCD9001C13703807801E023E1360A22603C03F13C01467A2D801 E0EB818014C3A201F013C32600F1811300A201F913E790387B00E6A2017F13FE013E137CA3011E 1378011C1338A2281A7F992B>119 D E /Fp 10 118 df 67 D80 D97 D99 D105 D<2701F803F8EB03F800FFD91FFFEB1FFF913B7C0FC07C0FC0913BE0 07E0E007E03C07F9C003E1C0032601FB80D9F3807FD9FF00EBF70049D901FE6D7EA2495CA3495C B3A4486C496C497EB500F0B500F0B512F0A344267EA549>109 D<3901F807F800FFEB1FFE9138 781F809138E00FC03A07F9C007E03801FB80EBFF00496D7EA25BA35BB3A4486C497EB500F1B512 E0A32B267EA530>II<1318A51338A41378A213F8A2120112031207001FB5FCB6FCA2D801F8C7FC B2EC0180AA3800FC031500137CEB7E07EB3F0EEB0FFCEB03F019367EB421>116 DI E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 392 509 a Fp(P)n(oin)n(t)29 b(to)g(P)n(oin)n(t)g(Comm)n(unication)843 656 y Fo(Marc)20 b(Snir)554 731 y(William)d(Gropp)k(and)f(Ewing)f(Lusk)772 848 y(Marc)n(h)i(15,)e(1993)164 1068 y Fn(1)83 b(P)n(oin)n(t)27 b(to)h(P)n(oin)n(t)f(Comm)n(unication)164 1192 y Fm(1.1)70 b(In)n(tro)r(duction)164 1285 y Fl(This)15 b(section)f(is)h(a)g(draft)g(of)g (the)g(curren)o(t)f(prop)q(osal)i(for)f(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o (unic)o(a-)164 1345 y(tion.)21 b(It)15 b(do)q(es)h(not)g(y)o(et)f(include)f (a)i(description)f(of)h(the)f(F)l(ortran)h(77)g(and)g(C)g(bindings.)237 1405 y(I)c(ha)o(v)o(e)g(tried)f(to)i(indicate,)e(wherev)o(er)g(appropriate,)j (gaps)f(and)g(unresolv)o(ed)e(issues,)164 1465 y(using)17 b(small)d(t)o(yp)q (e.)164 1646 y Fk(Discussion:)54 b Fj(The)19 b(follo)o(wing)h(subsections)f (of)f(the)h(in)o(tro)q(duction)h(con)o(tain)f(general)g(in-)164 1706 y(formation)d(on)h(the)g(design)h(of)f(MPI)g(pro)q(cedures.)26 b(The)17 b(material)h(should)g(b)q(e)f(mo)o(v)o(ed)g(to)f(a)164 1766 y(general)g(in)o(tro)q(duction)g(for)e(the)i(en)o(tire)f(do)q(cumen)o (t.)164 2031 y Fm(1.2)70 b(Data)23 b(T)n(yp)r(es)164 2123 y Fi(1.2.1)55 b(Handle)164 2216 y Fl(MPI)19 b(pro)q(cedures)h(use)g(at)g(v)m (arious)h(places)e Fh(hand)r(les)p Fl(.)34 b(Handles)19 b(are)h(used)g(to)g (access)164 2276 y(opaque)c(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(ject)f(can)i (b)q(e)f(created,)f(up)q(dated)i(and)f(destro)o(y)o(ed)g(only)f(b)o(y)164 2336 y(b)o(y)h(calling)f(suitable)h(MPI)g(pro)q(cedures,)g(and)h(pro)o (viding)f(the)g(handle)h(as)g(parameter.)164 2396 y(Opaque)11 b(ob)s(jects)g(hide)g(from)f(the)h(user)g(the)g(in)o(ternal)f(represen)o (tation)g(used)i(for)f(v)m(arious)164 2457 y(MPI)17 b(ob)s(jects,)h(th)o(us)f (allo)o(wing)h(to)g(ha)o(v)o(e)f(similar)f(calls)h(in)g(C)h(and)h(F)l (ortran,)f(allo)o(wing)961 2599 y(1)p eop %%Page: 2 2 bop 164 307 a Fl(to)15 b(o)o(v)o(ercome)e(problems)g(with)i(the)g(t)o(yping)g (rules)f(in)h(these)f(languages,)j(and)e(allo)o(wing)164 367 y(for)j(future)f(extension)g(of)g(their)g(functionalit)o(y)l(.)23 b(Handles)17 b(are)h(of)f(t)o(yp)q(e)g Fg(void)25 b(*)17 b Fl(in)g(C)164 428 y(and)g(of)f(t)o(yp)q(e)g Fg(integer)e Fl(in)i(F)l(ortran.) 237 488 y(An)21 b(opaque)h(ob)s(ject)f(can)h(b)q(e)f Fh(p)n(ersistent)h Fl(or)g Fh(ephemer)n(al)p Fl(.)37 b(A)21 b(p)q(ersisten)o(t)g(ob)s(ject)164 548 y(p)q(ersists)12 b(un)o(til)f(destro)o(y)o(ed)f(b)o(y)h(an)i(explicit)c (op)q(eration.)21 b(An)11 b(ephemeral)e(ob)s(ject)i(is)h(go)q(o)q(d)164 608 y(for)17 b(a)g(single)g(use;)f(th)o(us)h(an)h(ephemeral)c(ob)s(ject)i (asso)q(ciated)i(with)f(a)g(comm)o(unicati)o(on)164 668 y(op)q(eration)f (disapp)q(ears)g(once)f(this)h(op)q(eration)g(is)f(completed)e(\(or)i(once)g (this)g(ob)s(ject)g(is)164 729 y(not)i(needed)e(an)o(ymore)g(for)i(the)f (completion)e(of)i(the)g(op)q(eration\).)237 789 y(An)j(opaque)g(ob)s(ject)f (is)h(created)f(b)o(y)g(a)h(call)f(to)i Fg(MPI)p 1213 789 16 2 v 17 w(CREATE)p Fl(,)c(and)j(destro)o(y)o(ed)f(b)o(y)164 849 y(a)g(call)e(to)h Fg(MPI)p 437 849 V 18 w(FREE)p Fl(.)e(Additional)i(MPI) f(functions)h(are)h(a)o(v)m(ailable)e(to)i(create,)e(access)164 909 y(and)h(up)q(date)g(sp)q(eci\014c)e(opaque)i(ob)s(jects.)164 1029 y Fi(MPI)p 279 1029 17 2 v 20 w(CREA)-5 b(TE\(handle,)20 b(t)n(yp)r(e,)d(p)r(ersistence\))164 1121 y(OUT)i(handle)25 b Fl(handle)16 b(to)h(ob)s(ject)164 1219 y Fi(IN)h(t)n(yp)r(e)24 b Fl(state)d(v)m(alue)f(that)h(iden)o(ti\014es)e(the)i(t)o(yp)q(e)f(of)g(ob)s (ject)g(to)h(b)q(e)g(created)f(\(e.g.,)286 1279 y Fg(MPI)p 367 1279 16 2 v 17 w(COMMUNICATI)o(ON)o(,)j(MPI)p 847 1279 V 17 w(BUFFER,)g(MPI)p 1147 1279 V 18 w(CONTEXT)p Fl(,)13 b(etc.\).)164 1377 y Fi(IN)18 b(p)r(ersistence)24 b Fl(state)16 b(v)m(alue;)g(either)f Fg(MPI)p 1020 1377 V 17 w(PERSISTENT)e Fl(or)j Fg(MPI)p 1447 1377 V 18 w(EPHEMERAL)o Fl(.)164 1529 y Fi(MPI)p 279 1529 17 2 v 20 w(FREE\(handle\))164 1620 y(IN)i(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)237 1712 y(An)h(ob)s(ject)f(can)h(b)q(e)g(destro)o(y)o(ed)f (only)h(if)f(there)h(is)f(no)i(p)q(ending)f(op)q(eration)h(that)f(is)164 1772 y(using)h(this)g(ob)s(ject;)f(after)h(successful)f(return)g(of)h(the)g (routine,)f(the)h(handle)f(is)h(unde-)164 1832 y(\014ned.)164 1952 y Fi(MPI)p 279 1952 V 20 w(ASSOCIA)-5 b(TED\(handle,)21 b(t)n(yp)r(e\))164 2044 y(IN)d(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)164 2142 y Fi(OUT)j(t)n(yp)r(e)k Fl(state)237 2233 y(Returns)16 b(the)g(t)o(yp)q(e)f(of)h(the)g(ob)s(ject)f(the)h(handle)f(is)h(curren)o(tly) e(asso)q(ciated)j(with,)e(if)164 2293 y(suc)o(h)g(exists.)20 b(Returns)c(the)f(sp)q(ecial)g(t)o(yp)q(e)f Fg(MPI)p 1041 2293 16 2 v 18 w(NULL)g Fl(if)g(the)h(handle)h(is)f(not)h(curren)o(tly)164 2354 y(asso)q(ciated)h(with)f(an)o(y)g(ob)s(ject.)237 2414 y(MPI)h(ma)o(y)f(pro)o(vide)h(prede\014ned)h(opaque)g(ob)s(jects)f(and)h (prede\014ned,)g(static)f(han-)164 2474 y(dles)f(to)g(these)g(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(jects)g(ma)o(y)f(not)h(b)q(e)h(destro)o(y)o(ed.)961 2599 y(2)p eop %%Page: 3 3 bop 164 307 a Fi(List)13 b(of)g(handles)50 b Fl(An)11 b(MPI)g(call)f(ma)o(y)g (need)h(a)g(parameter)f(that)i(is)f(a)h Fh(list)h(of)g(hand)r(les)p Fl(.)164 367 y(In)k(C,)g(suc)o(h)g(list)g(will)f(b)q(e)h(a)h(record)f(with)g (one)h(comp)q(onen)o(t)e(b)q(eing)h(the)g(length)h(of)f(the)164 428 y(list,)j(the)g(other)g(comp)q(onen)o(ts)f(b)q(eing)h(an)h(arra)o(y)f(of) g(p)q(oin)o(ters.)33 b(In)20 b(F)l(ortran,)h(the)f(list)164 488 y(will)15 b(b)q(e)h(an)h(arra)o(y)g(of)f(in)o(tegers,)f(the)h(\014rst)h (one)f(of)h(whic)o(h)e(is)h(the)g(length)g(of)h(the)f(list.)164 657 y Fk(Discussion:)54 b Fj(The)19 b(mec)o(hanism)h(for)e(opaque)h(ob)s (jects)f(used)h(here)g(follo)o(ws)g(the)g(POSIX)164 714 y(F)l(ortran)14 b(binding)j(standard.)j(An)15 b(alternativ)o(e)h(c)o(hoice)g(is)g(to)f(ha)o (v)o(e)f(di\013eren)o(t)i(t)o(yp)q(e)f(declara-)164 770 y(tions)g(for)f(eac)o (h)h(t)o(yp)q(e)h(of)e(opaque)h(ob)s(ject.)k(Then,)d(opaque)f(ob)s(jects)f (are)h(created/destro)o(y)o(ed)164 826 y(lik)o(e)e(regular)f(v)m(ariables,)h (rather)f(than)f(b)o(y)h(MPI)g(calls;)i(they)e(are)f(still)j(accessed)e(and)g (up)q(dated)164 883 y(only)k(via)f(MPI)g(functions.)164 1193 y Fi(1.2.2)55 b(State)164 1286 y Fl(MPI)12 b(pro)q(cedures)i(use)e(at)i(v)m (arious)f(places)g(argumen)o(ts)f(with)h Fh(state)g Fl(t)o(yp)q(es.)20 b(The)13 b(v)m(alues)164 1346 y(of)j(suc)o(h)f(data)i(t)o(yp)q(e)e(are)g(all) g(iden)o(ti\014ed)g(b)o(y)g(names,)f(and)i(no)g(op)q(eration)h(is)e (de\014ned)g(on)164 1406 y(them.)k(F)l(or)14 b(example,)f(the)h Fg(MPI)p 759 1406 16 2 v 17 w(CREATE)e Fl(routine)i(has)i(a)e(state)h(t)o(yp) q(e)f(parameter)f(with)164 1466 y(v)m(alues)j Fg(MPI)p 390 1466 V 18 w(PERSISTEN)o(T)d Fl(and)k Fg(MPI)p 853 1466 V 17 w(EPHEMERAL)p Fl(.)237 1527 y(An)e Fg(enumeratio)o(n)d Fl(declared)j(in)g(an) h(included)e(MPI.h)g(\014le)h(will)f(b)q(e)i(used)f(in)g(C)h(for)164 1587 y(state)g(datat)o(yp)q(es.)22 b(The)17 b(F)l(ortran)f(77)h(mec)o(hanism) c(needs)j(to)h(b)q(e)f(decided.)164 1767 y Fk(Discussion:)67 b Fj(Named)22 b(in)o(teger)h(constan)o(ts)e(can)h(b)q(e)g(used)h(in)g(F)l (ortran)d(90,)j(using)g(the)164 1827 y Ff(PARAMETER)17 b Fj(mec)o(hanism.)28 b(The)18 b(constan)o(t)f(declarations)i(can)f(b)q(e)g(made)g(a)o(v)m(ailable) i(via)e(an)164 1888 y Ff(INCLUDE)e Fj(\014le.)25 b(F)l(ortran)16 b(77)g(do)q(es)h(not)f(seem)h(to)f(o\013er)g(an)o(y)h(con)o(v)o(enien)o(t)g (mec)o(hanism.)25 b(One)164 1948 y(p)q(ossibilit)o(y)c(is)e(to)f(sp)q(ecify)j (explicit)g(in)o(teger)e(v)m(alues,)h(and)f(allo)o(w)g(the)g(use)g(of)f (named)i(con-)164 2008 y(stan)o(ts)15 b(with)i(those)f(F)l(ortran)f(77)h (compilers)i(that)d(supp)q(ort)i(them)f(con)o(v)o(enien)o(tly)l(.)25 b(Another)164 2068 y(p)q(ossibilit)o(y)17 b(is)f(to)f(use)g(c)o(haracter)g (strings,)f(rather)h(than)g(in)o(tegers.)164 2319 y Fi(1.2.3)55 b(Named)18 b(constan)n(ts)164 2411 y Fl(MPI)j(pro)q(cedures)i(sometime)o(s)c (assign)k(a)g(sp)q(ecial)e(meaning)g(to)h(a)g(sp)q(ecial)g(v)m(alue)g(of)164 2471 y(a)f(basic)g(t)o(yp)q(e)f(parameter;)h(e.g.)34 b Fg(tag)20 b Fl(is)h(an)g(in)o(teger)e(v)m(alued)i(parameter)f(of)h(p)q(oin)o(t-)961 2599 y(3)p eop %%Page: 4 4 bop 164 307 a Fl(to-p)q(oin)o(t)23 b(comm)o(unic)o(ation)d(op)q(erations,)25 b(with)d(a)h(sp)q(ecial)f Fg(DONTCARE)d Fl(v)m(alue.)39 b(Suc)o(h)164 367 y(parameters)14 b(will)f(ha)o(v)o(e)h(a)h(range)g(of)g(regular)f(v)m (alues,)h(whic)o(h)f(is)g(a)h(prop)q(er)g(subrange)h(of)164 428 y(the)i(range)h(of)g(v)m(alues)f(of)h(the)f(corresp)q(onding)h(basic)f(t) o(yp)q(e;)h(sp)q(ecial)f(v)m(alues)g(\(suc)o(h)g(as)164 488 y(DONTCARE\))d(will)e(b)q(e)i(outside)g(the)g(regular)g(range.)21 b(The)15 b(range)g(of)g(regular)g(v)m(alues)164 548 y(can)d(b)q(e)g(queried,) f(and)h(sometimes)d(set,)j(using)g(en)o(vironmen)o(t)d(inquiry)i(or)h(en)o (vironmen)o(t)164 608 y(setting)21 b(functions)g(\(Section)f Fi(??)p Fl(\).)37 b(The)21 b(sp)q(ecial)f(v)m(alues)h(are)g(pro)o(vided)g(b)o (y)f(named)164 668 y(constan)o(t,)c(that)h(are)f(made)f(a)o(v)m(ailable)h (via)g(an)g(MPI.h)g(include)f(\014le)g(in)h(a)h(C)f(binding.)164 838 y Fk(Discussion:)237 894 y Fj(Need)e(to)e(agree)g(on)h(a)g(F)l(ortran)f (mec)o(hanism)h(for)f(named)h(constan)o(ts)f(\(see)h(the)g(discussion)164 951 y(ab)q(o)o(v)o(e\).)237 1007 y(Implemen)o(ters)19 b(should)g(try)e(to)g (detect)h(illegal)i(uses)e(of)g(\\sp)q(ecial)h(v)m(alues".)29 b(Th)o(us,)18 b(the)164 1064 y(use)e(of)e(the)i Ff(DONTCARE)e Fj(v)m(alue)i(to)f(tag)f(a)h(message)g(sen)o(t)f(should)j(b)q(e)f(\015agged)f (as)f(an)h(error.)164 1374 y Fi(1.2.4)55 b(Choice)164 1466 y Fl(MPI)13 b(functions)i(sometime)o(s)d(use)i(parameters)f(with)g(a)i Fh(choic)n(e)f Fl(\(or)h(union\))f(data)h(t)o(yp)q(e.)164 1527 y(I.e.,)10 b(distinct)g(calls)g(to)i(the)e(same)g(routine)h(ma)o(y)e(pass)j (b)o(y)f(reference)e(actual)i(parameters)164 1587 y(of)18 b(di\013eren)o(t)f (t)o(yp)q(es.)25 b(The)17 b(mec)o(hanism)e(for)i(pro)o(viding)h(suc)o(h)f (parameters)g(will)f(di\013er)164 1647 y(from)e(language)i(to)g(language.)22 b(In)15 b(C,)g(a)h(formal)e(parameter)g(of)i(t)o(yp)q(e)e Fg(void)25 b(*)15 b Fl(will)f(b)q(e)164 1707 y(used,)i(with)g(an)h(actual)f(p)q(oin)o (ter)g(parameter.)k(in)c(F)l(ortran,)g(w)o(e)g(shall)g(c)o(heat.)164 1876 y Fk(Discussion:)237 1933 y Fj(The)i(F)l(ortran)e(77)h(standard)g(sp)q (eci\014es)i(that)d(the)i(t)o(yp)q(e)f(of)g(actual)h(argumen)o(ts)e(need)i (to)164 1989 y(agree)f(with)g(the)g(t)o(yp)q(e)h(of)e(dumm)o(y)h(argumen)o (ts;)g(no)g(construct)g(equiv)m(alen)o(t)i(to)e(C)f(p)q(oin)o(ters)164 2046 y(is)e(a)o(v)m(ailable.)21 b(Th)o(us,)14 b(it)g(w)o(ould)g(seem)g(that)f (there)g(is)i(no)e(standard)g(conforming)h(mec)o(hanism)164 2102 y(to)g(supp)q(ort)h(c)o(hoice)g(parameters.)k(Ho)o(w)o(ev)o(er,)14 b(most)g(F)l(ortran)f(compiler)j(either)f(don't)g(c)o(hec)o(k)164 2159 y(t)o(yp)q(e)i(consistency)h(of)f(calls)h(to)e(external)h(routines,)h (or)e(supp)q(ort)h(a)g(sp)q(ecial)i(mec)o(hanism)e(to)164 2215 y(link)i(foreign)e(\(e.g.,)g(C\))f(routines.)27 b(I)18 b(suggest)e(that)h(w)o (e)g(accept)h(this)f(nonconformit)o(y)h(with)164 2272 y(F)l(ortran)10 b(77)h(standard.)18 b(I.e.,)12 b(w)o(e)f(accept)h(that)f(the)g(same)h (routine)g(ma)o(y)e(b)q(e)j(passed)e(an)h(actual)164 2328 y(parameter)i(of)h (a)g(di\013eren)o(t)g(t)o(yp)q(e)h(at)e(distinct)j(calls.)237 2385 y(Generic)g(routines)g(can)f(b)q(e)h(used)f(in)h(F)l(ortran)e(90)g(to)h (pro)o(vide)g(a)g(standard)g(conforming)164 2441 y(solution.)j(This)12 b(solution)g(will)g(b)q(e)g(consisten)o(t)f(with)g(our)g(nonstandard)g (conforming)g(F)l(ortran)961 2599 y Fl(4)p eop %%Page: 5 5 bop 164 307 a Fj(77)15 b(solution.)164 632 y Fm(1.3)70 b(Pro)r(cesses)164 725 y Fl(An)16 b(MPI)h(program)f(is)h(executed)e(b)o(y)h(sev)o(eral)g (autonomous)h(pro)q(cesses)g(that)g(execute)164 785 y(eac)o(h)f(their)g(o)o (wn)i(co)q(de,)e(in)h(a)g(MIMD)f(st)o(yle.)22 b(The)17 b(co)q(des)g(executed) f(b)o(y)g(eac)o(h)g(pro)q(cess)164 845 y(need)21 b(not)h(b)q(e)f(iden)o (tical.)35 b(The)21 b(pro)q(cesses)h(comm)o(unic)o(ate)d(via)i(calls)f(to)i (MPI)f(com-)164 905 y(m)o(unication)13 b(primitiv)o(es.)18 b(T)o(ypically)l(,)13 b(eac)o(h)i(pro)q(cessor)h(executes)e(in)h(its)g(o)o (wn)h(address)164 965 y(space,)i(although)g(shared-memory)e(implem)o(en)o (tati)o(ons)g(of)i(MPI)f(are)h(p)q(ossible.)26 b(This)164 1025 y(do)q(cumen)o(t)20 b(sp)q(eci\014es)h(the)f(b)q(eha)o(vior)h(of)h(a)f (parallel)g(program)g(assuming)g(that)g(only)164 1086 y(MPI)c(calls)f(are)i (used)f(for)h(comm)o(uni)o(cation.)k(The)17 b(in)o(teraction)f(of)i(an)f(MPI) g(program)164 1146 y(with)k(other)g(p)q(ossible)g(means)f(of)h(comm)o (unication)d(\(e.g.,)j(shared)h(memory\))c(is)i(not)164 1206 y(sp)q(eci\014ed.)29 b(In)18 b(particular,)h(it)g(is)f(assumed)h(that)g (message)f(bu\013ers)i(at)f(distinct)f(pro-)164 1266 y(cessors)f(are)f (disjoin)o(t.)237 1326 y(MPI)k(do)q(es)g(not)h(sp)q(ecify)e(the)h(execution)f (mo)q(del)g(for)h(eac)o(h)f(pro)q(cess.)34 b(A)19 b(pro)q(cess)164 1387 y(can)e(b)q(e)f(sequen)o(tial,)f(or)i(can)g(b)q(e)f(m)o(ultithreaded,)e (with)i(threads)h(p)q(ossibly)g(executing)164 1447 y(concurren)o(tly)l(.)23 b(Care)18 b(has)g(b)q(een)f(tak)o(en)g(to)h(mak)o(e)d(MPI)i(\\thread-safe",)i (b)o(y)d(a)o(v)o(oiding)164 1507 y(the)g(use)g(of)h(implici)o(t)d(global)i (states.)237 1567 y(The)e(initial)e(allo)q(cation)h(of)h(pro)q(cesses)g(to)g (an)g(MPI)e(computation)h(and)h(their)f(bind-)164 1627 y(ing)f(to)h(ph)o (ysical)f(pro)q(cessors)h(is)g(not)g(sp)q(eci\014ed)e(b)o(y)h(the)h(program)f (itself.)19 b(It)12 b(is)g(exp)q(ected)164 1688 y(that)17 b(v)o(endors)f (will)g(pro)o(vide)g(mec)o(hanism)o(s)e(to)j(do)g(so)g(either)f(at)h(load)g (time)d(or)j(at)g(run)164 1748 y(time.)26 b(Suc)o(h)18 b(mec)o(hanism)o(s)e (will)i(allo)o(w)g(to)h(sp)q(ecify)e(the)i(initial)e(n)o(um)o(b)q(er)f(of)j (required)164 1808 y(pro)q(cesses,)c(the)g(co)q(de)g(to)g(b)q(e)f(executed)g (b)o(y)g(eac)o(h)g(initial)g(pro)q(cess,)h(and)g(the)g(allo)q(cation)164 1868 y(of)d(pro)q(cesses)h(to)g(pro)q(cessors.)21 b(Also,)12 b(the)g(curren)o(t)f(prop)q(osal)j(do)q(es)f(not)f(pro)o(vide)f(for)i(dy-)164 1928 y(namic)j(creation)h(or)g(deletion)g(of)g(pro)q(cesses)h(during)g (program)f(execution,)f(although)164 1989 y(it)g(is)f(in)o(tended)g(to)i(b)q (e)f(consisten)o(t)g(with)g(suc)o(h)f(extension.)21 b(Finally)l(,)14 b(the)i(curren)o(t)f(pro-)164 2049 y(p)q(osal)h(do)q(es)g(not)f(sp)q(ecify)f (a)i(naming)e(sc)o(heme)e(for)k(pro)q(cesses.)21 b(W)l(e)15 b(prop)q(ose)h(to)f(alw)o(a)o(ys)164 2109 y(iden)o(tify)j(pro)q(cesses)j (according)g(to)f(their)g(relativ)o(e)e(rank)j(in)f(a)g(con)o(text)g (\(group\),)h(so)164 2169 y(that,)f(e\013ectiv)o(ely)l(,)d(pro)q(cesses)i (are)h(iden)o(ti\014ed)d(b)o(y)i(consecutiv)o(e)f(in)o(tegers.)29 b(Absolute,)164 2229 y(system-wide)13 b(unique)h(pro)q(cess)h(id's)g(are)f (\(will)g(b)q(e\))h(needed)f(only)g(if)g(dynamic)f(pro)q(cess)164 2290 y(creation)h(is)g(to)h(b)q(e)f(supp)q(orted)i(\(in)e(suc)o(h)g(ev)o(en)o (tualit)o(y)e(w)o(e)h(prop)q(ose)j(to)f(use)f(handles)h(to)164 2350 y(opaque)i Fh(pr)n(o)n(c)n(ess)f(structur)n(es)g Fl(for)g(that)h(purp)q (ose\).)961 2599 y(5)p eop %%Page: 6 6 bop 164 307 a Fm(1.4)70 b(Con)n(texts)164 520 y Fk(Discussion:)46 b Fj(This)18 b(section)f(con)o(tains)g(a)f(prop)q(osal)h(for)f(use)i(of)e (con)o(texts)g(that)g(will)i(sub-)164 580 y(sume)13 b(groups.)19 b(It)13 b(b)q(orro)o(ws)g(hea)o(vily)h(on)f(the)g(curren)o(t)g(group)g(prop)q (osal.)20 b(This)14 b(prop)q(osal)f(has)164 640 y(not)i(y)o(et)g(b)q(een)h (discussed)h(in)f(MPI)f(meetings.)237 821 y Fl(A)h Fi(con)n(text)f Fl(consists)i(of:)237 923 y Fe(\017)24 b Fl(A)d(set)g(of)h(pro)q(cesses)g (that)g(curren)o(tly)d(b)q(elong)j(to)g(the)f(con)o(text)g(\(p)q(ossibly)g (all)286 983 y(pro)q(cesses,)16 b(or)h(a)g(prop)q(er)f(subset\).)237 1084 y Fe(\017)24 b Fl(A)19 b Fi(ranking)g Fl(of)h(the)f(pro)q(cesses)g (within)g(that)g(con)o(text,)g(i.e.,)f(a)h(n)o(um)o(b)q(ering)f(of)286 1145 y(the)g(pro)q(cesses)g(in)g(that)g(con)o(text)g(from)f(0)h(to)g Fd(n)13 b Fe(\000)f Fl(1,)18 b(where)g Fd(n)g Fl(is)g(the)g(n)o(um)o(b)q(er) 286 1205 y(of)e(pro)q(cesses)h(in)f(that)h(con)o(text.)237 1307 y(A)f(pro)q(cess)h(ma)o(y)e(b)q(elong)h(to)h(sev)o(eral)e(con)o(texts)h (at)g(the)g(same)g(time.)237 1367 y(An)o(y)i(in)o(terpro)q(cess)f(comm)o (unication)f(o)q(ccurs)i(within)g(a)h(con)o(text,)e(and)i(messages)164 1427 y(sen)o(t)i(within)f(one)h(con)o(text)f(can)h(b)q(e)g(receiv)o(ed)e (only)i(within)f(the)h(same)f(con)o(text.)34 b(A)164 1487 y(con)o(text)13 b(is)h(sp)q(eci\014ed)g(using)h(a)f Fh(c)n(ontext)j(hand)r(le)f Fl(\(i.e.,)d(a)h(handle)g(to)h(an)g(opaque)f(ob)s(ject)164 1547 y(that)23 b(iden)o(ti\014es)d(a)j(con)o(text\).)38 b(Con)o(text)22 b(handles)g(cannot)h(b)q(e)f(transferred)g(for)h(one)164 1608 y(pro)q(cess)18 b(to)g(another;)g(they)f(can)h(b)q(e)f(used)h(only)f(on)h (the)f(pro)q(cess)h(where)f(they)g(where)164 1668 y(created.)237 1728 y(F)l(ollo)o(ws)f(examples)e(of)j(p)q(ossible)f(uses)h(for)f(con)o (texts.)164 1858 y Fi(Lo)r(osely)h(sync)n(hronous)i(library)g(call)h(in)n (terface)49 b Fl(Consider)16 b(the)f(case)h(where)g(a)164 1918 y(parallel)e(application)i(executes)e(a)h(\\parallel)g(call")g(to)h(a)f (library)g(routine,)g(i.e.,)e(where)164 1978 y(all)i(pro)q(cesses)i(transfer) f(con)o(trol)g(to)g(the)g(library)f(routine.)21 b(If)16 b(the)f(library)h(w)o (as)g(dev)o(el-)164 2038 y(op)q(ed)k(separately)l(,)f(then)h(one)f(should)h (b)q(ew)o(are)f(of)h(the)f(p)q(ossibilit)o(y)g(that)g(the)h(library)164 2099 y(co)q(de)12 b(ma)o(y)f(receiv)o(e)f(b)o(y)i(mistak)o(e)e(messages)h (send)i(b)o(y)e(the)h(caller)f(co)q(de,)i(and)g(vice-v)o(ersa.)164 2159 y(T)l(o)19 b(prev)o(en)o(t)f(suc)o(h)g(o)q(ccurrence)h(one)g(migh)o(t)e (use)i(a)g(barrier)f(sync)o(hronization)h(b)q(efore)164 2219 y(and)f(after)f(the)h(parallel)e(library)h(call.)24 b(Instead,)17 b(one)h(can)g(allo)q(cate)f(a)h(di\013eren)o(t)e(con-)164 2279 y(text)e(to)g(the)h(library)l(,)e(th)o(us)h(prev)o(en)o(ting)f(un)o(w)o(an)o (ted)h(in)o(terference.)k(No)o(w,)d(the)f(transfer)164 2339 y(of)j(con)o(trol)e(to)i(the)f(library)g(need)f(not)i(b)q(e)f(sync)o (hronized.)961 2599 y(6)p eop %%Page: 7 7 bop 164 307 a Fi(F)-5 b(unctional)25 b(decomp)r(osition)e(and)g(mo)r(dular)g (co)r(de)f(dev)n(elopmen)n(t)49 b Fl(Often,)164 367 y(a)18 b(parallel)f(application)g(is)h(dev)o(elop)q(ed)e(b)o(y)h(in)o(tegrating)h (sev)o(eral)e(distinct)h(functional)164 428 y(mo)q(dules,)h(that)h(is)g(eac)o (h)f(dev)o(elop)q(ed)g(separately)l(.)28 b(Eac)o(h)19 b(mo)q(dule)f(is)g(a)h (parallel)f(pro-)164 488 y(gram)13 b(that)h(runs)g(on)g(a)g(dedicated)f(set)h (of)g(pro)q(cesses,)g(and)g(the)g(computation)f(consists)164 548 y(of)j(phases)h(where)f(mo)q(dules)g(compute)e(separately)l(,)i(in)o (termixe)o(d)e(with)i(global)g(phases)164 608 y(where)i(all)g(pro)q(cesses)h (comm)o(unicate.)25 b(It)18 b(is)g(con)o(v)o(enien)o(t)f(to)i(allo)o(w)f(eac) o(h)g(mo)q(dule)g(to)164 668 y(use)g(its)g(o)o(wn)g(priv)m(ate)g(pro)q(cess)h (n)o(um)o(b)q(ering)d(sc)o(heme,)g(for)i(the)g(in)o(tramo)q(dule)f(compu-)164 729 y(tation.)27 b(This)18 b(is)f(ac)o(hiev)o(ed)g(b)o(y)g(using)h(a)h(priv)m (ate)e(mo)q(dule)g(con)o(text)g(for)h(in)o(tramo)q(dule)164 789 y(computation,)d(and)i(a)g(global)f(con)o(text)g(for)g(in)o(termo)q(dule) e(comm)o(unic)o(ation.)164 919 y Fi(Collectiv)n(e)f(comm)n(unication)51 b Fl(MPI)10 b(supp)q(orts)i(collectiv)o(e)c(comm)o(unic)o(ation)h(within)164 979 y(dynamically)h(created)i(groups)h(of)g(pro)q(cesses.)20 b(Eac)o(h)13 b(suc)o(h)f(group)h(can)g(b)q(e)f(represen)o(ted)164 1039 y(b)o(y)h(a)i(distinct)e(con)o(text.)19 b(This)c(pro)o(vides)e(a)h (simple)e(mec)o(hanism)f(to)j(ensure)g(that)g(com-)164 1099 y(m)o(unication)20 b(that)i(p)q(ertains)g(to)g(collectiv)o(e)c(comm)o (unication)h(within)i(one)h(group)g(is)164 1159 y(not)17 b(confused)f(with)g (collectiv)o(e)e(comm)o(uni)o(cation)g(within)h(another)i(group.)164 1289 y Fi(Ligh)n(t)n(w)n(eigh)n(t)i(gang)f(sc)n(heduling)50 b Fl(Consider)15 b(an)g(en)o(vironmen)o(t)d(where)i(pro)q(cesses)164 1349 y(are)j(m)o(ultith)o(treaded.)k(Con)o(texts)c(can)h(b)q(e)f(used)g(to)h (pro)o(vide)e(a)i(mec)o(hanism)c(whereb)o(y)164 1410 y(all)21 b(pro)q(cesses)i(are)f(time-shared)e(b)q(et)o(w)o(een)h(sev)o(eral)g (parallel)g(executions,)h(and)h(can)164 1470 y(con)o(text)h(switc)o(h)h(from) f(one)h(parallel)g(execution)f(to)h(another,)j(in)d(a)g(lo)q(osely)h(syn-)164 1530 y(c)o(hronous)d(manner.)39 b(A)22 b(thread)h(is)f(allo)q(cated)h(on)g (eac)o(h)f(pro)q(cess)h(to)g(eac)o(h)f(parallel)164 1590 y(execution,)14 b(and)i(a)g(di\013eren)o(t)e(con)o(text)g(is)i(used)f(to)h(iden)o(tify)d(eac) o(h)i(parallel)f(execution.)164 1650 y(Th)o(us,)f(tra\016c)g(from)e(one)i (execution)f(cannot)i(b)q(e)f(confused)f(with)h(tra\016c)g(from)e(another)164 1711 y(execution.)31 b(The)20 b(blo)q(c)o(king)g(and)h(un)o(blo)q(c)o(king)e (of)h(threads)h(due)f(to)g(comm)o(unicati)o(on)164 1771 y(ev)o(en)o(ts)14 b(pro)o(vide)g(a)i(\\lazy")f(con)o(text)f(switc)o(hing)h(mec)o(hanism)o(.)j (This)d(can)h(b)q(e)f(extended)164 1831 y(to)j(the)f(case)g(where)g(the)h (parallel)e(executions)h(are)g(spanning)i(distinct)d(pro)q(cess)i(sub-)164 1891 y(sets.)j(\(MPI)16 b(do)q(es)h(not)g(require)e(m)o(ultithreaded)e(pro)q (cesses.\))164 2061 y Fk(Discussion:)39 b Fj(A)14 b(con)o(text)f(handle)i (migh)o(t)f(b)q(e)g(implemen)o(ted)i(as)e(a)f(p)q(oin)o(ter)h(to)g(a)f (structure)164 2117 y(that)f(consists)h(of)g(con)o(text)f(lab)q(el)j(\(that)d (is)h(carried)h(b)o(y)f(messages)f(sen)o(t)h(within)h(this)g(con)o(text\))164 2173 y(and)20 b(a)g(con)o(text)g(mem)o(b)q(er)g(table,)i(that)d(translates)h (pro)q(cess)h(ranks)e(within)j(a)e(con)o(text)f(to)164 2230 y(absolute)g(addresses)g(or)g(to)f(routing)h(information.)31 b(Of)19 b(course,)h(other)f(implemen)o(tations)164 2286 y(are)c(p)q(ossible,) i(including)h(implemen)o(tations)e(that)f(do)g(not)g(require)h(eac)o(h)g(con) o(text)f(mem)o(b)q(er)164 2343 y(to)g(store)f(a)h(full)h(list)g(of)f(the)g (con)o(text)g(mem)o(b)q(ers.)237 2399 y(Con)o(texts)g(can)h(b)q(e)g(used)g (only)h(on)e(the)h(pro)q(cess)g(where)g(they)g(w)o(ere)f(created.)22 b(Since)17 b(the)164 2456 y(con)o(text)e(carries)g(information)h(on)f(the)g (group)g(of)g(pro)q(cesses)h(that)e(b)q(elong)i(to)f(this)h(con)o(text,)961 2599 y Fl(7)p eop %%Page: 8 8 bop 164 307 a Fj(a)14 b(pro)q(cess)g(can)g(send)g(a)g(message)g(within)h(a)e (con)o(text)h(only)g(to)g(other)f(pro)q(cesses)i(that)e(b)q(elong)164 364 y(to)i(that)g(con)o(text.)20 b(Th)o(us,)15 b(eac)o(h)h(pro)q(cess)f (needs)i(to)e(k)o(eep)g(trac)o(k)g(only)h(of)f(the)h(con)o(texts)f(that)164 420 y(where)j(created)f(at)g(that)g(pro)q(cess;)h(the)g(total)f(n)o(um)o(b)q (er)h(of)f(con)o(texts)f(p)q(er)i(pro)q(cess)g(is)g(lik)o(ely)164 477 y(to)d(b)q(e)g(small.)237 533 y(The)23 b(only)f(di\013erence)i(I)e(see)h (b)q(et)o(w)o(een)f(this)h(curren)o(t)f(de\014nition)i(of)e(con)o(text,)h (whic)o(h)164 589 y(subsumes)18 b(the)f(group)h(concept,)g(and)f(a)g(pared)h (do)o(wn)f(de\014nition,)j(if)e(that)e(I)i(assume)f(here)164 646 y(that)g(pro)q(cess)g(n)o(um)o(b)q(ering)h(is)g(relativ)o(e)g(to)f(the)g (con)o(text,)g(rather)g(then)g(b)q(eing)i(global,)f(th)o(us)164 702 y(requiring)f(a)e(con)o(text)g(mem)o(b)q(er)h(table.)22 b(I)16 b(argue)g(that)f(this)h(is)g(not)g(m)o(uc)o(h)f(added)i(o)o(v)o (erhead,)164 759 y(and)e(giv)o(es)h(m)o(uc)o(h)f(additional)i(needed)f (functionalit)o(y)l(.)239 853 y Fc(\017)24 b Fj(If)14 b(a)f(new)g(con)o(text) g(is)h(created)g(b)o(y)f(cop)o(ying)h(a)f(previous)h(con)o(text,)f(then)h (one)f(do)q(es)h(not)286 909 y(need)19 b(a)e(new)h(mem)o(b)q(er)g(table;)i (rather,)d(one)h(needs)h(just)e(a)h(new)g(con)o(text)f(lab)q(el)j(and)286 966 y(a)e(new)h(p)q(oin)o(ter)g(to)e(the)i(same)f(old)h(con)o(text)f(mem)o(b) q(er)g(table.)30 b(This)19 b(holds)g(true,)g(in)286 1022 y(particular,)c(for) g(con)o(texts)f(that)h(include)j(all)e(pro)q(cesses.)239 1116 y Fc(\017)24 b Fj(A)13 b(con)o(text)g(mem)o(b)q(er)h(table)g(mak)o(es)f(sure) h(that)f(a)g(message)g(is)h(sen)o(t)f(only)h(to)f(a)h(pro)q(cess)286 1172 y(that)f(can)h(execute)g(in)h(the)f(con)o(text)f(of)g(the)h(message.)19 b(The)14 b(alternativ)o(e)g(mec)o(hanism,)286 1229 y(whic)o(h)e(is)g(c)o(hec) o(king)g(at)f(reception,)i(is)f(less)g(e\016cien)o(t,)g(and)g(requires)g (that)f(eac)o(h)g(con)o(text)286 1285 y(lab)q(el)18 b(b)q(e)g(system-wide)f (unique.)26 b(This)17 b(requires)g(that,)f(to)g(the)h(least,)g(all)g(pro)q (cesses)286 1342 y(in)e(a)f(con)o(text)f(execute)i(a)f(collectiv)o(e)i (agreemen)o(t)d(algorithm)i(at)e(the)h(creation)h(of)e(this)286 1398 y(con)o(text.)239 1492 y Fc(\017)24 b Fj(The)14 b(use)g(of)f(relativ)o (e)h(addressing)g(within)h(eac)o(h)f(con)o(text)f(is)h(needed)h(to)e(supp)q (ort)h(true)286 1548 y(mo)q(dular)19 b(dev)o(elopmen)o(t)g(of)f(sub)q (computations)h(that)e(execute)i(on)f(a)g(subset)h(of)f(the)286 1605 y(pro)q(cesses.)h(There)14 b(is)f(also)g(a)g(big)h(adv)m(an)o(tage)e(in) i(using)g(the)f(same)g(con)o(text)f(construct)286 1661 y(for)i(collectiv)o(e) j(comm)o(unications)f(as)f(w)o(ell.)164 2009 y Fi(1.4.1)55 b(Con)n(text)19 b(Op)r(erations)164 2101 y Fl(A)h(global)i(con)o(text)e Fi(ALL)g Fl(is)h(prede\014ned.)35 b(All)20 b(pro)q(cesses)h(b)q(elong)h(to)f (this)g(con)o(text)164 2162 y(when)d(computation)g(starts.)28 b(MPI)18 b(do)q(es)h(not)g(sp)q(ecify)e(ho)o(w)i(pro)q(cesses)g(are)f (initially)164 2222 y(rank)o(ed)h(within)f(the)h(con)o(text)f(ALL.)h(It)g(is) g(exp)q(ected)f(that)h(the)g(start-up)h(pro)q(cedure)164 2282 y(used)d(to)g(initiate)f(an)i(MPI)e(program)h(\(at)g(load-time)f(or)h (run-time\))e(will)h(pro)o(vide)g(in-)164 2342 y(formation)d(or)h(con)o(trol) g(on)g(this)g(initial)e(ranking)j(\(e.g.,)e(b)o(y)g(sp)q(ecifying)g(that)h (pro)q(cesses)164 2402 y(are)20 b(rank)o(ed)g(according)g(to)h(their)e (pid's,)h(or)h(according)f(to)g(the)g(ph)o(ysical)f(addresses)961 2599 y(8)p eop %%Page: 9 9 bop 164 307 a Fl(of)15 b(the)g(executing)f(pro)q(cessors,)i(or)f(according)g (to)h(a)f(n)o(um)o(b)q(ering)e(sc)o(heme)g(sp)q(eci\014ed)i(at)164 367 y(load)i(time\).)164 548 y Fk(Discussion:)24 b Fj(If)17 b(w)o(e)g(think)h(of)f(adding)h(new)f(pro)q(cesses)h(at)e(run-time,)i(then)f Ff(ALL)g Fj(con)o(v)o(eys)164 608 y(the)e(wrong)g(impression,)h(since)g(it)g (is)f(just)g(the)g(initial)j(set)d(of)f(pro)q(cesses.)237 789 y Fl(The)i(follo)o(wing)g(op)q(erations)i(are)e(a)o(v)m(ailable)g(for)g (creating)g(new)g(con)o(texts.)164 909 y Fi(MPI)p 279 909 17 2 v 20 w(COPY)p 461 909 V 22 w(CONTEXT\(new)n(con)n(text,)i(con)n(text\))237 969 y Fl(Create)f(a)g(new)f(con)o(text)g(that)h(includes)e(all)h(pro)q (cesses)i(in)e(the)g(old)h(con)o(text.)k(The)164 1029 y(rank)c(of)f(the)g (pro)q(cesses)h(in)f(the)h(previous)f(con)o(text)f(is)h(preserv)o(ed.)21 b(The)16 b(call)g(m)o(ust)f(b)q(e)164 1090 y(executed)j(b)o(y)i(all)f(pro)q (cesses)i(in)e(the)h(old)g(con)o(text.)31 b(It)19 b(is)h(a)g(blo)q(c)o(king)f (call:)28 b(No)20 b(call)164 1150 y(returns)c(un)o(til)f(all)h(pro)q(cesses)h (ha)o(v)o(e)e(called)h(the)g(function.)21 b(The)16 b(parameters)f(are)164 1264 y Fi(OUT)k(new)n(con)n(text)24 b Fl(handle)12 b(to)h(newly)f(created)g (con)o(text.)19 b(The)13 b(handle)f(should)h(not)286 1324 y(b)q(e)j(asso)q (ciated)i(with)e(an)g(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1426 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 1649 y Fk(Discussion:)35 b Fj(I)11 b(considered)h(adding)f(a)f(string)g (parameter,)g(to)g(pro)o(vide)h(a)e(unique)j(iden)o(ti\014er)164 1706 y(to)j(the)h(next)f(con)o(text.)21 b(But,)15 b(in)i(an)e(en)o(vironmen)o (t)h(where)g(pro)q(cesses)g(are)f(single)i(threaded,)164 1762 y(this)j(is)h(not)f(m)o(uc)o(h)g(help:)31 b(Either)20 b(all)h(pro)q(cesses)g (agree)f(on)g(the)g(order)g(they)g(create)g(new)164 1819 y(con)o(texts,)15 b(or)h(the)g(application)i(deadlo)q(c)o(ks.)23 b(A)16 b(k)o(ey)g(ma)o(y)f (help)i(in)g(an)f(en)o(vironmen)o(t)h(where)164 1875 y(pro)q(cesses)f(are)g (m)o(ultithreaded,)h(to)e(distinguish)j(call)f(from)e(distinct)i(threads)f (of)f(the)h(same)164 1932 y(pro)q(cess;)f(but)g(it)h(migh)o(t)f(b)q(e)h (simpler)g(to)f(use)g(a)g(m)o(utex)g(algorithm)g(at)g(eac)o(h)g(pro)q(cess.) 237 1992 y Fk(Implemen)o(tation)23 b(note:)k Fj(No)19 b(comm)o(unication)h (is)f(needed)i(to)d(create)h(a)f(new)i(con-)164 2052 y(text,)g(b)q(ey)o(ond)g (a)f(barrier)h(sync)o(hronization;)i(all)e(pro)q(cesses)g(can)g(agree)f(to)g (use)h(the)f(same)164 2112 y(naming)c(sc)o(heme)f(for)g(successiv)o(e)i (copies)f(of)f(the)g(same)g(con)o(text.)19 b(Also,)c(no)f(new)g(rank)g(table) 164 2172 y(is)i(needed,)g(just)f(a)g(new)g(con)o(text)g(lab)q(el)h(and)g(a)f (new)g(p)q(oin)o(ter)h(to)e(the)i(same)f(old)g(table.)164 2413 y Fi(MPI)p 279 2413 V 20 w(NEW)p 438 2413 V 20 w(CONTEXT\(new)n(con)n(text,)j (con)n(text,)g(k)n(ey)-5 b(,)18 b(index\))961 2599 y Fl(9)p eop %%Page: 10 10 bop 164 307 a Fi(OUT)19 b(new)n(con)n(text)24 b Fl(handle)13 b(to)g(newly)g(created)f(con)o(text)g(at)i(calling)e(pro)q(cess.)21 b(This)286 367 y(handle)16 b(should)h(not)g(b)q(e)f(asso)q(ciated)h(with)f (an)h(ob)s(ject)f(b)q(efore)g(the)g(call.)164 469 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 571 y Fi(IN)i(k)n(ey)24 b Fl(in)o(teger)164 672 y Fi(IN)18 b(index)25 b Fl(in)o(teger)237 787 y(A)20 b(new)g(con)o(text)f(is)g(created)h(for)g(eac)o(h)f(distinct)g(v)m (alue)h(of)g Fg(key)p Fl(;)g(this)g(con)o(text)f(is)164 847 y(shared)e(b)o(y)g(all)f(pro)q(cesses)h(that)h(made)d(the)i(call)f(with)h (this)g(k)o(ey)e(v)m(alue.)23 b(Within)16 b(eac)o(h)164 907 y(new)k(con)o(text)g(the)g(pro)q(cesses)h(are)g(rank)o(ed)f(according)h(to)g (the)f(order)h(of)f(the)h Fg(index)164 967 y Fl(v)m(alues)16 b(they)g(pro)o(vided;)f(in)h(case)g(of)h(ties,)e(pro)q(cesses)i(are)f(rank)o (ed)g(according)g(to)h(their)164 1027 y(rank)f(in)g(the)g(old)h(con)o(text.) 237 1088 y(This)e(call)e(is)h(blo)q(c)o(king:)20 b(No)14 b(call)f(returns)i (un)o(til)e(all)h(pro)q(cesses)g(in)g(the)g(old)h(con)o(text)164 1148 y(executed)g(the)h(call.)237 1208 y(P)o(articular)g(uses)g(of)h(this)f (function)g(are:)237 1268 y(\(i\))h(Reordering)h(pro)q(cesses:)25 b(All)16 b(pro)q(cesses)i(pro)o(vide)f(the)h(same)e Fg(key)h Fl(v)m(alue,)g(and)164 1328 y(pro)o(vide)e(their)h(index)f(in)h(the)g(new)g (order.)237 1389 y(\(ii\))e(Splitting)g(a)h(con)o(text)f(in)o(to)h(sub)q(con) o(texts,)f(while)g(preserving)g(the)h(old)g(relativ)o(e)164 1449 y(order)23 b(among)f(pro)q(cesses:)35 b(All)21 b(pro)q(cesses)i(pro)o (vide)f(the)g(same)g Fg(index)f Fl(v)m(alue,)i(and)164 1509 y(pro)o(vide)15 b(a)i(k)o(ey)e(iden)o(tifying)g(their)g(new)h(sub)q(con)o (text.)164 1629 y Fi(MPI)p 279 1629 17 2 v 20 w(RANK\(rank,)i(con)n(text\)) 164 1743 y(OUT)h(rank)24 b Fl(in)o(teger)164 1845 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 1959 y(Return)h(the)g(rank)h (of)f(the)g(calling)g(pro)q(cess)g(within)g(the)g(sp)q(eci\014ed)g(con)o (text.)164 2080 y Fi(MPI)p 279 2080 V 20 w(SIZE\(size,)j(con)n(text\))164 2194 y(OUT)g(size)24 b Fl(in)o(teger)164 2296 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 2410 y(Return)h(the)g(n)o(um)o(b)q(er)f(of)h (pro)q(cesses)h(that)g(b)q(elong)f(to)h(the)f(sp)q(eci\014ed)g(con)o(text.) 949 2599 y(10)p eop %%Page: 11 11 bop 164 307 a Fi(Usage)25 b(note)49 b Fl(Use)21 b(of)i(con)o(texts)e(for)i (libraries:)32 b(Eac)o(h)22 b(library)f(ma)o(y)g(pro)o(vide)g(an)164 367 y(initialization)15 b(routine)i(that)g(is)g(to)g(b)q(e)g(called)f(b)o(y)h (all)f(pro)q(cesses,)h(and)h(that)f(generate)164 428 y(a)g(con)o(text)e(for)i (the)f(use)g(of)g(that)h(library)l(.)237 488 y(Use)11 b(of)h(con)o(texts)e (for)i(functional)f(decomp)q(osition:)18 b(A)11 b(harness)h(program,)g (running)164 548 y(in)18 b(the)g(con)o(text)g Fg(ALL)f Fl(generates)i(a)g (sub)q(con)o(text)f(for)h(eac)o(h)f(mo)q(dule)f(and)i(then)f(starts)164 608 y(the)e(submo)q(dule)f(within)h(the)g(corresp)q(onding)h(con)o(text.)237 668 y(Use)k(of)g(con)o(texts)g(for)g(collectiv)o(e)e(comm)o(unic)o(ation:)29 b(A)21 b(con)o(text)f(is)h(created)g(for)164 729 y(eac)o(h)16 b(group)h(of)f(pro)q(cesses)h(where)f(collectiv)o(e)e(comm)o(uni)o(cation)g (is)i(to)g(o)q(ccur.)237 789 y(Use)i(of)g(con)o(texts)g(for)g(con)o (text-switc)o(hing)f(among)h(sev)o(eral)f(parallel)g(executions:)164 849 y(A)h(pream)o(ble)e(co)q(de)i(is)g(used)g(to)g(generate)g(a)h(di\013eren) o(t)e(con)o(text)g(for)i(eac)o(h)e(execution;)164 909 y(this)j(pream)o(ble)d (co)q(de)j(needs)g(to)g(use)f(a)i(m)o(utual)d(exclusion)g(proto)q(col)j(to)f (mak)o(e)e(sure)164 969 y(eac)o(h)e(thread)g(claims)e(the)i(righ)o(t)g(con)o (text.)164 1139 y Fk(Discussion:)59 b Fj(If)20 b(pro)q(cess)g(handles)h(are)e (made)h(explicit)i(in)e(MPI,)g(then)g(an)f(additional)164 1195 y(function)c(needed)h(is)f Fk(MPI)p 646 1195 16 2 v 18 w(PR)o(OCESS\(pro)q (cess,)i(con)o(text,)f(rank\))p Fj(,)e(whic)o(h)i(returns)e(a)164 1252 y(handle)i(to)f(the)g(pro)q(cess)h(iden)o(ti\014ed)h(b)o(y)e(the)g Ff(rank)g Fj(and)g Ff(context)g Fj(parameters.)237 1312 y(A)10 b(p)q(ossible)i(addition)f(is)g(a)f(function)h(of)f(the)g(form)f Fk(MPI)p 1199 1312 V 18 w(CREA)l(TE)p 1434 1312 V 19 w(CONTEXT\(new)o(con)o (text,)164 1372 y(list)p 236 1372 V 20 w(of)p 298 1372 V 19 w(pro)q(cess)p 484 1372 V 18 w(handles\))19 b Fj(whic)o(h)g(creates)f(a)g (new)h(con)o(text)e(out)h(of)g(an)g(explicit)i(list)f(of)164 1432 y(mem)o(b)q(ers)e(\(and)g(rank)g(them)g(in)h(their)g(order)f(of)g(o)q (ccurrence)h(in)g(the)f(list\).)26 b(This,)18 b(coupled)164 1492 y(with)13 b(a)f(mec)o(hanism)h(for)e(requiring)j(the)e(spa)o(wning)h(of) f(new)g(pro)q(cesses)h(to)f(the)g(computation,)164 1553 y(will)20 b(allo)o(w)e(to)g(create)g(a)g(new)g(all)h(inclusiv)o(e)i(con)o(text)c(that)h (includes)i(the)e(additional)i(pro-)164 1613 y(cesses.)k(Ho)o(w)o(ev)o(er,)16 b(I)h(opp)q(ose)g(the)f(idea)h(of)g(requiring)g(dynamic)h(pro)q(cess)f (creation)f(as)g(part)164 1673 y(of)f(MPI.)f(Man)o(y)h(implemen)o(ters)h(w)o (an)o(t)e(to)g(run)i(MPI)f(in)h(an)f(en)o(vironmen)o(t)g(where)g(pro)q (cesses)164 1733 y(are)g(statically)h(allo)q(cated)g(at)f(load-time.)164 1983 y Fi(1.4.2)55 b(Error)18 b(Handling)164 2076 y Fl(It)12 b(is)g(assumed)g(that)g(MPI)g(is)g(implem)o(en)o(te)o(d)e(on)i(top)h(of)g(an) f(error-free)g(comm)o(unicati)o(on)164 2136 y(subsystem:)18 b(A)12 b(message)g(sen)o(t)g(is)g(alw)o(a)o(ys)g(receiv)o(ed)e(correctly)l(,) i(and)h(the)f(user)g(do)q(es)h(not)164 2196 y(need)e(to)g(c)o(hec)o(k)e(for)i (transmission)g(errors,)g(time-outs,)g(and)g(the)g(lik)o(es.)18 b(In)11 b(other)g(w)o(ords,)164 2256 y(MPI)21 b(do)q(es)h(not)g(pro)o(vide)e (mec)o(hanisms)f(to)i(deal)g(with)h(failures)e(in)i(the)f(underlying)164 2316 y(comm)o(unic)o(ation)15 b(subsystem)g({)j(it)e(is)h(the)f(resp)q (onsibilit)o(y)g(of)h(the)f(MPI)h(implem)o(en)n(ter)164 2377 y(to)i(insulate)g(the)g(user)f(from)g(suc)o(h)h(errors)g(\(or)g(to)h (re\015ect)e(them)f(as)j(global)f(program)164 2437 y(failures\).)i(The)16 b(same)f(holds)i(true)f(for)g(no)q(de)h(failures.)949 2599 y(11)p eop %%Page: 12 12 bop 237 307 a Fl(Of)15 b(course,)h(MPI)f(programs)g(ma)o(y)f(still)g(b)q(e)i (erroneous.)21 b(A)15 b Fi(program)j(error)d Fl(can)164 367 y(o)q(ccur)i(when)h(an)f(MPI)g(call)g(is)g(called)f(with)h(an)h(incorrect)e (parameter)g(\(non-existing)164 428 y(destination)f(in)g(a)h(send)g(op)q (eration,)g(bu\013er)f(to)q(o)i(small)c(in)i(a)h(receiv)o(e)d(op)q(eration,)j (etc.\))164 488 y(This)k(t)o(yp)q(e)g(of)h(error)f(w)o(ould)h(o)q(ccur)f(in)g (an)o(y)g(impleme)o(n)o(tation.)31 b(In)20 b(addition,)h(a)g Fi(re-)164 548 y(source)g(error)e Fl(ma)o(y)e(o)q(ccur)i(when)g(a)h(program)f (exceeds)e(the)i(amoun)o(t)f(of)i(a)o(v)m(ailable)164 608 y(system)e (resources)i(\(n)o(um)o(b)q(er)e(of)i(p)q(ending)h(messages,)f(system)e (bu\013ers,)j(etc.\).)31 b(The)164 668 y(o)q(ccurrence)13 b(of)i(this)f(t)o (yp)q(e)f(of)i(error)f(dep)q(ends)g(on)h(the)f(amoun)o(t)f(of)i(a)o(v)m (ailable)e(resources)164 729 y(in)19 b(the)g(system)f(and)j(the)e(resource)g (allo)q(cation)h(mec)o(hanism)c(used;)21 b(this)e(ma)o(y)f(di\013er)164 789 y(from)f(system)g(to)h(system.)26 b(The)18 b(recomme)o(nded)d(impleme)o (n)o(tation)h(pro\014le)h(pro)o(vides)164 849 y(sev)o(eral)d(mec)o(hanism)o (s)f(to)i(alleviate)e(the)i(p)q(ortabilit)o(y)f(problem)f(this)i(represen)o (ts.)20 b(One)164 909 y(can)c(also)h(write)f Fi(safe)g Fl(programs,)g(that)h (are)f(not)h(sub)s(ject)e(to)i(resource)f(errors.)237 969 y(All)21 b(MPI)g(pro)q(cedure)h(calls)f(return)h(an)h(error)e(parameter)g(that)h (indicates)g(suc-)164 1029 y(cessful)g(completion)f(of)i(the)f(op)q(eration,) j(or)e(the)f(error)h(condition)f(that)h(o)q(ccurred,)164 1090 y(otherwise.)237 1150 y(The)h(recommende)o(d)d(implem)o(en)o(tation)g (pro\014le)j(in)f(a)h(POSIX)f(en)o(vironmen)o(t)e(is)164 1210 y(for)h(an)o(y)f(MPI)g(routine)g(that)g(encoun)o(ters)g(a)h(reco)o(v)o (erable)e(error)h(to)h(store)f(an)h(error)164 1270 y(n)o(um)o(b)q(er)f(in)h (a)h(global)g(v)m(ariable)f(\()p Fh(errno)h Fl(in)f(a)h(C)g(en)o(vironmen)o (t\))d(and)j(generate)f(an)164 1330 y Fh(MPI)d(err)n(or)e(signal)p Fl(,)i(using)g(a)f(sp)q(ecial)g(signal)g(v)m(alue.)27 b(The)18 b(default)g(handler)g(for)g(this)164 1391 y(signal)13 b(terminates)e(the)h (execution)g(of)h(all)f(in)o(v)o(olv)o(ed)e(pro)q(cesses,)k(with)f(a)g (suitable)f(error)164 1451 y(message)19 b(b)q(eing)i(returned)f(to)g(the)g (user.)33 b(Ho)o(w)o(ev)o(er,)19 b(the)h(user)h(can)f(pro)o(vide)f(his)i(or) 164 1511 y(her)14 b(o)o(wn)g(signal)h(handling)f(routine.)20 b(In)14 b(particular,)g(the)g(user)g(can)g(sp)q(ecify)f(a)i(\\no)q(op")164 1571 y(signal)i(handler,)f(th)o(us)g(relegating)g(all)g(error)h(handling)f (to)h(the)g(user)f(co)q(de,)g(using)h(the)164 1631 y(error)f(parameters)f (returned)h(b)o(y)g(the)g(MPI)g(calls.)237 1692 y(MPI)f(calls)f(ma)o(y)g (initiate)g(op)q(erations)i(that)g(con)o(tin)o(ue)e(async)o(hronously)h (after)g(the)164 1752 y(call)k(returned.)30 b(Th)o(us,)20 b(the)f(op)q (eration)i(ma)o(y)d(return)h(with)g(a)h(co)q(de)g(indicating)f(suc-)164 1812 y(cessful)14 b(completion,)e(y)o(et)i(later)g(cause)h(an)g(error)f (exception)g(to)h(b)q(e)f(raised.)21 b(If)14 b(there)g(is)164 1872 y(a)i(subsequen)o(t)f(call)g(that)h(relates)f(to)g(the)h(same)e(op)q (eration)i(\(e.g.,)f(a)h(call)e(that)i(v)o(eri\014es)164 1932 y(that)f(an)f(async)o(hronous)i(op)q(eration)f(has)g(completed\))d(then)i (the)g(error)g(parameter)f(as-)164 1993 y(so)q(ciated)18 b(with)f(this)g (call)g(will)f(b)q(e)i(used)f(to)h(indicate)e(the)h(nature)h(of)g(the)f (error.)24 b(In)18 b(a)164 2053 y(few)13 b(cases,)h(the)g(error)f(ma)o(y)f(o) q(ccur)i(after)g(all)f(calls)g(that)h(relate)f(to)h(the)f(op)q(eration)i(ha)o (v)o(e)164 2113 y(completed,)e(so)k(that)g(no)f(error)g(parameter)f(can)h(b)q (e)h(used)f(to)g(indicate)f(the)h(nature)h(of)164 2173 y(the)h(error)h (\(e.g.,)f(an)h(error)f(in)h(a)g(send)f(with)h(the)f(ready)g(mo)q(de\).)28 b(In)18 b(suc)o(h)g(cases,)h(an)164 2233 y(error)d(will)f(b)q(e)i (undetected,)e(if)g(the)h(user)h(disabled)e(the)h(MPI)g(error)g(signal.)164 2414 y Fk(Discussion:)40 b Fj(The)16 b(alternativ)o(e)f(c)o(hoice)h(is)g(to)f (ha)o(v)o(e)f(fatal)h(and)h(non-fatal)f(signals.)949 2599 y Fl(12)p eop %%Page: 13 13 bop 164 452 a Fm(1.5)70 b(Messages)164 544 y Fl(A)16 b(message)f(consists)i (of)g(an)f Fh(envelop)n(e)i Fl(and)f Fh(data)p Fl(.)164 674 y Fi(1.5.1)55 b(Data)164 766 y Fl(The)17 b(data)h(part)g(of)f(a)h(message)e (consists)i(of)f(a)h(sequence)e(of)h(v)m(alues,)g(eac)o(h)g(of)g(a)h(basic) 164 826 y(datat)o(yp)q(e)k(in)f(the)g(host)h(language.)37 b(Th)o(us,)22 b(in)f(F)l(ortran,)i(a)f(message)e(consists)i(of)g(a)164 887 y(sequence)14 b(of)h(v)m(alues)h(that)f(are)g(eac)o(h)g(of)g(t)o(yp)q(e)g Fg(INTEGER)p Fl(,)d Fg(REAL)p Fl(,)h Fg(DOUBLE)24 b(PRECISION)o Fl(,)164 947 y Fg(COMPLEX)p Fl(,)16 b Fg(LOGICAL)p Fl(,)f(or)k(\(length)g (1\))f Fg(CHARACTER)p Fl(.)d(A)k(message)e(ma)o(y)g(mix)g(v)m(alues)i(of)164 1007 y(di\013eren)o(t)c(t)o(yp)q(es.)164 1188 y Fk(Discussion:)40 b Fj(Ma)o(y)15 b(also)g(need)h Ff(DOUBLE)23 b(COMPLEX)14 b Fj(in)i(F)l(ortran.)164 1438 y Fi(1.5.2)55 b(En)n(v)n(elop)r(e)164 1530 y Fl(The)16 b(follo)o(wing)g(information)f(is)h(asso)q(ciated)i(with)e (eac)o(h)f(message:)164 1632 y Fi(source)24 b Fl(The)16 b(rank)h(the)f (sending)g(pro)q(cess)164 1734 y Fi(destination)25 b Fl(The)17 b(rank)f(of)h(the)f(receiving)e(pro)q(cess)164 1835 y Fi(tag)24 b Fl(User)16 b(de\014ned)164 1937 y Fi(con)n(text)24 b Fl(handle)237 2039 y(The)15 b(range)g(of)g(v)m(alid)f(v)m(alues)h(for)g(the)f Fi(source)g Fl(and)h Fi(destination)h Fl(\014elds)e(is)h Fg(0)25 b(...)164 2099 y(n-1)p Fl(,)16 b(where)h Fg(n)f Fl(is)h(the)g(n)o(um)o(b)q (er)f(of)h(pro)q(cesses)h(in)f(the)g(curren)o(t)f(con)o(text.)23 b(The)17 b(ranges)164 2159 y(of)j(v)m(alid)g(v)m(alues)g(for)h Fg(tag)e Fl(is)g(impleme)o(n)o(tation)e(dep)q(enden)o(t,)k(and)f(can)g(b)q(e) h(found)f(b)o(y)164 2219 y(calling)13 b(a)h(suitable)g(query)f(function,)h (as)g(describ)q(ed)g(in)f(Section)h Fi(??)p Fl(.)21 b Fg(Context)11 b Fl(should)164 2279 y(b)q(e)16 b(a)h(con)o(text)e(shared)i(b)o(y)f(b)q(oth)h (source)f(and)h(destination.)237 2340 y(The)h Fg(tag)e Fl(\014eld)h(can)h(b)q (e)g(arbitrarily)e(set)i(b)o(y)f(the)g(application,)g(and)h(can)g(b)q(e)g (used)164 2400 y(to)f(distinguish)f(di\013eren)o(t)f(messages.)949 2599 y(13)p eop %%Page: 14 14 bop 237 307 a Fl(The)12 b(actual)g(mec)o(hanism)d(used)j(to)g(asso)q(ciate)h (an)f(en)o(v)o(elop)q(e)f(with)g(a)i(message)e(is)g(im-)164 367 y(plemen)o(tation)h(dep)q(enden)o(t;)i(some)f(of)i(the)f(information)f (\(e.g.,)h Fi(sender)f Fl(or)i Fi(receiv)n(er)p Fl(\))164 428 y(ma)o(y)g(b)q(e)h(implicit,)c(and)17 b(need)f(not)h(b)q(e)f(explicitly)e (carried)h(b)o(y)h(a)h(message.)164 572 y Fm(1.6)70 b(Data)23 b(Bu\013ers)164 664 y Fl(The)18 b(basic)h(p)q(oin)o(t)f(to)h(p)q(oin)o(t)f (comm)o(unicati)o(on)e(op)q(erations)k(are)e Fi(send)g Fl(and)h Fi(receiv)n(e)p Fl(.)164 724 y(A)13 b Fi(send)h Fl(op)q(eration)h(creates)f (a)g(message;)g(the)f(message)h(data)g(is)g(assem)o(bled)e(from)h(the)164 785 y Fi(send)k(bu\013er)p Fl(.)j(A)14 b Fi(receiv)n(e)f Fl(op)q(eration)j (consumes)d(a)i(message;)f(the)g(message)g(data)h(is)164 845 y(mo)o(v)o(ed)g(in)o(to)i(the)f Fi(receiv)n(e)j(bu\013er)p Fl(.)k(The)17 b(sp)q(eci\014cation)g(of)g(send)g(or)h(receiv)o(e)c(bu\013ers) 164 905 y(uses)i(the)g(same)g(syn)o(tax.)237 965 y(A)j(bu\013er)g(consists)h (of)g(a)f(sequence)f Fi(bu\013er)k(comp)r(onen)n(ts)p Fl(.)30 b(Eac)o(h)19 b(comp)q(onen)o(t)164 1025 y(consists)14 b(of)h(a)f(sequence)f (v)m(ariables)h(of)h(the)f(same)f(basic)h(t)o(yp)q(e.)20 b(There)13 b(are)h(three)g(basic)164 1086 y(t)o(yp)q(es)i(of)h(bu\013er)f(comp)q(onen)o (ts:)164 1199 y Fi(blo)r(c)n(k)25 b Fl(A)15 b(sequence)h(of)g(con)o(tiguous)h (v)m(alues)f(of)h(the)f(same)f(basic)h(t)o(yp)q(e,)f(sp)q(eci\014ed)h(b)o(y) 286 1301 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1382 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e(\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1462 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)164 1564 y Fi(v)n(ector)24 b Fl(A)14 b(sequence)f(of)i(equally)f(spaced)h(and)g(equally)e(sized)h(blo)q (c)o(ks)h(of)g(elemen)o(ts)d(of)286 1624 y(the)k(same)f(basic)h(t)o(yp)q(e,)g (sp)q(eci\014ed)f(b)o(y)286 1726 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1807 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e (\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1887 y Fi(stride)24 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(b)q(et)o(w)o(een)h(the)h(start)h (of)f(eac)o(h)g(blo)q(c)o(k)286 1968 y Fi(len)n(blk)25 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(in)h(eac)o(h)h(blo)q(c)o(k)g(\()p Fg(lenblk)c Fe(\024)h Fg(stride)p Fl(\))286 2049 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)286 2151 y(Note)10 b(that)i(a)f (constan)o(t)g(stride)f(b)q(ecomes)g(con)o(tiguous)h(when)g Fg(stride)24 b(=)h(lenblk)p Fl(.)286 2211 y(A)c(v)o(ector)g(bu\013er)g(comp)q (onen)o(t)g(can)h(b)q(e)f(an)h(arbitrary)g(submatrix)e(of)i(a)g(t)o(w)o(o-) 286 2271 y(dimensional)14 b(matrix.)164 2372 y Fi(indexed)24 b Fl(A)16 b(sequence)f(of)i(elemen)o(t)o(s)d(of)j(the)f(same)f(basic)h(t)o (yp)q(e,)f(sp)q(eci\014ed)h(b)o(y)286 2474 y Fi(start)24 b Fl(initial)15 b(elemen)o(t)949 2599 y(14)p eop %%Page: 15 15 bop 286 307 a Fi(list)p 363 307 17 2 v 21 w(of)p 429 307 V 20 w(indices)26 b Fl(List)12 b(of)g(displacemen)o(ts)d(of)j(the)f(elemen)o (ts)e(in)j(the)f(bu\013er)h(com-)393 367 y(p)q(onen)o(ts,)17 b(relativ)o(e)d(to)j(the)f(initial)f(elemen)n(t.)286 443 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)237 534 y(F)l(or)f(example,)d(if)i(a)h(F)l(ortran)g(arra)o(y)g(is)f(declared)g (as)h Fg(double)24 b(precision)e(a\(10\))p Fl(,)164 594 y(then)15 b(the)f(tuple)g Fd(<)p Fg(\(a\(3\),)24 b(\(0,3,6,2\))o(,)f(DOUBLE)p Fd(>)12 b Fl(sp)q(eci\014es)j(a)g(bu\013er)g(comp)q(onen)o(t)164 655 y(with)h(en)o(tries)f Fg(a\(3\))24 b(a\(6\),)g(a\(9\),)g(a\(5\))p Fl(.)164 824 y Fk(Discussion:)56 b Fj(Do)19 b(w)o(e)g(allo)o(w)g(en)o(tries)h (to)f(b)q(e)h(rep)q(eated)f(in)h(an)g(indexed)g(bu\013er)g(comp)q(o-)164 880 y(nen)o(t?)237 937 y(Do)15 b(w)o(e)g(allo)o(w)g(di\013eren)o(t)g (bu\013er)h(comp)q(onen)o(ts)f(to)f(o)o(v)o(erlap?)237 993 y(Do)e(w)o(e)g(require)h(in)g(an)f(v)o(ector)g(bu\013er)g(comp)q(onen)o(t)g (that)g Ff(len)g Fj(b)q(e)g(a)g(m)o(ultiple)i(of)e Ff(lenblk)p Fj(?)237 1234 y Fl(A)j(bu\013er)h(is)f(describ)q(ed)f(b)o(y)h(an)h(opaque)f (ob)s(ject)g(accessed)g(via)g(a)h Fi(bu\013er)h(handle)p Fl(.)164 1294 y(Suc)o(h)f(ob)s(ject)g(is)g(created)f(and)i(destro)o(y)o(ed)f(via)g (calls)f(to)i Fg(MPI)p 1292 1294 16 2 v 17 w(CREATE)e Fl(and)h Fg(MPI)p 1652 1294 V 18 w(FREE)p Fl(.)164 1354 y(It)k(is)h(asso)q(ciated)h (with)f(successiv)o(e)e(bu\013er)i(comp)q(onen)o(ts)f(b)o(y)g(calling)h(in)f (succession)164 1415 y(one)f(of)g(the)g(functions)g Fg(MPI)p 695 1415 V 17 w(ADD)p 790 1415 V 18 w(BLOCK)p Fl(,)d Fg(MPI)p 1046 1415 V 18 w(ADD)p 1142 1415 V 17 w(VECTOR)h Fl(or)i Fg(MPI)p 1472 1415 V 17 w(ADD)p 1567 1415 V 18 w(INDEX)p Fl(,)d(in)164 1475 y(order)h(to)g(app)q(end)h(a)g(comp)q(onen)o(t)e(to)h(the)g(bu\013er)g (asso)q(ciated)h(with)f(a)g(bu\013er)g(handle.)164 1535 y(A)f(bu\013er)h(ob)s (ject)f(can)g(b)q(e)h(destro)o(y)o(ed)e(only)h(if)g(there)g(is)g(no)h(p)q (ending)g(comm)o(unicati)o(on)164 1595 y(op)q(eration)j(using)g(it.)31 b(After)19 b(a)h(bu\013er)g(ob)s(ject)f(is)g(destro)o(y)o(ed)g(the)g(asso)q (ciated)i(bu\013er)164 1655 y(handle)16 b(is)g(unde\014ned.)164 1776 y Fi(MPI)p 279 1776 17 2 v 20 w(ADD)p 427 1776 V 21 w(BLOCK\()i (bu\013er,)g(start,)g(len,)h(datat)n(yp)r(e\))237 1896 y Fl(App)q(end)d(a)h (blo)q(c)o(k)f(comp)q(onen)o(t)f(to)i(bu\013er.)k(The)16 b(parameters)g(are:) 164 1981 y Fi(INOUT)i(bu\013er)24 b Fl(bu\013er)17 b(handle)164 2077 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g (elemen)o(t)e(\(c)o(hoice\))164 2173 y Fi(IN)18 b(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o(teger\))164 2269 y Fi(IN)18 b(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d(\(status\))164 2414 y Fi(MPI)p 279 2414 V 20 w(ADD)p 427 2414 V 21 w(VEC\()19 b(bu\013er,)f(len,)g(stride,)g(len)n (blk,)i(datat)n(yp)r(e)e(\))949 2599 y Fl(15)p eop %%Page: 16 16 bop 237 307 a Fl(App)q(end)16 b(a)h(v)o(ector)e(bu\013er)i(comp)q(onen)o(t)e (to)i(bu\013er.)k(The)c(parameters)e(are:)164 409 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 511 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g(elemen)o(t)e(\(c)o(hoice\))164 612 y Fi(IN)18 b(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o (teger\))164 714 y Fi(IN)18 b(stride)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(b)q(et)o(w)o(een)i(the)h(start)h(of)g(eac)o(h)e(blo)q(c) o(k)h(\(in)o(teger\))164 816 y Fi(IN)i(len)n(blk)26 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(in)j(eac)o(h)g(blo)q(c)o(k)f(\(in)o(teger\))164 917 y Fi(IN)j(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d (\(status\))164 1079 y Fi(MPI)p 279 1079 17 2 v 20 w(ADD)p 427 1079 V 21 w(INDEX\()k(bu\013er,)g(start,)g(list)p 1077 1079 V 21 w(of)p 1143 1079 V 20 w(indices,)i(datat)n(yp)r(e\))237 1200 y Fl(App)q(end)c(an)h(indexed)e(bu\013er)i(comp)q(onen)o(t)e(to)i (bu\013er.)k(The)c(parameters)e(are:)164 1301 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 1403 y Fi(start)24 b Fl(initial)15 b(p)q(osition)i(for)f(indexing)g(\(c)o(hoice\))164 1505 y Fi(list)p 241 1505 V 21 w(of)p 307 1505 V 20 w(indices)26 b Fl(list)15 b(of)i(relativ)o(e)d(indices)i(of)g(en)o(tries)f(\(arra)o(y)h(of)h(in)o (tegers\))164 1606 y Fi(IN)h(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d(\(status\))237 1708 y(Consider,)i(for)h(example,)c(the)j (follo)o(wing)g(fragmen)o(t)f(of)i(F)l(ortran)g(co)q(de)164 1822 y Fg(DOUBLE)23 b(PRECISION)g(A\(10,20\))164 1883 y(INTEGER)g(B,)i (C\(5,10\))164 1943 y(INTEGER)e(BH)164 2003 y(...)164 2063 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 2123 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(B,)h(1,)g(MPI_INT\))164 2183 y(CALL)f(MPI_ADD_VE)o(C)f(\(BH,)h(A\(1,3\),)f(11,)h(4,)h(2,)g (MPI_DOUBLE\))164 2244 y(CALL)f(MPI_ADD_IN)o(DEX)o(\(BH)o(,)f(C\(3,7\),)g (\(4,2,1\),)f(MPI_INT\))949 2599 y Fl(16)p eop %%Page: 17 17 bop 237 307 a Fl(Then)15 b(the)g(bu\013er)g(asso)q(ciated)g(with)g(the)f (handle)h Fg(BH)f Fl(consists)h(of)g(the)g(sequence)f(of)164 367 y(v)m(ariables)237 428 y Fg(B,)25 b(A\(1,3\),)e(A\(2,3\),)g(A\(5,3\),)g (A\(6,3\),)g(A\(9,3\),)g(A\(10,3\),)g(A\(3,4\),)g(A\(4,4\),)164 488 y(A\(7,4\),)g(A\(8,4\),)g(A\(1,5\),)g(C\(2,8\),)g(C\(5,7\),)g(C\(4,7\))p Fl(.)237 548 y(A)11 b(message)f(created)g(from)g(this)h(bu\013er)g(will)f (consist)h(of)g(a)g(sequence)f(of)h(one)g(in)o(teger,)164 608 y(follo)o(w)o(ed)k(b)o(y)h(elev)o(en)e(double)j(precision)e(reals,)h(follo)o (w)o(ed)f(b)o(y)h(three)f(in)o(tegers.)237 668 y(A)d(bu\013er)g(handle)g(can) g(b)q(e)g(used)g(for)g(comm)o(unic)o(ation,)e(ev)o(en)h(if)g(it)h(is)g(not)g (asso)q(ciated)164 729 y(with)i(an)o(y)f(v)m(ariables)h(\(i.e.,)e(ev)o(en)h (if)g(it)h(w)o(as)g(not)g(set)g(b)o(y)g(an)o(y)f Fg(MPI)p 1354 729 16 2 v 18 w(ADD)p 1450 729 V 17 w(xxx)g Fl(call\).)19 b(Suc)o(h)164 789 y(handle)i(is)g(asso)q(ciated)h(with)e(an)i(empt)o(y)d(bu\013er,)j(and)f (a)h(message)e(created)g(from)g(it)164 849 y(con)o(tains)c(no)h(data.)164 1018 y Fk(Discussion:)237 1075 y Fj(The)h(main)g(mo)q(di\014cations)h(w.r.t.) 25 b(the)18 b(prop)q(osal)f(of)g(Gropp)h(and)f(Lusk)h(is)g(measuring)164 1131 y(length)g(in)g(elemen)o(ts,)f(rather)g(than)g(b)o(ytes.)25 b(Seems)17 b(more)g(natural,)g(since)h(t)o(yp)q(e)f(is)g(kno)o(wn.)164 1188 y(Also,)k(ob)s(ject)e(creation)h(uses)f(a)h(generic)g(function,)i (rather)d(than)g(a)g(function)i(sp)q(eci\014c)g(to)164 1244 y(bu\013er)c(descriptors.)28 b(As)17 b(result,)h(I)g(ga)o(v)o(e)f(up)h(on)f (the)h(size)g(parameter.)26 b(This)18 b(ma)o(y)f(not)g(b)q(e)164 1301 y(suc)o(h)k(a)g(loss:)32 b(it)21 b(is)g(not)g(clear)h(that)e(sp)q (ecifying)j(a)d(maxim)o(um)h(length)h(at)e(bu\013er)h(ob)s(ject)164 1357 y(creation)h(is)g(useful,)h(since)g(indices)g(of)e(arbitrary)g(size)i (ma)o(y)d(need)j(to)d(b)q(e)j(stored)d(in)j(the)164 1413 y(ob)s(ject.)164 1724 y Fi(1.6.1)55 b(Data)19 b(Con)n(v)n(ersion)164 1816 y Fl(The)d(t)o(yp)q(es)g(and)h(the)f(lo)q(cations)h(of)g(the)f(en)o(tries)f (used)h(to)h(create)e(a)i(message)f(is)g(solely)164 1876 y(determined)f(from) i(the)g(information)g(in)g(the)h(bu\013er)g(descriptor,)f(using)h(the)g (storage)164 1937 y(asso)q(ciation)g(rules)e(sp)q(eci\014ed)g(b)o(y)g(the)g (host)i(language)f(and)h(its)e(implem)o(en)n(tation;)e(the)164 1997 y(t)o(yp)q(e)22 b(and)h(the)g(lo)q(cations)g(of)g(these)g(en)o(tries)e (do)i(not)h(dep)q(end)e(on)i(the)e(declaration)164 2057 y(for)g(the)g (corresp)q(onding)g(v)m(ariables)g(in)g(the)f(calling)g(program.)38 b(It)21 b(is)h(not)g(required)164 2117 y(that)h(the)f(data)h(t)o(yp)q(es)f (sp)q(eci\014ed)f(in)h(a)h(bu\013er)g(descriptor)e(matc)o(h)g(the)h(data)h(t) o(yp)q(es)164 2177 y(of)h(the)f(corresp)q(onding)i(elemen)o(ts)20 b(in)j(the)h(host)g(program.)43 b(Ho)o(w)o(ev)o(er,)23 b(in)g(case)h(of)164 2238 y(mismatc)o(hes,)c(the)i(corresp)q(ondence)g(b)q(et)o(w)o(een)f(en)o (tries)g(in)h(the)g(host)g(program)g(and)164 2298 y(en)o(tries)10 b(in)g(a)h(message)f(created)h(with)f(the)h(bu\013er)g(descriptor)g(ma)o(y)e (b)q(e)i(implem)o(en)o(tati)o(on)164 2358 y(dep)q(enden)o(t.)34 b(No)20 b(data)h(con)o(v)o(ersion)f(o)q(ccurs)h(when)g(data)g(is)f(mo)o(v)o (ed)f(from)g(a)i(sender)164 2418 y(bu\013er)c(in)o(to)e(a)i(message.)949 2599 y(17)p eop %%Page: 18 18 bop 237 307 a Fl(Consider)17 b(the)f(follo)o(wing)g(fragmen)o(t)f(of)h(F)l (ortran)h(co)q(de)164 421 y Fg(REAL)24 b(A\(100\))164 482 y(INTEGER)f(BH)164 542 y(...)164 602 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f (MPI_PERSIST)o(EN)o(T\))164 662 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(1\),)g (1,)h(MPI_REAL\))164 722 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(2\),)g(1,)h (MPI_INT\))164 782 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(3\),)g(1,)h (MPI_LOGICA)o(L\))164 843 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(4\),)g(1,)h (MPI_COMPLE)o(X\))164 903 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(6\),)g(1,)h (MPI_DOUBLE)o(\))164 963 y(CALL)f(MPI_ADD_BL)o(OCK)o(\(BH)o(,)f(A\(8\),)g(4,) i(MPI_CHAR\))237 1077 y Fl(A)17 b(message)g(created)g(from)f(this)h(bu\013er) h(will)f(consist)g(of)h(a)g(sequence)e(con)o(taining)164 1137 y(one)k(real,)f(one)h(in)o(teger,)f(one)h(logical,)f(one)h(complex,)e(one)i (double,)g(and)g(four)g(c)o(har-)164 1198 y(acters.)31 b(No)19 b(data)i(con)o(v)o(ersion)d(o)q(ccurs)i(when)g(v)m(alues)f(are)h(copied)f (from)f(the)h(sender)164 1258 y(bu\013er)e(to)g(the)f(message.)22 b(Th)o(us,)16 b(the)h(\014rst)g(en)o(try)e(in)i(the)f(message)g(is)g(a)h (real)g(n)o(um)o(b)q(er)164 1318 y(equal)g(to)h Fg(A\(1\))p Fl(;)f(the)h(second)g(en)o(try)f(is)g(an)i(in)o(teger)d(n)o(um)o(b)q(er)g (that)j(happ)q(ens)g(to)f(ha)o(v)o(e)164 1378 y(the)13 b(same)f(binary)g (represen)o(tation)h(as)g Fg(A\(2\))p Fl(;)g(the)f(third)h(en)o(try)f(is)h(a) g(logical)g(v)m(alue)f(that)164 1438 y(happ)q(ens)k(to)f(ha)o(v)o(e)g(the)f (same)g(binary)h(represen)o(tation)g(as)g Fg(A\(3\))p Fl(;)f(the)h(fourth)g (en)o(try)f(in)164 1499 y(a)g(complex)e(n)o(um)o(b)q(er)g(with)h(a)i(binary)e (represen)o(tation)g(iden)o(tical)f(to)j Fg(A\(4\),)23 b(A\(5\))p Fl(;)13 b(the)164 1559 y(\014fth)k(en)o(try)f(is)h(a)g(double)g(precision)f (v)m(alue)h(with)f(a)i(binary)e(represen)o(tation)h(iden)o(tical)164 1619 y(to)j Fg(A\(6\),)j(A\(7\))p Fl(;)c(and)h(if)f(w)o(ords)h(ha)o(v)o(e)f (four)g(b)o(ytes)g(then)g(the)g(last)h(four)g(en)o(tries)e(are)164 1679 y(b)o(ytes)f(that)g(mak)o(e)f(the)h(binary)g(represen)o(tation)g(of)h Fg(A\(8\))p Fl(,)d(in)i(the)h(b)o(yte)e(order)h(of)h(the)164 1739 y(executing)d(mac)o(hine.)237 1800 y(The)f(corresp)q(ondence)g(b)q(et)o (w)o(een)g(the)g(\014rst)g(sev)o(en)f(en)o(tries)g(of)i(the)e(arra)o(y)i Fg(A)e Fl(and)i(the)164 1860 y(\014rst)g(\014v)o(e)g(en)o(tries)f(of)h(the)g (message)g(created)f(from)g(this)h(bu\013er)h(is)f(determined)d(b)o(y)j(the) 164 1920 y(rules)k(of)h(F)l(ortran)g(77)g(on)g(storage)h(asso)q(ciation:)29 b(Eac)o(h)19 b(v)m(ariable)g(of)h(t)o(yp)q(e)f Fg(INTEGER)p Fl(,)164 1980 y Fg(REAL)p Fl(,)11 b(or)j Fg(LOGICAL)c Fl(o)q(ccup)o(y)j(one)g Fh(numeric)i(stor)n(age)g(unit)p Fl(;)f(a)f(v)m(ariable)g(of)h(t)o(yp)q(e)e Fg(DOUBLE)164 2040 y Fl(or)21 b Fg(COMPLEX)c Fl(o)q(ccup)o(y)j(t)o(w)o(o)g(n) o(umeric)e(storage)j(units.)33 b(Th)o(us,)21 b(the)f(same)f(corresp)q(on-)164 2100 y(dence)c(will)f(hold)i(for)g(an)o(y)g(implem)o(e)o(n)o(tation.)i(Ho)o (w)o(ev)o(er,)c(di\013eren)o(t)h(implem)o(en)n(tations)164 2161 y(ma)o(y)20 b(ha)o(v)o(e)g(di\013eren)o(t)g(binary)i(enco)q(dings)f(of)h (in)o(teger,)f(real)g(and)h(logical)e(v)m(alues,)i(so)164 2221 y(that)17 b(the)f(actual)g(v)m(alues)g(transferred)g(b)o(y)g(the)g(message)g (ma)o(y)e(di\013er.)237 2281 y(The)21 b(corresp)q(ondence)h(b)q(et)o(w)o(een) e(the)h(en)o(try)f Fg(A\(8\))g Fl(of)h(the)g(arra)o(y)l(,)h(and)g(the)f(last) 164 2341 y(four)j(c)o(haracter)g(en)o(tries)e(in)i(the)f(message)g(is)h (implem)o(en)o(tation)d(dep)q(enden)o(t,)k(since)164 2401 y(the)19 b(F)l(ortran)i(language)f(do)q(es)h(not)f(sp)q(ecify)f(a)h(corresp)q(ondence) g(b)q(et)o(w)o(een)e(c)o(haracter)164 2462 y(storage)j(units)g(and)g(n)o (umeric)d(storage)j(units)f(\(an)h(arra)o(y)g(of)g(c)o(haracters)f(cannot)h (b)q(e)949 2599 y(18)p eop %%Page: 19 19 bop 164 307 a Fl(equiv)m(alenced)16 b(with)i(an)g(arra)o(y)g(of)g(in)o (tegers\).)26 b(Di\013eren)o(t)17 b(results)h(ma)o(y)e(o)q(ccur)i(in)g(big-) 164 367 y(endians)e(or)h(small-endians)e(mac)o(hines,)f(or)i(in)g(32)h(bit)f (or)h(64)g(bit)f(mac)o(hines.)237 428 y(The)i(same)e(holds,)i(symmetri)o (call)o(y)l(,)c(when)k(a)g(message)e(is)i(receiv)o(ed.)k(En)o(tries)17 b(are)164 488 y(mo)o(v)o(ed)i(from)g(the)i(message)f(in)o(to)h(the)f(receiv)o (er)f(memory)f(according)j(to)g(the)g(infor-)164 548 y(mation)h(pro)o(vided)g (b)o(y)g(the)g(bu\013er)i(descriptor,)f(with)g(no)g(regard)g(to)g(the)g(w)o (a)o(y)f(the)164 608 y(corresp)q(onding)17 b(v)m(ariables)f(are)h(declared)e (in)h(the)g(receiving)f(program.)237 668 y(When)c(data)h(is)f(mo)o(v)o(ed)e (in)h(a)i(homogeneous)f(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)i(no)q(des)i(ha) o(ving)164 729 y(the)21 b(same)f(arc)o(hitecture,)g(then)h(no)g(data)h(con)o (v)o(ersion)e(o)q(ccur)h(at)h(an)o(y)f(p)q(oin)o(t)g(during)164 789 y(data)16 b(transfer.)21 b(Assume,)13 b(in)i(the)g(previous)g(example,)d (that)k(an)f(iden)o(tically)e(declared)164 849 y(bu\013er)j(descriptor)g(is)f (used)h(to)g(receiv)o(e)e(data)i(in)o(to)g(an)g(iden)o(tically)e(declared)h (arra)o(y)h(at)164 909 y(the)g(receiving)e(pro)q(cess.)21 b(Then,)16 b(when)g(these)f(t)o(w)o(o)h(no)q(des)g(comm)o(unicate,)c(the)k(v)m(alues)164 969 y(in)g(the)h(\014rst)g(eigh)o(t)f(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(of)h(the)g(sender)f(will)g(b)q(e)h(copied)f(in)o(to)g(the)h (\014rst)164 1029 y(eigh)o(t)e(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(at)h(the)f(receiv)o(er)e(\(assuming)i(that)h(reals)f(o)q(ccup)o(y)h (the)f(same)164 1090 y(storage)21 b(as)g(four)f(b)o(ytes\).)33 b(In)20 b(particular,)g(in)g(a)g(homogeneous)h(en)o(vironmen)o(t,)c(it)j(is) 164 1150 y(p)q(ossible)c(to)h(comm)o(unic)o(ate)d(using)i(only)g(c)o (haracter)g(t)o(yp)q(ed)g(messages.)237 1210 y(When)22 b(data)g(is)f(mo)o(v)o (ed)e(in)o(to)i(a)h(heterogeneous)g(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)h (no)q(des)164 1270 y(ha)o(ving)14 b(distinct)f(arc)o(hitectures,)g(data)i (con)o(v)o(ersion)e(ma)o(y)g(o)q(ccur)h(during)g(the)g(transfer:)164 1330 y(Eac)o(h)21 b(en)o(try)e(is)i(con)o(v)o(erted)e(from)h(the)g(data)h (represen)o(tation)f(used)h(on)g(the)g(sending)164 1391 y(no)q(de)c(to)f(the) g(data)i(represen)o(tation)d(used)i(in)e(the)h(receiving)f(no)q(de.)237 1451 y(Consider)i(the)f(follo)o(wing)g(fragmen)o(t)f(of)h(F)l(ortran)h(co)q (de.)164 1565 y Fg(REAL)24 b(X,)h(Y)164 1625 y(CHARACTER)d(\(4\))j(Z)164 1685 y(INTEGER)e(BH)164 1746 y(...)164 1806 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f (MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 1866 y(CALL)i(MPI_ADD_BL)o(OCK)e (\(BH,)i(X,)h(1,)g(MPI_REAL\))164 1926 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i (Y,)h(4,)g(MPI_CHAR\))164 1986 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(Z,)h(4,)g (MPI_CHAR\))237 2100 y Fl(Assume)11 b(that)i(the)g(same)e(arra)o(ys)i Fg(X,)25 b(Y,)g(Z)12 b Fl(and)h(bu\013er)g(handle)g Fg(BH)f Fl(are)g(created)h(at)164 2161 y(t)o(w)o(o)g(pro)q(cesses)h(A)e(and)i(B;)e (this)h(handle)g(is)g(used)g(b)o(y)g(A)f(to)i(create)e(and)i(send)f(a)h (message)164 2221 y(for)e(B,)f(b)o(y)h(B)f(to)i(receiv)o(e)c(this)j(message.) 19 b(F)l(urther)12 b(assume)f(that)i(b)q(oth)g(A)e(and)i(B)e(run)h(on)164 2281 y(distinct)g(no)q(des,)j(with)e(p)q(ossibly)g(di\013eren)o(t)g(32)h(bit) f(arc)o(hitectures.)18 b(Then)c Fg(X)f Fl(at)g(pro)q(cess)164 2341 y(B)19 b(is)g(assigned)h(the)g(v)m(alue)f(of)g Fg(X)h Fl(at)f(pro)q(cess)h(A)f(\(up)h(to)g(rounding)g(errors)g(that)f(ma)o(y)164 2401 y(o)q(ccur)12 b(during)g(con)o(v)o(ersion\);)f Fg(Z)h Fl(on)g(pro)q(cess)g(B)f(is)h(assigned)g(the)f(c)o(haracter)g(string)h(v)m (alue)164 2462 y(of)18 b Fg(Z)f Fl(on)h(pro)q(cess)g(A;)f(if)f(b)q(oth)j(no)q (des)f(use)g(ASCI)q(I)f(enco)q(ding,)g(then)h(no)g(con)o(v)o(ersion)e(is)949 2599 y(19)p eop %%Page: 20 20 bop 164 307 a Fl(required)14 b(for)h(the)g(c)o(haracters.)21 b(On)15 b(the)g(other)h(hand,)f(v)m(ariable)g Fg(Y)g Fl(on)h(pro)q(cess)g(B)e (ma)o(y)164 367 y(b)q(e)19 b(allo)q(cated)g(a)g(v)m(alue)f(that)h(di\013ers)g (from)f(the)g(v)m(alue)h(of)g(v)m(ariable)f Fg(Y)g Fl(on)h(pro)q(cess)h(A.) 164 428 y(This)12 b(ma)o(y)f(o)q(ccur)h(if)g(the)g(t)o(w)o(o)g(no)q(des)h (use)f(a)h(di\013eren)o(t)e(b)o(yte)g(sequence)g(\(little-endian)g(vs)164 488 y(big-endian\),)i(or)g(use)f(a)h(di\013eren)o(t)e(binary)i(represen)o (tation)e(for)i(reals)f(\(IEEE)h(vs)f(HEX\).)164 548 y(Th)o(us,)18 b(in)f(order)h(to)g(ensure)g(correct)f(execution)g(in)g(a)h(heterogeneous)g (en)o(vironmen)o(t,)164 608 y(it)e(is)g(imp)q(ortan)o(t)g(that)g(the)h(t)o (yp)q(es)f(of)g(v)m(alues)h(in)f(a)h(message)e(matc)o(h)g(the)h(t)o(yp)q(es)g (of)h(the)164 668 y(corresp)q(onding)g(v)m(alues)f(in)g(the)g(sending)h(and)g (in)f(the)g(receiving)e(program.)164 813 y Fm(1.7)70 b(Receiv)n(e)20 b(Criteria)164 905 y Fl(The)15 b(selection)f(of)h(a)g(message)f(b)o(y)h(a)g (receiv)o(e)e(op)q(eration)i(is)g(done)g(uniquely)f(according)164 965 y(to)23 b(the)g(v)m(alue)g(of)g(the)g(message)f(en)o(v)o(elop)q(e.)40 b(The)22 b(receiv)o(e)f(op)q(eration)j(sp)q(eci\014es)e(an)164 1025 y Fi(en)n(v)n(elop)r(e)k(pattern)p Fl(;)g(a)d(message)f(can)h(b)q(e)h (receiv)o(ed)c(b)o(y)j(that)g(receiv)o(e)e(op)q(eration)164 1086 y(only)c(if)f(its)h(en)o(v)o(elop)q(e)f(matc)o(hes)f(that)j(pattern.)24 b(A)17 b(pattern)g(sp)q(eci\014es)g(v)m(alues)g(for)g(the)164 1146 y Fg(source)p Fl(,)h Fg(tag)h Fl(and)h Fg(context)d Fl(\014elds)j(of)g (the)g(message)f(en)o(v)o(elop)q(e.)30 b(In)20 b(addition,)g(the)164 1206 y(v)m(alue)15 b(for)h(the)f Fg(dest)e Fl(\014eld)i(is)g(set,)g (implicitl)o(y)l(,)d(to)k(b)q(e)f(equal)g(to)h(the)f(receiving)e(pro)q(cess) 164 1266 y(id.)38 b(The)22 b(receiv)o(er)e(ma)o(y)g(sp)q(ecify)i(a)g Fi(DONTCARE)g Fl(v)m(alue)g(for)g Fg(source)p Fl(,)f(or)h Fg(tag)p Fl(,)164 1326 y(indicating)e(that)i(an)o(y)e(source)h(and/or)h(tag)g(are)f (acceptable.)34 b(It)21 b(cannot)g(sp)q(ecify)g(a)164 1387 y(DONTCARE)d(v)m(alue)h(for)f Fg(context)e Fl(or)j Fg(dest)p Fl(.)26 b(Th)o(us,)19 b(a)g(message)f(can)h(b)q(e)f(receiv)o(ed)164 1447 y(b)o(y)k(a)g(receiv)o(e)e(op)q(eration)j(only)e(if)h(it)f(is)h (addressed)h(to)f(the)g(receiving)e(task,)k(has)f(a)164 1507 y(matc)o(hing)18 b(con)o(text,)h(has)h(matc)o(hing)e(source)h(unless)h (source=DONTCARE)f(in)g(the)164 1567 y(pattern,)d(and)h(has)g(a)f(matc)o (hing)f(tag)i(unless)f(tag=DONTCARE)h(in)f(the)g(pattern.)237 1627 y(The)k(length)f(of)h(the)g(receiv)o(ed)d(message)i(m)o(ust)f(b)q(e)i (less)g(or)g(equal)f(the)g(length)h(of)164 1688 y(the)d(receiv)o(e)f (bu\013er.)26 b(I.e.,)16 b(all)h(incoming)f(data)i(m)o(ust)f(\014t,)g (without)h(truncation,)g(in)o(to)164 1748 y(the)f(receiv)o(e)f(bu\013er.)25 b(It)17 b(is)h(erroneous)g(to)g(receiv)o(e)d(a)j(message)f(whic)o(h)g(length) g(exceed)164 1808 y(the)e(receiv)o(e)d(bu\013er,)j(and)g(the)g(outcome)e(of)j (program)e(where)h(this)f(o)q(ccurs)h(is)g(undeter-)164 1868 y(mined.)164 2013 y Fm(1.8)70 b(Comm)n(unicati)o(on)21 b(Mo)r(de)164 2105 y Fl(A)16 b(sending)g(op)q(eration)h(can)g(o)q(ccur)f(in)g(one)g(of)h(t) o(w)o(o)f(mo)q(des:)164 2207 y Fi(REGULAR)24 b Fl(The)e(send)g(ma)o(y)f (start)i(whether)f(or)g(not)h(a)g(matc)o(hing)d(receiv)o(e)g(has)286 2267 y(b)q(een)c(p)q(osted.)164 2369 y Fi(READ)n(Y)25 b Fl(The)16 b(send)h(ma)o(y)d(start)j(only)f(if)g(a)g(matc)o(hing)f(receiv)o(e)f(has)j(b) q(een)f(p)q(osted.)949 2599 y(20)p eop %%Page: 21 21 bop 237 307 a Fl(A)21 b Fi(ready)k(send)c Fl(can)h(start)g(only)g(if)f(a)h (matc)o(hing)e(receiv)o(e)g(is)h(already)h(p)q(osted;)164 367 y(otherwise)17 b(the)g(op)q(eration)i(is)e(erroneous)h(and)g(its)f(outcome)g (is)g(unde\014ned.)25 b(In)17 b(some)164 428 y(systems,)12 b(this)g(will)g(allo)o(w)h(to)g(optimize)d(comm)o(unic)o(ation)h(and)i(a)o(v) o(oid)f(a)h(hand-shaking)164 488 y(op)q(eration)k(that)g(is)f(otherwise)g (required.)164 668 y Fk(Discussion:)28 b Fj(I)19 b(deleted)i(the)e(symmetric) g(ready)g(receiv)o(e.)32 b(Will)21 b(reviv)o(e)f(it)f(if)g(there)h(is)f(a)164 729 y(requiremen)o(t)d(for)e(it.)164 993 y Fm(1.9)70 b(Comm)n(unicati)o(on)21 b(Ob)t(jects)164 1086 y Fl(An)13 b(opaque)i(comm)o(uni)o(cation)c(ob)s(ject)i (iden)o(ti\014es)g(v)m(arious)h(prop)q(erties)g(of)g(a)g(comm)o(uni-)164 1146 y(cation)j(op)q(eration,)g(suc)o(h)g(as)g(the)g(bu\013er)g(descriptor)g (that)g(is)g(asso)q(ciated)g(with)g(it,)f(its)164 1206 y(con)o(text,)e(the)g (tag)h(and)g(destination)f(parameters)g(to)h(b)q(e)f(used)h(for)g(a)g(send,)f (or)h(the)f(tag)164 1266 y(and)f(source)g(parameters)f(to)h(b)q(e)g(used)g (for)h(a)f(receiv)o(e.)18 b(In)12 b(addition,)h(this)g(ob)s(ject)f(stores)164 1326 y(information)17 b(ab)q(out)h(the)g(status)g(of)g(the)g(last)f(comm)o (unication)e(op)q(eration)j(that)g(w)o(as)164 1387 y(p)q(erformed)f(with)g (this)h(ob)s(ject.)26 b(This)18 b(ob)s(ject)f(is)h(accessed)f(using)i(a)f (comm)o(unicati)o(on)164 1447 y(handle.)237 1507 y(One)i(can)h(consider)f (comm)o(uni)o(cation)e(op)q(erations)j(to)g(consist)g(of)f(the)g(follo)o (wing)164 1567 y(sub)q(op)q(erations:)164 1681 y Fi(INIT\(op)r(eration,)d (params,)i(handle\))25 b Fl(Pro)q(cess)15 b(pro)o(vides)f(all)f(relev)m(an)o (t)g(parame-)286 1742 y(ters)18 b(for)g(its)g(participation)g(in)g(the)f (comm)o(unication)e(op)q(eration)k(\(t)o(yp)q(e)f(of)g(op-)286 1802 y(eration,)e(data)i(bu\013er,)f(tag,)g(participan)o(ts,)f(etc.\).)21 b(An)c(ob)s(ject)f(is)g(created)g(that)286 1862 y(iden)o(ti\014es)f(the)h(op) q(eration.)164 1964 y Fi(ST)-5 b(AR)g(T\(handle\))26 b Fl(The)16 b(comm)o(unic)o(ation)e(op)q(eration)j(is)f(started)164 2065 y Fi(COMPLETE\(handle\))25 b Fl(The)16 b(comm)o(unication)d(op)q(eration)k (is)f(completed.)164 2167 y Fi(FREE\(handle\))24 b Fl(The)e(comm)o(unication) d(ob)s(ject,)j(and)h(asso)q(ciated)g(resources)f(are)286 2227 y(freed.)164 2341 y(Correct)16 b(in)o(v)o(o)q(cation)g(of)g(these)g(sub)q(op) q(erations)i(is)f(a)f(sequence)f(of)i(the)f(form)520 2451 y Fi(INIT)i Fl(\()p Fi(ST)-5 b(AR)g(T)20 b(COMPLETE)p Fl(\))1224 2431 y Fb(\003)1263 2451 y Fi(FREE)p Fd(:)949 2599 y Fl(21)p eop %%Page: 22 22 bop 164 307 a Fl(I.e.,)10 b(an)i(ob)s(ject)f(needs)h(b)q(e)f(created)h(b)q (efore)f(comm)o(unicati)o(on)f(o)q(ccurs;)j(it)e(can)g(b)q(e)h(reused)164 367 y(only)h(after)g(the)g(previous)g(use)g(has)h(completed;)d(and)j(it)f (needs)g(to)g(b)q(e)h(freed)e(ev)o(en)o(tually)164 428 y(\(of)i(course,)g (one)h(can)f(assume)f(that)i(all)e(ob)s(jects)h(are)g(freed)g(at)g(program)g (termination,)164 488 y(b)o(y)i(default\).)237 548 y(The)21 b(ab)q(o)o(v)o(e)f(scenario)g(p)q(ertains)h(to)g Fh(p)n(ersistent)g Fl(ob)s(jects.)33 b(One)20 b(can)h(also)g(create)164 608 y Fh(ephemer)n(al)15 b Fl(ob)s(jects.)20 b(Suc)o(h)14 b(ob)s(ject)f(p)q (ersists)i(only)f(un)o(til)f(the)h(comm)o(uni)o(cation)e(op)q(era-)164 668 y(tion)j(is)g(completed,)d(at)k(whic)o(h)e(p)q(oin)o(t)h(it)g(is)g (destro)o(y)o(ed.)20 b(Th)o(us,)15 b(correct)f(in)o(v)o(o)q(cation)h(of)164 729 y(sub)q(op)q(erations)j(with)e(an)h(ephemeral)d(ob)s(ject)i(is)g Fi(INIT)i(ST)-5 b(AR)g(T)19 b(COMPLETE)p Fl(.)237 789 y(A)i(user)g(ma)o(y)e (directly)h(in)o(v)o(ok)o(es)f(these)i(sub)q(op)q(erations.)37 b(This)22 b(w)o(ould)f(allo)o(w)g(to)164 849 y(amortize)16 b(the)i(o)o(v)o(erhead)f(of)h(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o (er)i(man)o(y)f(successiv)o(e)164 909 y(uses)23 b(of)h(the)e(same)g(handle,)i (and)g(allo)o(ws)f(to)g(o)o(v)o(erlap)g(comm)o(uni)o(cation)d(and)k(com-)164 969 y(putation.)29 b(Simpler)16 b(comm)o(unication)g(op)q(erations)k(com)o (bine)c(sev)o(eral)i(of)h(these)f(sub-)164 1029 y(op)q(erations)24 b(in)o(to)e(one)g(op)q(eration,)j(th)o(us)d(simplifying)e(the)i(use)g(of)h (comm)o(unicati)o(on)164 1090 y(primitiv)o(e)o(s.)f(Th)o(us,)c(one)f(only)h (needs)f(to)h(sp)q(ecify)e(precisely)g(the)h(seman)o(tics)f(of)h(these)164 1150 y(sub)q(op)q(erations)i(in)e(order)g(to)h(sp)q(ecify)e(the)h(seman)o (tics)f(of)h(MPI)g(p)q(oin)o(t)g(to)h(p)q(oin)o(t)f(com-)164 1210 y(m)o(unication)d(op)q(erations.)237 1270 y(W)l(e)h(sa)o(y)f(that)h(a)g (comm)o(unication)d(op)q(eration)j(\(send)g(or)g(receiv)o(e\))e(is)h Fi(p)r(osted)g Fl(once)164 1330 y(a)23 b Fi(start)g Fl(sub)q(op)q(eration)i (w)o(as)f(in)o(v)o(ok)o(ed;)g(the)f(op)q(eration)h(is)f Fi(completed)f Fl(once)h(the)164 1391 y Fi(complete)17 b Fl(sub)q(op)q(eration)j(completes.) k(A)17 b(send)h(and)h(a)f(receiv)o(e)e(op)q(eration)j Fi(matc)n(h)164 1451 y Fl(if)d(the)h(receiv)o(e)d(pattern)j(sp)q(eci\014ed)g(b)o(y)f(the)h (receiv)o(e)d(matc)o(hes)i(the)g(message)g(en)o(v)o(elop)q(e)164 1511 y(created)g(b)o(y)f(the)h(send.)164 1640 y Fi(1.9.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)164 1733 y Fl(An)g(ob)s(ject)g (for)g(a)h(send)f(op)q(eration)h(is)f(created)g(b)o(y)g(a)g(call)g(to)g Fi(MPI)p 1452 1733 17 2 v 21 w(INIT)p 1598 1733 V 20 w(SEND)p Fl(.)164 1793 y(A)g(call)g(to)g Fi(MPI)p 487 1793 V 21 w(INIT)p 633 1793 V 20 w(RECV)h Fl(is)f(similarly)d(used)k(for)g(creating)f(an)h(ob)s (ject)f(for)h(a)164 1853 y(receiv)o(e)8 b(op)q(eration.)21 b(The)11 b(creation)f(of)h(a)g(comm)o(unication)d(ob)s(ject)i(is)h(a)g(lo)q (cal)g(op)q(eration)164 1913 y(that)17 b(need)e(not)i(in)o(v)o(olv)o(e)d (comm)o(unicati)o(on)g(with)i(a)h(remote)e(pro)q(cess.)164 2034 y Fi(MPI)p 279 2034 V 20 w(INIT)p 424 2034 V 20 w(SEND)g(\(handle,)g (bu\013er)p 972 2034 V 20 w(handle,)h(dest,)f(tag,)g(con)n(text,)f(mo)r(de,) 164 2094 y(p)r(ersistence\))237 2214 y Fl(Creates)j(a)f(send)h(comm)o(uni)o (cation)d(ob)s(ject.)20 b(P)o(arameters)15 b(are)164 2313 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i (not)f(b)q(e)f(asso)q(ciated)i(with)286 2373 y(an)o(y)16 b(ob)s(ject)g(b)q (efore)g(the)g(call.)164 2474 y Fi(IN)i(bu\013er)p 394 2474 V 20 w(handle)25 b Fl(handle)16 b(to)h(send)f(bu\013er)h(descriptor)949 2599 y(22)p eop %%Page: 23 23 bop 164 307 a Fi(IN)18 b(dest)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i (destination)f(\(in)o(teger\))164 409 y Fi(IN)i(tag)25 b Fl(user)16 b(tag)h(for)g(messages)e(sen)o(t)h(with)g(this)g(handle)h(\(in)o(teger\))164 511 y Fi(IN)h(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages) e(sen)o(t)h(with)g(this)h(handle)164 612 y Fi(IN)h(mo)r(de)24 b Fl(send)11 b(mo)q(de)f(\(state)h(t)o(yp)q(e,)g(with)f(t)o(w)o(o)h(v)m (alues:)19 b Fg(MPI)p 1324 612 16 2 v 17 w(REGULAR)8 b Fl(and)k Fg(MPI)p 1699 612 V 17 w(READY)p Fl(\))164 714 y Fi(IN)18 b(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g(\(state)h(t)o(yp)q(e,)g(with)g(t)o(w)o(o)g (v)m(alues:)18 b Fg(MPI)p 1623 714 V 18 w(PERSISTEN)o(T)286 774 y Fl(and)f Fg(MPI)p 462 774 V 17 w(EPHEMERAL)p Fl(\))164 936 y Fi(MPI)p 279 936 17 2 v 20 w(INIT)p 424 936 V 20 w(RECV)c(\(handle,)h (bu\013er)p 975 936 V 19 w(handle,)g(source,)f(tag,)h(con)n(text,)e(p)r(er-) 164 996 y(sistence\))237 1117 y Fl(Create)k(a)h(receiv)o(e)d(handle.)21 b(P)o(arameters)15 b(are)164 1218 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i(not)f(b)q(e)f(asso)q(ciated)i(with) 286 1279 y(an)o(y)16 b(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1380 y Fi(IN)i(bu\013er)p 394 1380 V 20 w(handle)25 b Fl(handle)16 b(to)h(receiv)o(e)d(bu\013er)i(descriptor.)164 1482 y Fi(IN)i(source)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i(source,)f(or)g(DONTCARE)g(\(in)o (teger\).)164 1584 y Fi(IN)i(tag)25 b Fl(user)d(tag)h(for)f(messages)g (receiv)o(ed)e(with)i(this)g(handle,)h(or)f(DONTCARE)286 1644 y(\(in)o(teger\).)164 1746 y Fi(IN)c(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages)e(receiv)o(ed)f(with)j(this)f(handle.)164 1847 y Fi(IN)i(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g(\(state)h (t)o(yp)q(e,)g(with)g(t)o(w)o(o)g(v)m(alues:)18 b Fg(MPI)p 1623 1847 16 2 v 18 w(PERSISTEN)o(T)286 1907 y Fl(and)f Fg(MPI)p 462 1907 V 17 w(EPHEMERAL)p Fl(\))237 2009 y(See)f(Section)g(1.5.2)g(for)h(a) f(discussion)h(of)f(source,)g(tag)h(and)g(con)o(text.)164 2178 y Fk(Discussion:)48 b Fj(I)18 b(ha)o(v)o(e)f(not)f(included)k(prop)q(osals)e (for)e(partially)i(sp)q(eci\014ed)i(message)d(han-)164 2235 y(dles,)f(that)e(some)h(p)q(eoples)i(seem)e(to)g(desire.)237 2291 y(I)h(ha)o(v)o(e)e(merged)i(all)g(handle)g(setup)g(in)o(to)f(one)g (call.)949 2599 y Fl(23)p eop %%Page: 24 24 bop 164 307 a Fi(1.9.2)55 b(Comm)n(unication)21 b(Start)164 460 y(MPI)p 279 460 17 2 v 20 w(ST)-5 b(AR)g(T\(handle\))164 561 y(IN)18 b(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)237 663 y(The)j Fg(MPI)p 419 663 16 2 v 17 w(START)f Fl(function)g(starts)i(the)f (execution)e(of)j(a)f(comm)o(unic)o(ation)e(op)q(era-)164 723 y(tion)k(\(send)h(or)f(receiv)o(e\).)28 b(A)19 b(sender)g(should)h(not)g(up)q (date)g(the)f(send)h(bu\013er)f(after)h(a)164 784 y(send)e(op)q(eration)g (has)h(started)f(and)g(un)o(til)f(it)g(is)g(completed.)24 b(A)17 b(receiv)o(er)e(should)j(not)164 844 y(access)f(the)f(receiv)o(e)f(bu\013er)i (after)f(a)h(receiv)o(e)e(op)q(eration)i(w)o(as)h(started)f(and)g(un)o(til)f (it)g(is)164 904 y(completed.)21 b(A)16 b(program)h(that)g(do)q(es)h(not)f (satisfy)g(this)g(condition)f(is)h(erroneous)g(and)164 964 y(its)f(outcome)f(is)h(undetermined.)164 1094 y Fi(1.9.3)55 b(Comm)n(unication)21 b(Completion)164 1247 y(MPI)p 279 1247 17 2 v 20 w(W)-6 b(AIT)19 b(\()f(handle,)h(return)p 864 1247 V 20 w(status)p 1030 1247 V 20 w(handle\))164 1348 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)164 1450 y Fi(OUT)19 b(return)p 466 1450 V 19 w(handle)26 b Fl(handle)16 b(that)g(is)g(asso)q(ciated)i(with)e (return)g(status)h(ob)s(ject.)237 1552 y(A)23 b(call)f(to)h Fi(MPI)p 574 1552 V 21 w(W)-6 b(AIT)23 b Fl(returns)g(when)g(the)g(send)g(op) q(eration)h(iden)o(ti\014ed)d(b)o(y)164 1612 y Fg(handle)16 b Fl(is)i(complete.)24 b(The)18 b(completion)e(of)i(a)g(send)h(op)q(eration)f (indicates)g(that)g(the)164 1672 y(sender)h(is)g(no)o(w)h(free)e(to)i(up)q (date)f(the)g(lo)q(cations)h(in)f(the)g(send)g(bu\013er,)h(or)g(an)o(y)f (other)164 1732 y(lo)q(cation)e(that)f(can)h(b)q(e)f(referenced)f(b)o(y)h (the)g(send)g(op)q(eration.)23 b(Ho)o(w)o(ev)o(er,)14 b(it)h(do)q(es)i(not) 164 1792 y(indicate)11 b(that)i(the)f(message)g(has)h(b)q(een)f(receiv)o(ed;) f(rather)h(it)g(ma)o(y)f(ha)o(v)o(e)h(b)q(een)g(bu\013ered)164 1853 y(b)o(y)k(the)g(comm)o(uni)o(cation)e(subsystem.)237 1913 y(The)k(completion)e(of)h(a)i(receiv)o(e)c(op)q(eration)j(indicates)f(that)h (the)g(receiv)o(er)d(is)j(no)o(w)164 1973 y(free)e(to)i(access)f(the)g(lo)q (cations)h(in)f(the)g(receiv)o(e)e(bu\013er,)i(whic)o(h)g(con)o(tain)g(the)g (receiv)o(ed)164 2033 y(message,)f(or)i(an)o(y)g(other)f(lo)q(cation)h(that)g (can)f(b)q(e)h(referenced)e(b)o(y)h(the)g(receiv)o(e)e(op)q(era-)164 2093 y(tion.)21 b(It)16 b(do)q(es)h(not)g(indicate)e(that)i(the)f(matc)o (hing)e(send)j(op)q(eration)g(has)g(completed.)237 2154 y(The)g(call)e (returns)i(a)g(handle)f(to)h(an)g(opaque)g(ob)s(ject)f(that)h(con)o(tains)g (information)164 2214 y(on)g(the)f(completed)e(op)q(eration)j({)g(the)f Fi(return)i(status)e Fl(ob)s(ject.)164 2334 y Fi(MPI)p 279 2334 V 20 w(ST)-5 b(A)g(TUS)20 b(\(handle,)f(\015ag,)g(return)p 1029 2334 V 19 w(handle\))164 2436 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation) 14 b(handle)949 2599 y(24)p eop %%Page: 25 25 bop 164 307 a Fi(OUT)19 b(\015ag)25 b Fl(logical)164 409 y Fi(OUT)19 b(return)p 466 409 17 2 v 19 w(handle)26 b Fl(handle)16 b(that)g(is)g(asso)q(ciated)i(with)e(return)g(status)h(ob)s(ject.)237 511 y(A)h(call)g(to)h Fi(MPI)p 561 511 V 20 w(ST)-5 b(A)g(TUS)20 b Fl(returns)f Fg(flag=true)c Fl(if)j(the)g(op)q(eration)h(iden)o(ti\014ed) 164 571 y(b)o(y)d Fg(handle)f Fl(is)i(complete,)e(In)h(suc)o(h)h(case,)g(the) g(return)f(handle)h(p)q(oin)o(ts)h(to)f(an)h(opaque)164 631 y(ob)s(ject)h(that)h(con)o(tains)g(information)e(on)j(the)e(completed)e (information.)31 b(It)19 b(returns)164 691 y Fg(flag=false)o Fl(,)12 b(otherwise.)21 b(In)14 b(suc)o(h)h(case,)f(the)h(v)m(alue)g(of)g (the)g(return)f(handle)h(is)g(unde-)164 751 y(\014ned.)237 812 y(Implem)o(en)o(tation)e(notes:)237 872 y(A)g(call)f(to)h Fg(MPI)p 510 872 16 2 v 17 w(WAIT)f Fl(blo)q(c)o(ks)g(only)h(the)f(executing) g(thread.)20 b(If)13 b(the)f(executing)g(pro-)164 932 y(cess)h(is)f(m)o (ultithreaded,)f(then)i(other)g(threads)g(within)g(the)f(pro)q(cess)i(can)f (b)q(e)g(sc)o(heduled)164 992 y(for)j(execution.)237 1052 y(The)h(use)h(of)f (a)h(blo)q(c)o(king)f(receiv)o(e)e(op)q(eration)j(\()p Fg(MPI)p 1197 1052 V 17 w(WAIT)p Fl(\))e(allo)o(ws)h(the)g(op)q(erating)164 1112 y(system)k(to)h(desc)o(hedule)f(the)h(blo)q(c)o(k)o(ed)f(thread)i(and)f (sc)o(hedule)f(another)i(thread)g(for)164 1173 y(execution,)c(if)h(suc)o(h)g (is)g(a)o(v)m(ailable.)32 b(The)20 b(use)g(of)g(a)h(non)o(blo)q(c)o(king)e (receiv)o(e)f(op)q(eration)164 1233 y(\()p Fg(MPI)p 264 1233 V 17 w(STATUS)p Fl(\))11 b(allo)o(ws)i(the)f(user)h(to)g(sc)o(hedule)f (alternativ)o(e)g(activities)f(within)h(a)h(single)164 1293 y(thread)j(of)h(execution.)237 1353 y(The)h(in)o(tended)e(implem)o(en)o (tation)f(of)j Fg(MPI)p 1027 1353 V 17 w(STATUS)d Fl(is)j(for)g(that)g(op)q (eration)g(to)g(re-)164 1413 y(turn)e(as)g(so)q(on)h(as)g(p)q(ossible.)k(Ho)o (w)o(ev)o(er,)14 b(if)h(rep)q(eatedly)g(called)g(for)h(an)g(op)q(eration)g (that)164 1474 y(is)g(enabled,)f(it)h(m)o(ust)f(ev)o(en)o(tually)f(succeed.) 237 1534 y(The)g(return)f(status)h(ob)s(ject)f(for)g(a)h(send)g(op)q(eration) g(carries)f(no)h(information.)19 b(The)164 1594 y(return)13 b(status)h(ob)s(ject)f(for)h(a)f(receiv)o(e)e(op)q(eration)j(carries)f (information)f(on)i(the)f(source,)164 1654 y(tag)h(and)h(length)e(of)h(the)g (receiv)o(ed)d(message.)20 b(These)14 b(\014elds)f(are)h(required)e(b)q (ecause)i(the)164 1714 y(receiv)o(e)h(op)q(eration)j(ma)o(y)e(ha)o(v)o(e)h (sp)q(eci\014ed)g Fg(DONTCARE)e Fl(in)i(either)f(source)i(or)g(tag)g (\014eld,)164 1775 y(and)f(the)f(message)f(ma)o(y)g(ha)o(v)o(e)g(b)q(een)i (shorter)f(than)h(the)f(receiv)o(e)e(bu\013er.)164 1895 y Fi(MPI)p 279 1895 17 2 v 20 w(RETURN)p 546 1895 V 20 w(ST)-5 b(A)g(T\()19 b(handle,)g(len,)g(source,)f(tag\))164 1997 y(IN)g(handle)26 b Fl(handle)16 b(to)g(return)g(status)h(ob)s(ject)164 2098 y Fi(OUT)i(len)24 b Fl(di\013erence)15 b(b)q(et)o(w)o(een)h(length)g(of)g (receiv)o(e)e(bu\013er)i(and)h(length)f(of)g(receiv)o(ed)286 2159 y(message,)24 b(in)f(b)o(ytes.)42 b(Th)o(us,)25 b(the)e(v)m(alue)g (returned)g(is)g(zero)g(if)g(the)g(receiv)o(ed)286 2219 y(message)11 b(matc)o(hes)g(the)g(the)h(receiv)o(e)e(bu\013er,)j(p)q(ositiv)o(e)e(if)h(it) f(is)h(shorter)g(\(in)o(teger\).)164 2320 y Fi(OUT)19 b(source)24 b Fl(rank)16 b(of)h(message)e(sender)h(in)g(message)g(con)o(text)f(\(in)o (teger\).)164 2422 y Fi(OUT)k(tag)24 b Fl(tag)17 b(of)g(receiv)o(ed)d (message)h(\(in)o(teger\).)949 2599 y(25)p eop %%Page: 26 26 bop 164 420 a Fk(Discussion:)237 477 y Fj(I)18 b(put)g(the)g(di\013erence)h (b)q(et)o(w)o(een)f(message)f(bu\013er)h(and)g(message)f(length)h(as)g(the)g (v)m(alue)164 533 y(returned,)e(rather)f(than)g(length)h(of)g(receiv)o(ed)g (message,)f(so)h(that)e(it)i(migh)o(t)g(b)q(e)g(easy)f(to)g(test)164 589 y(for)g(exact)f(matc)o(h.)237 650 y(The)f(use)g(of)e(a)i(return)f(status) f(ob)s(ject,)h(rather)g(than)h(a)f(list)h(of)f(parameters)f(ma)o(y)h (simplify)164 710 y(the)k(use)g(of)g(MPI)f(routines,)i(if)f(the)g(v)m(alues)h (stored)e(in)i(the)f(ob)s(ject)f(are)h(seldom)g(c)o(hec)o(k)o(ed.)23 b(A)164 770 y(prede\014ned)17 b(return)e(status)f(ob)s(ject)h(should)h(b)q(e) g(pro)o(vided,)g(to)e(ease)h(programming.)164 1020 y Fi(1.9.4)55 b(Multiple)20 b(Completions)164 1113 y Fl(It)f(is)g(con)o(v)o(enien)o(t)e(to) i(b)q(e)g(able)g(to)h(w)o(ait)e(for)i(the)f(completion)e(of)i(an)o(y)g(or)h (all)e(the)h(op-)164 1173 y(erations)f(in)g(a)g(set,)g(rather)g(than)g(ha)o (ving)g(to)g(w)o(ait)g(for)g(sp)q(eci\014c)g(message.)25 b(A)17 b(call)h(to)164 1233 y Fg(MPI)p 245 1233 16 2 v 17 w(WAITANY)g Fl(or)j Fg(MPI)p 604 1233 V 17 w(STATUSANY)c Fl(can)k(b)q(e)f(used)h(to)f(w)o (ait)h(for)f(the)g(completion)f(of)164 1293 y(one)e(out)g(of)f(sev)o(eral)g (op)q(erations;)h(a)g(call)f(to)g Fg(MPI)p 1079 1293 V 18 w(WAITALL)e Fl(can)i(b)q(e)h(used)g(to)f(w)o(ait)h(for)164 1353 y(all)f(p)q(ending)g(op)q (erations)i(in)e(a)g(list.)164 1474 y Fi(MPI)p 279 1474 17 2 v 20 w(W)-6 b(AIT)h(ANY)20 b(\()e(list)p 710 1474 V 21 w(of)p 776 1474 V 21 w(handles,)h(index,)f(return)p 1338 1474 V 20 w(handle\))237 1594 y Fl(Blo)q(c)o(ks)k(un)o(til)g(one)g(of)h(the)g(op)q (erations)h(asso)q(ciated)f(with)g(the)f(comm)o(unicati)o(on)164 1654 y(handles)16 b(in)f(the)g(arra)o(y)h(has)g(completed.)j(Returns)d(the)f (index)g(of)g(that)h(handle)g(in)f(the)164 1715 y(arra)o(y)l(,)h(and)g (returns)g(the)g(status)g(of)h(that)f(op)q(eration)h(in)e(the)h(ob)s(ject)f (asso)q(ciated)i(with)164 1775 y(the)f(return)p 384 1775 15 2 v 17 w(handle.)21 b(The)c(parameters)e(are:)164 1889 y Fi(IN)j(list)p 324 1889 17 2 v 22 w(of)p 391 1889 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 1991 y Fi(OUT)19 b(index)24 b Fl(index)16 b(of)g(handle)g(for)h(op)q(eration)g (that)g(completed)d(\(in)o(teger\).)164 2092 y Fi(OUT)19 b(return)p 466 2092 V 19 w(handle)26 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)i(with)e (return)g(status)h(ob)s(ject.)286 2153 y(Set)c(to)h(return)f(status)h(of)f (op)q(eration)h(that)g(completed.)237 2267 y(The)11 b(successful)f(execution) g(of)h Fg(MPI)p 892 2267 16 2 v 17 w(WAITANY\(lis)o(t)p 1220 2267 V 15 w(of)p 1287 2267 V 18 w(handles,)23 b(index,)g(return)p 1871 2267 V 17 w(handle\))164 2327 y Fl(is)11 b(equiv)m(alen)o(t)e(to)i(the)g (successful)f(execution)g(of)h Fg(MPI)p 1127 2327 V 17 w(WAIT\(handl)o(e[i)o (],)22 b(return)p 1710 2327 V 16 w(handle\))p Fl(,)164 2387 y(where)15 b Fg(i)g Fl(is)h(the)f(v)m(alue)g(returned)g(b)o(y)g Fg(index)f Fl(and)i Fg(handle[i])d Fl(is)i(the)g Fg(i)p Fl(-th)h(handle)f(in) 164 2447 y(the)h(list.)949 2599 y(26)p eop %%Page: 27 27 bop 237 307 a Fl(If)15 b(more)f(then)h(one)g(op)q(eration)i(is)e(enabled)g (and)g(can)h(terminate,)d(one)i(is)g(arbitrar-)164 367 y(ily)20 b(c)o(hosen)i(\(sub)s(ject)f(to)g(the)h(restrictions)f(on)h(op)q(eration)g (termination)e(order,)i(see)164 428 y(Section)16 b(1.13\).)164 548 y Fg(MPI)p 245 548 16 2 v 17 w(WAITANY)23 b(\()j(list)p 623 548 V 17 w(of)p 692 548 V 17 w(handles,)d(index,)g(return)p 1275 548 V 17 w(status)p 1448 548 V 16 w(handle\))164 608 y Fl(is)241 690 y Fg({MPI_WAIT_)o(RE)o(CV)f(\(handle[1],)g(return_han)o(dle)o (\);)g(index)i(=)h(0})g(||)g(...)164 750 y(||)241 810 y({MPI_WAIT_)o(RE)o(CV) d(\(handle[n],)g(return_han)o(dle)o(\);)g(index)i(=)h(n-1})237 892 y Fl(\(\\)p Fe(jj)p Fl(")18 b(indicates)f(c)o(hoice;)f(one)i(of)f(the)g (alternativ)o(es)g(is)g(c)o(hosen,)g(nondeterministi-)164 953 y(cally)l(.\))164 1073 y Fi(MPI)p 279 1073 17 2 v 20 w(ST)-5 b(A)g(TUSANY)21 b(\()d(list)p 777 1073 V 22 w(of)p 844 1073 V 20 w(handles,)h(index,)f(return)p 1405 1073 V 20 w(handle\))237 1193 y Fl(Causes)c(either)e(one)h(or)h(none)f(of)g(the)g(op)q(erations)h (asso)q(ciated)g(with)f(the)g(comm)o(uni-)164 1254 y(cation)g(handles)f(to)h (return.)20 b(In)12 b(the)g(former)g(case,)g(it)g(has)i(the)e(same)f(return)i (seman)o(tics)164 1314 y(as)j(a)g(call)f(to)h Fg(MPI)p 492 1314 16 2 v 17 w(WAIT)p 613 1314 V 17 w(ANY)p Fl(.)e(In)i(the)f(later)g (case,)g(it)g(returns)h(a)g(v)m(alue)f(of)h(-1)g(in)g Fg(index)164 1374 y Fl(and)h Fg(return)p 418 1374 V 16 w(handle)d Fl(is)i(unde\014ned.)21 b(The)c(parameters)e(are:)164 1462 y Fi(IN)j(list)p 324 1462 17 2 v 22 w(of)p 391 1462 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to) f(comm)o(unication)d(ob)s(jects.)164 1557 y Fi(OUT)19 b(index)24 b Fl(index)e(of)h(handle)g(for)g(op)q(eration)h(that)g(completed,)d(or)j(-1)f (if)g(none)286 1617 y(completed)14 b(\(in)o(teger\).)164 1712 y Fi(OUT)19 b(return)p 466 1712 V 19 w(handle)26 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)i(with)e(return)g(status)h(ob)s(ject.)286 1772 y(Set)11 b(to)h(return)f(status)i(of)f(op)q(eration)g(that)g(completed,) e(if)h(an)o(y;)h(unde\014ned)g(when)286 1833 y Fg(index)24 b(=)h(-1)p Fl(.)237 1920 y Fi(MPI)p 352 1920 V 21 w(W)-6 b(AIT)h(ALL\(list)p 728 1920 V 20 w(of)p 793 1920 V 21 w(handles,)19 b(list)p 1106 1920 V 21 w(of)p 1172 1920 V 20 w(return)p 1348 1920 V 20 w(handles\))237 2041 y Fl(Blo)q(c)o(ks)14 b(un)o(til)f(all)i(comm)o(uni)o(cation)d(op)q (erations)k(asso)q(ciated)g(with)e(handles)h(in)f(the)164 2101 y(list)i(complete,)e(and)j(return)g(the)f(status)h(of)g(all)f(these)h(op)q (erations.)23 b(The)17 b(parameters)164 2161 y(are:)164 2249 y Fi(IN)h(list)p 324 2249 V 22 w(of)p 391 2249 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 2344 y Fi(OUT)19 b(list)p 384 2344 V 21 w(of)p 450 2344 V 20 w(return)p 626 2344 V 20 w(handles)25 b Fl(Must)18 b(ha)o(v)o(e)f(the)h(same) f(length)h(as)h(the)f(\014rst)g(list.)286 2404 y(Eac)o(h)11 b(return)g(status)g(ob)s(ject)g(is)g(set)f(to)i(the)e(return)h(status)h(of)f (the)g(corresp)q(onding)286 2464 y(op)q(eration)17 b(in)f(the)g(\014rst)g (list.)949 2599 y(27)p eop %%Page: 28 28 bop 164 307 a Fm(1.10)70 b(Blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Blo)q(c)o(king)d(send)h(and)h(receiv)o(e)d(op)q(erations)j(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(er-)164 460 y(ations)k(in)o(to)f (one)h(call.)32 b(The)21 b(op)q(eration)g(returns)f(only)g(when)h(the)f(comm) o(unicati)o(on)164 520 y(completes)g(and)i(no)g(comm)o(unic)o(ation)e(ob)s (ject)h(p)q(ersists)h(after)f(the)h(call)f(completed.)164 580 y(Ho)o(w)o(ev)o(er,)14 b(the)i(bu\013er)h(descriptor)e(ob)s(ject)h(needs)g(b) q(e)h(created)e(ahead)i(of)g(the)f(call.)237 640 y(W)l(e)g(use)g(the)g(follo) o(wing)g(naming)g(con)o(v)o(en)o(tion)f(for)h(suc)o(h)g(op)q(erations:)777 695 y Fa(")824 737 y Fe(\000)822 797 y Fi(R)885 695 y Fa(#)8 b(")966 737 y Fi(SEND)962 797 y(RECV)1145 695 y Fa(#)237 898 y Fl(The)16 b(\014rst)h(letter)e(\(v)o(oid)h(or)g Fi(R)p Fl(\))g(indicates)g (the)g(start)h(mo)q(de)e(\(regular)h(or)h(ready\).)164 1018 y Fi(MPI)p 279 1018 17 2 v 20 w(SEND)i(\(bu\013er)p 639 1018 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1078 y Fl(is)164 1170 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1231 y(MPI_START\()o(han)o (dl)o(e\))164 1291 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1443 y Fi(MPI)p 279 1443 V 20 w(RSEND)c(\(bu\013er)p 681 1443 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1503 y Fl(is)164 1596 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1656 y(MPI_START\()o(han)o(dl)o (e\))164 1716 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1868 y Fi(MPI)p 279 1868 V 20 w(RECV\(bu\013er)p 626 1868 V 20 w(handle,)c (source,)f(tag,)g(con)n(text,)g(return)p 1514 1868 V 19 w(handle\))164 1929 y Fl(is)164 2021 y Fg(MPI_INIT_R)o(ECV)o(\(h)o(and)o(le,)k(buffer_han)o (dl)o(e,)g(source,)h(tag,)i(context,)d(EPHEMERAL\))164 2081 y(MPI_START\()o(han)o(dl)o(e\))164 2141 y(MPI_WAIT\(h)o(and)o(le)o(,re)o(tur) o(n_h)o(an)o(dle)o(\))237 2233 y Fi(Implemen)n(tation)d(note:)237 2293 y Fl(While)k(these)g(functions)h(can)g(b)q(e)g(impleme)o(n)o(ted)d(via)i (calls)g(to)i(functions)e(that)164 2354 y(implem)o(en)o(t)c(sub)q(op)q (erations,)25 b(as)e(describ)q(ed)f(in)g(this)g(subsection,)h(an)g(e\016cien) o(t)d(im-)164 2414 y(plemen)o(tation)d(ma)o(y)h(optimize)f(a)o(w)o(a)o(y)i (these)g(m)o(ultiple)d(calls,)j(pro)o(vided)f(it)h(do)q(es)h(not)164 2474 y(c)o(hange)c(the)g(b)q(eha)o(vior)g(of)h(correct)e(programs.)949 2599 y(28)p eop %%Page: 29 29 bop 164 307 a Fm(1.11)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Non)o(blo)q(c)o(king)g(send)h(and)g(receiv)o(e)d(op)q(erations)k (com)o(bine)d(the)h(\014rst)h(t)o(w)o(o)g(sub)q(op)q(era-)164 460 y(tions)d(\()p Fg(INIT)e Fl(and)i Fg(START)p Fl(\))e(in)o(to)h(one)h (call.)30 b(They)19 b(use)h(ephemeral)d(comm)o(unicati)o(on)164 520 y(ob)s(jects,)e(so)h(that)f(the)g(op)q(eration)i(is)e(completed,)d(and)k (the)f(asso)q(ciated)i(resources)e(are)164 580 y(freed,)g(b)o(y)h(using)h (one)f(of)h(the)f(functions)g Fg(MPI)p 1014 580 16 2 v 17 w(WAIT,)24 b(MPI)p 1263 580 V 17 w(STATUS,)f(MPI)p 1563 580 V 18 w(WAITANY,)164 640 y(MPI)p 245 640 V 17 w(STATUSANY)p Fl(,)11 b(or)j Fg(MPI)p 656 640 V 17 w(WAITALL)p Fl(.)d(Here,)i(to)q(o,)i(a)g(bu\013er)f(ob)s(ject)f (has)i(to)f(b)q(e)g(created)164 700 y(ahead)j(of)f(the)g(comm)o(unication)d (initiation)j(op)q(eration.)237 761 y(W)l(e)h(use)g(the)f(same)g(naming)g (con)o(v)o(en)o(tion)g(as)i(for)f(blo)q(c)o(king)f(op)q(erations:)24 b(a)17 b(pre\014x)164 821 y(of)f Fi(R)g Fl(indicates)g(the)f Fg(READY)g Fl(mo)q(de.)20 b(In)c(addition,)g(a)g(pre\014x)g(of)g Fi(I)g Fl(is)g(used)g(to)h(indicate)164 881 y Fh(imme)n(diate)f Fl(\(i.e.,)e(non)o(blo)q(c)o(king\))i(execution.)164 1001 y Fi(MPI)p 279 1001 17 2 v 20 w(ISEND)j(\(handle,)g(bu\013er)p 856 1001 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1062 y Fl(is)164 1163 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o (e,)g(dest,)i(tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1224 y(MPI_START\()o(han)o(dl)o(e\))164 1385 y Fi(MPI)p 279 1385 V 20 w(IRSEND)18 b(\(handle,)h(bu\013er)p 897 1385 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1446 y Fl(is)164 1547 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1608 y(MPI_START\()o(han)o(dl)o (e\))164 1769 y Fi(MPI)p 279 1769 V 20 w(IRECV\(handle,)15 b(bu\013er)p 839 1769 V 19 w(handle,)f(source,)f(tag,)g(con)n(text,)g(return) p 1706 1769 V 19 w(status)p 1871 1769 V 21 w(handle\))164 1830 y Fl(is)164 1931 y Fg(MPI_INIT_R)o(ECV)o(\(h)o(and)o(le,)22 b(buffer_han)o(dl)o(e,)g(source,)h(tag,)i(context,)d(EPHEMERAL\))164 1991 y(MPI_START\()o(han)o(dl)o(e\))164 2136 y Fm(1.12)70 b(Blo)r(c)n(k)22 b(Sending)h(Op)r(erations)164 2228 y Fl(The)h(most)g(frequen)o(t)f(t)o(yp)q (e)g(of)i(bu\013er)f(used)g(is)g(a)h(con)o(tiguous)g(bu\013er)f(of)h(n)o (umeric)164 2288 y(storage)18 b(units,)f(i.e.,)f(a)h(con)o(tiguous)h (bu\013er)g(of)f(w)o(ords)h(that)g(ma)o(y)d(con)o(tain)i(either)f(IN-)164 2349 y(TEGER,)23 b(REAL)g(or)h(LOGICAL)f(v)m(alues)g(\(in)g(F)o(OR)l(TRAN\).) e(In)h(a)i(homogeneous)164 2409 y(en)o(vironmen)o(t)14 b(suc)o(h)j(messages)g (can)g(b)q(e)g(used)h(to)f(send)h(arbitrary)f(sequences)f(of)h(con-)164 2469 y(tiguous)g(items,)d(where)i(eac)o(h)f(item)g(o)q(ccupies)h(an)g(in)o (teger)g(n)o(um)o(b)q(er)e(of)j(w)o(ords.)949 2599 y(29)p eop %%Page: 30 30 bop 237 307 a Fl(W)l(e)17 b(sp)q(ecialize)f(the)i(functions)f(in)g(the)h(t)o (w)o(o)f(previous)g(subsections)h(to)g(this)f(case,)164 367 y(th)o(us)f(a)o(v)o(oiding)f(the)g(need)g(for)h(the)g(creation)f(of)h(a)g (bu\013er)g(descriptor)f(ob)s(ject.)21 b(W)l(e)15 b(use)164 428 y(the)i(same)f(naming)g(sc)o(heme)f(used)i(in)g(the)f(previous)h (subsections,)g(and)h(app)q(end)g(a)f Fi(B)164 488 y Fl(in)f(the)g(function)g (name,)f(for)h Fg(BLOCK)p Fl(.)164 608 y Fi(MPI)p 279 608 17 2 v 20 w(SENDB)j(\(start,)f(len,)g(dest,)g(tag,)g(con)n(text\))164 668 y Fl(is)164 770 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 830 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o (dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 890 y(MPI_SEND)f(\(buffer_h)o(and)o (le,)f(len,)i(dest,)g(tag,)g(context\))164 1052 y Fi(MPI)p 279 1052 V 20 w(RSENDB)18 b(\(bu\013er)p 720 1052 V 20 w(handle,)h(dest,)f (tag,)g(con)n(text\))164 1112 y Fl(is)164 1214 y Fg(MPI_CREATE)o(\(bu)o(ff)o (er_)o(han)o(dle)o(,)k(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1274 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\)) 164 1335 y(MPI_RSEND)e(\(buffer_han)o(dle)o(,)g(len,)j(dest,)e(tag,)i (context\))237 1436 y Fi(MPI)p 352 1436 V 21 w(RECVB\(bu\013er)p 740 1436 V 19 w(handle,)14 b(source,)f(tag,)g(con)n(text,)g(return)p 1607 1436 V 20 w(status)p 1773 1436 V 20 w(handle\))164 1496 y Fl(is)164 1598 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1658 y(MPI_ADD\(bu)o(ffe)o(r_)o (han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 1719 y(MPI_RECV\(b)o(uff)o (er)o(_ha)o(ndl)o(e,)e(source,)h(tag,)h(context,)f(return_sta)o(tus)o(_ha)o (nd)o(le\))164 1880 y Fi(MPI)p 279 1880 V 20 w(ISENDB)c(\(handle,)f(bu\013er) p 895 1880 V 20 w(handle,)h(dest,)f(tag,)g(con)n(text\))164 1941 y Fl(is)164 2042 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 2103 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 2163 y(MPI_ISEND\()o(han)o(dl)o (e,)e(buffer_hand)o(le,)g(dest,)i(tag,)g(context\))164 2325 y Fi(MPI)p 279 2325 V 20 w(IRSENDB)18 b(\(handle,)h(bu\013er)p 937 2325 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 2385 y Fl(is)949 2599 y(30)p eop %%Page: 31 31 bop 164 307 a Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 367 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 428 y(MPI_IRSEND)o(\(ha)o(nd)o (le,)e(buffer_han)o(dle)o(,)g(dest,)i(tag,)g(context\))164 587 y Fi(MPI)p 279 587 17 2 v 20 w(IRECVB\(handle,)14 b(bu\013er)p 878 587 V 20 w(handle,)g(source,)f(tag,)g(con)n(text,)g(return)p 1746 587 V 19 w(status)p 1911 587 V 21 w(handle\))164 647 y Fl(is)164 746 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 807 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 867 y(MPI_IRECV\()o(han)o(dl)o(e,) e(buffer_hand)o(le,)g(source,)h(tag,)h(context\))164 1075 y Fk(Discussion:)237 1132 y Fj(I)18 b(use)f(w)o(ord)g(coun)o(t,)g(rather)f (than)h(b)o(yte)g(coun)o(t.)26 b(I)18 b(b)q(eliev)o(e)h(w)o(ord)d(messages)h (are)g(m)o(uc)o(h)164 1188 y(more)e(prev)m(alen)o(t)g(than)g(b)o(yte)g (messages,)f(and)h(it's)g(a)g(blessing)h(not)f(to)f(ha)o(v)o(e)h(to)f(m)o (ultiply)j(b)o(y)164 1245 y(4)h(for)g(eac)o(h)h(message.)30 b(Byte)19 b(messages)f(are)g(still)j(a)o(v)m(ailable)f(the)f(hard)f(w)o(a)o (y)l(.)30 b(Also,)20 b(w)o(ord)164 1301 y(messages)15 b(\014t)g(w)o(ell)h (with)f(the)h(F)l(ortran)e(n)o(umeric)i(storage)e(unit)i(concept.)237 1357 y(If)21 b(w)o(e)f(are)g(to)g(add)h(more)f(functions,)j(m)o(y)d(next)g (addition)i(w)o(ould)f(b)q(e)g(a)f(v)o(ector)g(send)164 1414 y(op)q(eration.)164 1739 y Fm(1.13)70 b(Correctness)164 1951 y Fk(Discussion:)27 b Fj(The)18 b(material)h(in)g(this)g(section)g(has)g(not) f(y)o(et)g(b)q(een)h(discussed)h(b)o(y)f(MPIF.)164 2011 y(Some)d(or)g(all)h (of)f(it)h(is)f(lik)o(ely)i(to)e(mo)o(v)o(e)g(to)f(Section)i Fk(??)p Fj(.)23 b(It)16 b(is)h(incorp)q(orated)g(here)f(for)g(com-)164 2072 y(pleteness.)164 2322 y Fi(1.13.1)55 b(Order)164 2414 y Fl(MPI)15 b(preserv)o(es)g(the)h(order)g(of)g(messages)f(b)q(et)o(w)o(een)g (an)o(y)h(\014xed)f(pair)h(of)h(pro)q(cesses.)k(In)164 2474 y(other)16 b(w)o(ords,)g(if)f(pro)q(cess)h(A)f(executes)f(t)o(w)o(o)i (successiv)o(e)e(send)i Fg(start)d Fl(sub)q(op)q(erations,)949 2599 y(31)p eop %%Page: 32 32 bop 164 307 a Fl(pro)q(cess)13 b(B)e(executes)g(t)o(w)o(o)h(successiv)o(e)e (receiv)o(e)g Fg(start)g Fl(op)q(erations,)k(and)e(b)q(oth)h(receiv)o(es)164 367 y(matc)o(h)19 b(either)h(sends,)i(then)f(the)f(\014rst)h(receiv)o(e)e (will)h(receiv)o(e)e(the)j(message)f(sen)o(t)g(b)o(y)164 428 y(the)g(\014rst)g(send,)h(and)g(the)f(second)g(receiv)o(e)e(will)h(receiv)o (e)f(the)i(message)g(sen)o(t)f(b)o(y)h(the)164 488 y(second)h(send.)36 b(Th)o(us,)22 b(if)e(a)i(t)o(w)o(o)f(messages)f(from)g(the)h(same)f(source)h (can)g(satisfy)h(a)164 548 y(p)q(ending)d(receiv)o(e,)c(the)j(\014rst)h (message)e(sen)o(t)h(is)g(accepted;)g(if)g(a)g(message)g(can)g(satisfy)164 608 y(t)o(w)o(o)e(p)q(ending)h(receiv)o(es,)c(the)j(\014rst)h(receiv)o(e)d(p) q(osted)j(is)f(satis\014ed.)237 668 y(The)h(last)g(paragraph)i(assumes)e (that)g(the)g(send)g Fg(start)f Fl(op)q(erations)i(are)f(ordered)164 729 y(b)o(y)k(the)g(program)h(order)f(at)h(pro)q(cess)g(A,)f(and)h(the)f (receiv)o(e)f Fg(start)f Fl(op)q(erations)k(are)164 789 y(ordered)f(b)o(y)f (the)h(program)f(order)h(at)g(pro)q(cess)h(B.)e(If)g(a)h(pro)q(cess)h(is)e(m) o(ultithreaded)164 849 y(and)f(the)f(op)q(erations)h(are)f(executed)f(b)o(y)g (distinct)h(threads,)g(then)g(the)g(seman)o(tics)f(of)164 909 y(the)d(threaded)g(system)e(ma)o(y)h(not)h(de\014ne)g(an)g(order)h(b)q(et)o (w)o(een)e(the)g(t)o(w)o(o)h(op)q(erations,)h(in)164 969 y(whic)o(h)g(case)g (the)g(condition)g(is)g(v)o(oid.)164 1099 y Fi(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)164 1192 y Fl(W)l(e)22 b(can)h(mo)q(del)f(the)g (execution)g(of)h(MPI)f(programs)h(as)h(an)f(in)o(teraction)f(b)q(et)o(w)o (een)164 1252 y(executing)h(pro)q(cesses)h(that)g(execute)e(eac)o(h)h(their)g (o)o(wn)h(program,)h(and)f(the)g Fi(com-)164 1312 y(m)n(unication)c (subsystem)p Fl(.)g(The)c(comm)o(unic)o(ation)d(subsystem)i(ma)o(y)f(ha)o(v)o (e)h(v)m(arious)164 1372 y(constrain)o(ts)h(on)h(the)f(amoun)o(t)g(of)g (resources)g(it)g(can)h(use.)k(E.g.:)237 1432 y(Bounds)e(on)f(the)g(n)o(um)o (b)q(er)e(and)i(total)h(sizes)e(of)h(activ)o(e)f(comm)o(unic)o(ation)f(ob)s (jects.)164 1492 y(Suc)o(h)g(b)q(ound)h(can)g(b)q(e)f(global,)g(p)q(er)h(no)q (de,)f(or)h(p)q(er)f(pair)g(of)h(comm)o(unic)o(ating)d(no)q(des.)237 1553 y(Bounds)g(on)g(the)f(n)o(um)o(b)q(er)f(and)i(total)g(sizes)f(of)h (messages)f(bu\013ered)g(in)h(the)f(system.)164 1613 y(Suc)o(h)19 b(b)q(ound)h(can,)g(again,)g(b)q(e)g(global,)g(p)q(er)f(no)q(de,)h(or)g(p)q (er)f(pair)h(of)f(comm)o(unicati)o(ng)164 1673 y(no)q(de.)j(In)16 b(addition,)f(a)i(message)e(ma)o(y)g(b)q(e)h(bu\013ered)g(at)g(the)g(sender,) g(at)g(the)g(receiv)o(er,)164 1733 y(at)h(b)q(oth,)f(or)h(p)q(erhaps)g(at)g (another)g(place)e(altogether.)237 1793 y(Th)o(us,)g(it)g(will)f(b)q(e)h (di\016cult)e(to)j(set)f(rules)f(on)i(resource)e(managemen)o(t)f(of)i(the)g (com-)164 1854 y(m)o(unication)g(subsystem.)21 b(Ho)o(w)o(ev)o(er,)15 b(it)h(is)g(generally)g(exp)q(ected)f(that)i(impleme)o(n)o(ters)164 1914 y(will)e(pro)o(vide)f(information)h(on)h(the)g(mec)o(hanism)c(used)k (for)g(resource)g(allo)q(cation,)f(and)164 1974 y(that)22 b(query)g(and)g (set)g(functions)g(will)f(allo)o(w)h(to)g(query)g(and)g(p)q(ossibly)g(con)o (trol)g(the)164 2034 y(amoun)o(t)15 b(of)i(a)o(v)m(ailable)f(resources.)237 2094 y(W)l(e)d(pro)o(vide)f(in)g(this)h(section)g(a)g(set)g(of)g(minim)o(al)d (requiremen)o(ts)g(on)j(the)g(comm)o(uni-)164 2155 y(cation)h(subsystem.)19 b(Programs)14 b(that)g(execute)e(on)j(an)o(y)e(subsystem)g(that)h(ful\014ls)f (these)164 2215 y(minim)o(al)18 b(requiremen)o(ts)g(are)j Fi(safe)g Fl(and)h(will)d(p)q(ort)j(to)f(an)o(y)g(MPI)f(impleme)o(n)o(tation.)164 2275 y Fi(Unsafe)h Fl(programs)h(ma)o(y)d(execute)h(on)i(some)e(MPI)h(implem) o(e)o(n)o(tations,)f(dep)q(ending)164 2335 y(on)h(the)f(amoun)o(t)f(of)i(a)o (v)m(ailable)e(resources)h(and)h(the)f(implem)o(en)o(tation)d(used)k(for)f (the)164 2395 y(MPI)14 b(comm)o(unication)e(subsystem.)19 b(Finally)14 b Fi(erroneous)g Fl(programs)h(nev)o(er)e(execute)164 2456 y(correctly)l(.)19 b(\(While)14 b(it)g(is)g(desirable)g(to)g(detect)g (erroneous)h(programs,)g(it)f(is)g(not)h(p)q(ossi-)949 2599 y(32)p eop %%Page: 33 33 bop 164 307 a Fl(ble)13 b(to)g(do)h(so)g(at)g(compile)d(time,)h(and)i(often)f (prohibitiv)o(e)f(to)i(do)f(so)i(a)e(run)h(time.)k(Th)o(us,)164 367 y(the)e(do)q(cumen)o(t)f(do)q(es)j(not)f(sp)q(ecify)e(a)i(b)q(eha)o(vior) g(for)g(erroneous)g(programs,)f(although)164 428 y(the)g(desired)g(b)q(eha)o (vior)g(is)g(to)g(return)g(a)h(useful)f(error)g(message.\))224 542 y(1.)24 b(If)15 b(a)h(pro)q(cess)h(executes)d(an)j Fg(INIT)d Fl(op)q(eration,)i(then)g(the)f(op)q(eration)i(ev)o(en)o(tually)286 602 y(succeeds,)c(or)g(a)g Fh(r)n(esour)n(c)n(e)h(exc)n(eption)g Fl(o)q(ccurs.)21 b(The)13 b(standard)h(do)q(es)g(not)g(sp)q(ecify)286 662 y(when)23 b(a)h(resource)g(exception)e(is)h(allo)o(w)o(ed)g(to)h(o)q (ccur.)43 b(It)23 b(is)g(exp)q(ected)f(that)286 722 y(an)g(op)q(erational)h (de\014nition)e(will)g(b)q(e)h(made)f(a)o(v)m(ailable,)h(in)f(the)h(form)f (of)h(test)286 782 y(programs)14 b(that)f(ha)o(v)o(e)g(to)h(execute)e(with)h (no)h(resource)g(exceptions.)19 b(It)13 b(is)g(highly)286 843 y(desirable)e(to)i(ha)o(v)o(e)e(generous)i(b)q(ounds)h(on)e(the)g(n)o(um)o(b) q(er)f(of)h(concurren)o(tly)f(activ)o(e)286 903 y(comm)o(unic)o(ation)20 b(ob)s(jects)j(eac)o(h)f(pro)q(cess)h(ma)o(y)e(ha)o(v)o(e,)i(so)h(that,)g(in) e(practice,)286 963 y Fg(INIT)15 b Fl(op)q(erations)i(will)e(alw)o(a)o(ys)h (b)q(e)h(guaran)o(teed)f(to)h(succeed.)224 1065 y(2.)24 b(Eac)o(h)d(pro)q (cess)g(can)g(initiate)f(a)h(comm)o(uni)o(cation)d(op)q(eration)k(for)f(eac)o (h)f(activ)o(e)286 1125 y(comm)o(unic)o(ation)e(ob)s(ject.)32 b(I.e.)f(correct)20 b Fg(START)e Fl(op)q(erations)j(alw)o(a)o(ys)f(succeed) 286 1185 y(\(ev)o(en)o(tually\).)224 1287 y(3.)k(A)10 b(send)h(op)q(eration)h (is)f Fi(enabled)g Fl(if)f(the)h(sending)g(pro)q(cess)g(has)h(issued)e(a)i Fg(COMPLETE)286 1347 y Fl(op)q(eration)20 b(and)g(the)g(receiving)e(pro)q (cess)i(has)g(issued)g(a)f Fg(START)f Fl(op)q(eration)j(for)286 1407 y(a)e(matc)o(hing)f(receiv)o(e.)28 b(Symmetri)o(call)o(y)l(,)16 b(a)k(receiv)o(e)d(op)q(eration)j(is)f Fi(enabled)g Fl(if)286 1467 y(the)h(receiving)e(pro)q(cess)j(has)f(issued)g(a)g Fg(COMPLETE)d Fl(op)q(eration)k(and)g(the)f(send-)286 1528 y(ing)j(pro)q(cess)h(has)g (issued)f(a)g Fg(START)f Fl(op)q(eration)i(for)f(a)h(matc)o(hing)d(send.)42 b(An)286 1588 y(enabled)19 b(op)q(eration)i(ma)o(y)d(b)q(ecome)g Fi(disabled)j Fl(either)e(b)q(ecause)h(it)f(completes)286 1648 y(successfully)c(or,)h(in)g(the)g(case)g(of)h(a)g(receiv)o(e,)c(b)q(ecause)k (the)f(matc)o(hing)f(message)286 1708 y(is)h(successfully)f(receiv)o(ed)f(b)o (y)i(another)g(receiv)o(e)e(op)q(eration.)286 1789 y Fi(An)30 b(enabled)h(op)r(eration)f(either)f(completes)g(successfully)i(or)f(b)r(e-) 286 1849 y(comes)18 b(p)r(ermanen)n(tly)g(disabled.)224 1951 y Fl(4.)24 b(A)16 b Fg(FREE)f Fl(op)q(eration)i(alw)o(a)o(ys)f(succeeds)f (\(ev)o(en)o(tually\).)237 2065 y(The)f(four)h(conditions)f(guaran)o(tee)h (progress)g(in)e(the)h(comm)o(unication)d(subsystem.)164 2125 y(The)19 b(third)f(condition)h(guaran)o(tee)g(\(w)o(eak\))f(fairness)h(among) f(comp)q(eting)g(comm)o(uni-)164 2186 y(cation)e(requests.)237 2246 y(Examples)f(\(in)o(v)o(olving)g(t)o(w)o(o)h(pro)q(cessors)h(with)f (ranks)h(0)g(and)f(1\))237 2306 y(The)g(follo)o(wing)g(program)g(is)g(safe,)g (and)h(should)g(alw)o(a)o(ys)f(succeed.)164 2468 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))949 2599 y Fl(33)p eop %%Page: 34 34 bop 164 307 a Fg(IF)25 b(\(rank.EQ.0)o(\))d(THEN)241 367 y(CALL)i (MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))241 428 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 488 y(ELSE)101 b(!)25 b(rank.EQ.1)241 548 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o (f,)e(len,)i(0,)h(tag,)f(context\))241 608 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o (f,)e(len,)i(0,)h(tag,)f(context\))164 668 y(END)h(IF)237 765 y Fl(The)16 b(follo)o(wing)g(program)g(is)g(erroneous,)h(and)g(should)f(alw)o (a)o(ys)g(fail.)164 934 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 994 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 1054 y(CALL)i(MPI_RECVB\()o(rec)o(vbu) o(f,)e(len,)i(1,)h(tag,)f(context\))241 1114 y(CALL)g(MPI_SENDB\()o(sen)o (dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 1174 y(ELSE)101 b(!)25 b(rank.EQ.1)241 1235 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i (0,)h(tag,)f(context\))241 1295 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,) i(0,)h(tag,)f(context\))164 1355 y(END)h(IF)237 1463 y Fl(The)17 b(receiv)o(e)e(op)q(eration)j(of)f(the)f(\014rst)i(pro)q(cess)f(m)o(ust)f (complete)e(b)q(efore)k(its)e(send,)164 1523 y(and)k(can)g(complete)e(only)h (if)h(the)f(matc)o(hing)f(send)i(of)g(the)g(second)g(pro)q(cessor)h(is)e(ex-) 164 1584 y(ecuted;)24 b(the)e(receiv)o(e)f(op)q(eration)i(of)g(the)f(second)h (pro)q(cess)g(m)o(ust)e(complete)f(b)q(efore)164 1644 y(its)f(send)h(and)g (can)g(complete)d(only)j(if)f(the)g(matc)o(hing)f(send)i(of)g(the)f(\014rst)h (pro)q(cess)g(is)164 1704 y(executed.)g(This)c(program)g(will)f(deadlo)q(c)o (k.)237 1764 y(The)i(follo)o(wing)h(program)f(is)g(unsafe,)g(and)h(ma)o(y)e (succeed)h(or)g(fail,)g(dep)q(ending)g(on)164 1824 y(implem)o(en)o(tati)o (on.)164 1993 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 2053 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 2113 y(CALL)i(MPI_SENDB\()o(sen)o (dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))241 2173 y(CALL)g(MPI_RECVB\()o(rec) o(vbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 2233 y(ELSE)101 b(!)25 b(rank.EQ.1)241 2293 y(CALL)f(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i (0,)h(tag,)f(context\))241 2354 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,) i(0,)h(tag,)f(context\))164 2414 y(END)h(IF)949 2599 y Fl(34)p eop %%Page: 35 35 bop 237 307 a Fl(The)19 b(message)f(sen)o(t)g(b)o(y)g(eac)o(h)h(pro)q(cess)g (has)g(to)g(b)q(e)g(copied)f(out)i(b)q(efore)e(the)h(send)164 367 y(op)q(eration)d(returns)g(and)g(the)f(receiv)o(e)e(op)q(eration)j (starts.)22 b(F)l(or)16 b(the)f(program)g(to)h(com-)164 428 y(plete,)i(it)h(is)g(necessary)g(that)h(at)f(least)g(one)h(of)f(the)g(t)o(w)o (o)g(messages)g(sen)o(t)g(is)g(bu\013ered)164 488 y(out)e(of)g(either)f(pro)q (cesses')h(address)g(space.)23 b(Th)o(us,)17 b(this)g(program)f(can)h (succeed)f(only)164 548 y(if)h(the)g(comm)o(uni)o(cation)e(system)g(has)j (su\016cien)o(t)e(bu\013er)i(space)f(to)h(bu\013er)f Fg(len)f Fl(w)o(ords)164 608 y(of)h(data.)237 668 y(If)d(additional)h(requiremen)o(ts) d(will)i(b)q(ecome)f(part)i(of)g(the)g(standard)h(\(e.g.,)e(b)q(ounds)164 729 y(on)23 b(the)f(minimal)d(n)o(um)o(b)q(er)i(of)i(concurren)o(tly)e(activ) o(e)h(handles)g(that)h(need)f(b)q(e)h(sup-)164 789 y(p)q(orted,)16 b(then)g(further)g(programs)h(b)q(ecome)d(safe.)949 2599 y(35)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:17:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02635; Mon, 15 Mar 93 17:17:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15024; Mon, 15 Mar 93 17:16:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:16:47 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15005; Mon, 15 Mar 93 17:16:44 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA26351; Mon, 15 Mar 93 17:16:44 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 171443.29617; Mon, 15 Mar 1993 17:14:43 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701 for ) id AA16876; Mon, 15 Mar 1993 15:15:28 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA02007; Mon, 15 Mar 93 15:15:27 -0600 Date: Mon, 15 Mar 93 15:15:27 -0600 Message-Id: <9303152115.AA02007@brisk.kai.com> To: mpi-pt2pt@CS.UTK.EDU Cc: Subject: READY Send Rationale Please. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Someone please give me a rationale for Ready Send. Doesn't Ready Send add overhead to every point-to-point communication? In a dumb implementation (i.e. one that isn't able to analyse the usage and detect that I never - ever - use Ready Send) receives will feel constrained to notify senders in all cases on the off chance that one is a Ready Send. This restricts good implementations that do not require such a dumb primitive. Further, given the non-ready state caveat "outcome is undefined" and a system where I cannot know, or choose not to know, the readiness of receives - am I able to optimize away the use of Ready Sends? If I am, as seems likely, I suggest we optimize it out of the standard. It is certainly going to complicate the formal specification and its removal further simplifies the number of calls. The only rationale I could think of (excepting that I have a dumb implementation of communication ;-) ) is memory mapped I/O devices - but a) I thought for some reason that that was not within the scope of MPI, and b) in any case, such should not be allowed to impact on the general primitives - there are better solutions. But please, if someone has a plausible rationale to keep Ready Send in this standard, let's hear it. Steven PS. We'll get to a single well defined "send" and "recv" yet! ;-) From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:39:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02984; Mon, 15 Mar 93 17:39:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15838; Mon, 15 Mar 93 17:38:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:38:48 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15830; Mon, 15 Mar 93 17:38:47 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA17275; Mon, 15 Mar 93 17:38:45 EST Received: by b125.super.org (4.1/SMI-4.1) id AA00438; Mon, 15 Mar 93 17:38:44 EST Date: Mon, 15 Mar 93 17:38:44 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303152238.AA00438@b125.super.org> To: mpi-pt2pt@CS.UTK.EDU In-Reply-To: Steven Ericsson Zenith's message of Mon, 15 Mar 93 15:15:27 -0600 <9303152115.AA02007@brisk.kai.com> Subject: READY Send Rationale Please. The Ready Send was added because it can reduce overhead in some systems. As the current draft states: "On some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required." On many message passing systems the sender must probe the receiver to make sure that space is available before the send can occur. On some system(s) this is done for each set of packets sent. With Ready Send, the sender knows that the receive is posted and does not need to send a probe message to check and this can reduce latency and increase throughput. A good algorithmic example of its use is for a global combine where the result winds up in every node. If you use a minimum spanning tree to a root node, the receives can be posted before the nodes contribution to the global value is passed on. When the root node uses the inverse tree to fan the answer back out, it can use a Ready Send without any chance of losing data. I think it has reasonable uses. One can question whether is deserves to be in the standard given the added complexity of additional choices. However, after two votes of yes for this construct, I think we should leave it unless there is a global move to simplify MPI. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04186; Mon, 15 Mar 93 17:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16444; Mon, 15 Mar 93 17:58:16 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:58:15 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16436; Mon, 15 Mar 93 17:58:09 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 15 Mar 93 14:56 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA01275; Mon, 15 Mar 93 14:54:26 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA06683; Mon, 15 Mar 93 14:54:24 PST Date: Mon, 15 Mar 93 14:54:24 PST From: rj_littlefield@pnlg.pnl.gov Subject: context creation To: mpi-pt2pt@CS.UTK.EDU, snir@watson.ibm.com Cc: d39135@sodium.pnl.gov Message-Id: <9303152254.AA06683@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@CS.UTK.EDU The new point-to-point proposal distributed on 3/15/93 seems to be generally well thought out. However, there is a simple application program design that I do not see how to implement using only the proposed context-forming routines. The same situation arose with the old group proposal, and I have posted to mpi-collcom and mpi-context a suggested fix. What follows here is a similar suggestion for the new point-to-point proposal. The application design in question uses a master-slaves strategy to asynchronously parcel out chunks of work, with each chunk being done by several processes working collaboratively. The idea is that as chunks of work finish, the processes that were assigned to them go into an idle pool, and when the pool gets big enough, some processes are pulled out and assigned to the next chunk of work. Collective communication between processes working on each chunk is required, so it seems natural to organize them into MPI contexts. However, using a synchronous context partitioning routine, such as MPI_NEW_CONTEXT, is not attractive because the varying chunk size implies that each group of processes can finish their work at different times, and synchronous partitioning would delay the assignment of new work until all currently active chunks had finished. This prompts me to suggest the addition of a less synchronous context-forming routine. This routine would be called only by those processes wishing to join the new context, rather than all processes belonging to the old context. MPI_FORM_CONTEXT (newcontext,oldcontext,nprocs,knownmembers,tag) where newcontext is output as a handle for the new context. oldcontext is a handle for the context that contains all of the participating processes. nprocs is the number of processes that will belong to the new context. knownmembers is a set of ranks, within oldcontext, of some or all members of the new context. Each member of the new context must provide the same set of knownmembers. tag is a user-provided integer tag, sufficiently unique to disambiguate overlapping contexts that might be formed simultaneously (see discussion below) In the appication design discussed above, the master would use point-to-point messages in the ALL context to inform individual processes of their assignments and collaborators, and the processes assigned to each chunk would organize themselves by calling MPI_FORM_CONTEXT. The knownmembers argument specifies a set of processes that can act as organizers of the context. In earlier discussions, some people have argued that group- or context-creation calls should not have to specify all members because that would violate scalability. If knownmembers is allowed to be less than the full set, then a tag is required to handle the case of overlapping process sets being handled simultaneously. Suppose that processes 2 and 3 each attempt to form a 2-process context with process 1, and process 1 is the only specified knownmember. (I.e., process 1 makes two calls, each specifying a 2-process context but not indicating which other process.) Then messages from 2 and 3 can be received at 1 in either order, and the tag is required to keep things straight. Personally, I would consider it acceptable to require that knownmembers specify all the members of the context, in which case there is no need for tag because things are kept straight by sequencing. Have I missed something about how to use MPI_NEW_CONTEXT? Is there a better way to implement this type of application than MPI_FORM_CONTEXT, or does MPI_FORM_CONTEXT have some drawback that I don't see? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 21:40:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07549; Mon, 15 Mar 93 21:40:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23801; Mon, 15 Mar 93 21:39:23 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 21:39:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay1.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23793; Mon, 15 Mar 93 21:39:20 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay1.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA08956; Mon, 15 Mar 93 21:39:18 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 213832.9336; Mon, 15 Mar 1993 21:38:32 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701 for ) id AA22840; Mon, 15 Mar 1993 18:28:24 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA02416; Mon, 15 Mar 93 18:28:23 -0600 Date: Mon, 15 Mar 93 18:28:23 -0600 Message-Id: <9303160028.AA02416@brisk.kai.com> To: mpi-pt2pt@CS.UTK.EDU Cc: Subject: READY Send rationale From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Ah yes, I recall this discussion now, Marc gave a good description of the spanning tree algorithm using it in Dallas last. My instinct is to complain about the implementations that benefit from it. And I don't really like our design being driven to provide efficiency in certain implementations because of a system characteristic. However, I should have come to understand this earlier perhaps. We need to say something else to guide implementors least they make the mistake I mentioned earlier. For example, we should say that Ready Send can always be replaced by Send but that Send may never be replaced by Ready Send. This gives us some greater portability and permits vendors to implement Ready Send as Send if they choose. Further, we should include the rationale that drove us to design in Ready Send. The above caveat solves my specification problem. Steven From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:22:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10313; Thu, 18 Mar 93 08:22:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03936; Thu, 18 Mar 93 08:20:28 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03919; Thu, 18 Mar 93 08:20:23 -0500 Date: Thu, 18 Mar 93 13:20:05 GMT Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: document of March 15 To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I just got back from a session of leave and customer visits, and what a lovely lot of email I find! I read the document of March 15, "Point to Point Communication", by Marc Bill and Rusty. I suggest that we delete certain sections of this document as they are not within the remit of the point-to-point subcommittee. This material is: a) Much of section 1.2 and in particular from "List of handles" to end of section 1.2.4, as it is the remit of the bindings subcommittee to make proposals to the committee on the subject of bindings. b) Much of section 1.4 and in particular from beginning to end of section 1.4.1 as it is the remit of the contexts subcommittee to make proposals to the committee on the subject of contexts. Comments? Flames?? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10347; Thu, 18 Mar 93 08:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04209; Thu, 18 Mar 93 08:26:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:26:58 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04201; Thu, 18 Mar 93 08:26:47 -0500 Date: Thu, 18 Mar 93 13:26:37 GMT Message-Id: <9150.9303181326@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: btw To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all Should have mentioned, that apart from the material referred to in the previous message, on a first skim-read the document looks generally great. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:47:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10749; Thu, 18 Mar 93 08:47:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04894; Thu, 18 Mar 93 08:45:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:45:06 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04866; Thu, 18 Mar 93 08:45:03 -0500 Message-Id: <9303181345.AA04866@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1703; Thu, 18 Mar 93 08:44:53 EST Date: Thu, 18 Mar 93 08:33:45 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Reply-To: SNIR@watson.ibm.com Subject: *************** Referenced Note *************** Received: from CS.UTK.EDU by watson.ibm.com (IBM VM SMTP V2R3) with TCP; Thu, 18 Mar 93 08:22:12 EST Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03936; Thu, 18 Mar 93 08:20:28 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03919; Thu, 18 Mar 93 08:20:23 -0500 Date: Thu, 18 Mar 93 13:20:05 GMT Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: document of March 15 To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I just got back from a session of leave and customer visits, and what a lovely lot of email I find! I read the document of March 15, "Point to Point Communication", by Marc Bill and Rusty. I suggest that we delete certain sections of this document as they are not within the remit of the point-to-point subcommittee. This material is: a) Much of section 1.2 and in particular from "List of handles" to end of section 1.2.4, as it is the remit of the bindings subcommittee to make proposals to the committee on the subject of bindings. b) Much of section 1.4 and in particular from beginning to end of section 1.4.1 as it is the remit of the contexts subcommittee to make proposals to the committee on the subject of contexts. Comments? Flames?? >>> The sections on language objects and on context need to be moved >>> to a general introduction in the final document. The section on >>> context should be construed as my proposal to the context committee >>> and should be discussed in this subcommittee. The section(s) on >>> language objects identifies which objects are used by the point to >>> point communication functions. The comments on their actual >>> embodiment in Fortran and C should be construed as my proposal to the >>> language binding subcommittee and should be discussed there. I >>> willfully trespassed in my document, in order to provide a fuller >>> view and speed up the process. >>> I shall be happy to see this text replaced by a more elaborate >>> proposal once the relevant subcommittees catch up. Best Wishes Lyndon >>> Marc /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:57:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10932; Thu, 18 Mar 93 08:57:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05304; Thu, 18 Mar 93 08:55:04 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:55:03 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05274; Thu, 18 Mar 93 08:54:58 -0500 Date: Thu, 18 Mar 93 13:54:52 GMT Message-Id: <9194.9303181354@subnode.epcc.ed.ac.uk> From: L J Clarke To: SNIR@watson.ibm.com, mpi-pt2pt@cs.utk.edu In-Reply-To: Marc Snir's message of Thu, 18 Mar 93 08:33:45 EST Reply-To: lyndon@epcc.ed.ac.uk Hi Marc Fine! Then, can we please a) annotate the document to make this clear, and b) exclude such sections from discussion in the meeting of the point-to-point subcommittee. Best Wishes Lyndon > *************** Referenced Note *************** > > Received: from CS.UTK.EDU by watson.ibm.com (IBM VM SMTP V2R3) with TCP; > Thu, 18 Mar 93 08:22:12 EST > Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) > id AA03936; Thu, 18 Mar 93 08:20:28 -0500 > X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST > Errors-To: owner-mpi-pt2pt@CS.UTK.EDU > Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) > id AA03919; Thu, 18 Mar 93 08:20:23 -0500 > Date: Thu, 18 Mar 93 13:20:05 GMT > Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> > From: L J Clarke > Subject: document of March 15 > To: mpi-pt2pt@cs.utk.edu > Reply-To: lyndon@epcc.ed.ac.uk > > Hi all > > I just got back from a session of leave and customer visits, and what a > lovely lot of email I find! > > I read the document of March 15, "Point to Point Communication", by Marc > Bill and Rusty. > > I suggest that we delete certain sections of this document as they are > not within the remit of the point-to-point subcommittee. This material > is: > > a) Much of section 1.2 and in particular from "List of handles" to end > of section 1.2.4, as it is the remit of the bindings subcommittee to > make proposals to the committee on the subject of bindings. > > b) Much of section 1.4 and in particular from beginning to end of > section 1.4.1 as it is the remit of the contexts subcommittee to make > proposals to the committee on the subject of contexts. > > Comments? Flames?? > > >>> The sections on language objects and on context need to be moved > >>> to a general introduction in the final document. The section on > >>> context should be construed as my proposal to the context committee > >>> and should be discussed in this subcommittee. The section(s) on > >>> language objects identifies which objects are used by the point to > >>> point communication functions. The comments on their actual > >>> embodiment in Fortran and C should be construed as my proposal to the > >>> language binding subcommittee and should be discussed there. I > >>> willfully trespassed in my document, in order to provide a fuller > >>> view and speed up the process. > >>> I shall be happy to see this text replaced by a more elaborate > >>> proposal once the relevant subcommittees catch up. > > Best Wishes > Lyndon > > >>> Marc > > > /--------------------------------------------------------\ > e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) > c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c > \--------------------------------------------------------/ > > > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Mar 23 22:49:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03843; Tue, 23 Mar 93 22:49:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15784; Tue, 23 Mar 93 22:48:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Mar 1993 22:48:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15774; Tue, 23 Mar 93 22:48:47 -0500 Message-Id: <9303240348.AA15774@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1885; Tue, 23 Mar 93 22:48:48 EST Date: Tue, 23 Mar 93 22:48:47 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir \\ William Gropp and Ewing Lusk} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following subsections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. The actual binding of these objects in Fortran and C will be discussed in the language binding subcommittee. } \subsection{Data Types} \subsubsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. Handles are of type {\tt void *} in C and of type {\tt integer} in Fortran. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). An opaque object is created by a call to {\tt MPI\_CREATE}, and destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to create, access and update specific opaque objects. {\bf \ \\ MPI\_CREATE(handle, type, persistence)} \begin{description} \item[OUT handle] handle to object \item[IN type] state value that identifies the type of object to be created (e.g., {\tt MPI\_COMMUNICATION, MPI\_BUFFER, MPI\_CONTEXT}, etc.). \item[IN persistence] state value; either {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}. \end{description} {\bf \ \\ MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} An object can be destroyed only if there is no pending operation that is using this object; after successful return of the routine, the handle is undefined. {\bf \ \\ MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \paragraph*{List of handles} An MPI call may need a parameter that is a {\em list of handles}. In C, such list will be a record with one component being the length of the list, the other components being an array of pointers. In Fortran, the list will be an array of integers, the first one of which is the length of the list. \discuss{ The mechanism for opaque objects used here follows the POSIX Fortran binding standard. An alternative choice is to have different type declarations for each type of opaque object. Then, opaque objects are created/destroyed like regular variables, rather than by MPI calls; they are still accessed and updated only via MPI functions. Some people proposed that {\tt list\_of\_handles} would not contain the list length, and the list length be passed as an additional, separate parameter. There are really three choices: \begin{enumerate} \item {\tt list\_of\_handles} contains a list length entry \item {\tt list\_of\_handles} is terminated with a special {\tt end-of-list} entry. \item {\tt list\_of\_handles} is not self-delimiting; and additional length parameter is needed whenever a list of handles is used. \end{enumerate} Comments? } \subsubsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_CREATE} routine has a state type parameter with values {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}. An {\tt enumeration} declared in an included MPI.h file will be used in C for state datatypes. The Fortran 77 mechanism needs to be decided. \discuss{ Named integer constants can be used in Fortran 90, using the {\tt PARAMETER} mechanism. The constant declarations can be made available via an {\tt INCLUDE} file. Fortran 77 does not seem to offer any convenient mechanism. One possibility is to specify explicit integer values, and allow the use of named constants with those Fortran 77 compilers that support them conveniently. Another possibility is to use character strings, rather than integers. } \subsubsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). The special values are provided by named constant, that are made available via an MPI.h include file in a C binding. \discuss{ Need to agree on a Fortran mechanism for named constants (see the discussion above). Implementers should detect, whenever possible, illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent will be flagged as an error. } \subsubsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. In C, a formal parameter of type {\tt void *} will be used, with an actual pointer parameter. in Fortran, we shall cheat. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \subsection{Processes} An MPI program is executed by several autonomous processes that execute each their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow to specify the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution (total number of processes is fixed), although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. Absolute, system-wide unique process id's are (will be) needed only if dynamic process creation is to be supported (in such eventuality we propose to use handles to opaque {\em process structures} for that purpose). \subsection{Contexts} \discuss{ This section contains a proposal for use of contexts that will subsume groups. It borrows heavily on the current group proposal. This proposal has will be discussed in the ``context'' subcommittee.} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \paragraph*{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \paragraph*{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \paragraph*{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \paragraph*{Lightweight gang scheduling} Consider an environment where processes are multithreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \subsubsection{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. However, each context creation operation is likely to be a blocking collective operation that requires barrier synchronization. It is important to guarantee that no message is sent to a process in a new context before that process has joined the new context. Thus, a call to a context creation function will not return until all processes in the new context have called the function. (Of course, one can imagine ingenious methods to relax this requirement.) } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \paragraph*{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. (In a MIMD environment, ``harness program'' may be replaced by ``initial coordination code''.) Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ A possible additional context creation function is {\bf MPI\_CREATE(newcontext, oldcontext, list\_of\_ranks)}, which picks an explicit list of members for a new context (identified by their rank in a previous context) and create that new context. This function must be called by all members of the list, and all must supply the same list. The processes are ranked in the new context according to their rank in the list. Dynamic process deletion can be handled by creating first a new context that does not include processes to be deleted, next having those processes terminate. Dynamic process creation can be handled by allowing {\em group extension}: A possible mechanism is a function of the form {\bf MPI\_EXTEND\_CONTEXT(context, number)} which adds {\tt number} processes to {\tt context}, with the new processes ranked above the old ones. Additional functionality is required for process creation/deletion in a heterogeneous environment, in order to control where new processes are allocated. None of this requires explicit, absolute, process names. However, if process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } \subsubsection{Error Handling} MPI provides the user with reliable message transmission: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. Where the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementer of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as global program failures. Similarly MPI itself provides no mechanisms for handling node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to store an error number in a global variable ({\em errno} in a C environment) and generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. One might want different signals for different modules. The details of such proposal need be elaborated in an appropriate ``profile'' subcommittee. } \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may mix values of different types. \discuss{ May also need {\tt DOUBLE COMPLEX} in Fortran.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the specified context. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. {\tt Context} should be a context shared by both source and destination. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. A buffer consists of a sequence {\bf buffer components}. Each component consists of a sequence variables of the same basic type. There are three basic types of buffer components: \begin{description} \item[block] A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[datatype] Type of elements \end{description} \item[vector] A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk \le stride$) \item[datatype] Type of elements \end{description} Note that a constant stride becomes contiguous when {\tt stride = lenblk}. A vector buffer component can be an arbitrary submatrix of a two-dimensional matrix. \item[indexed] A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[list\_of\_indices] List of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} \end{description} For example, if a Fortran array is declared as {\tt double precision a(10)}, then the tuple {\tt $<$(a(3), (0,3,6,2), DOUBLE$>$} specifies a buffer component with entries {\tt a(3) a(6), a(9), a(5)}. \discuss{ Do we allow negative displacements? Do we allow entries to be repeated in an indexed buffer component? Do we allow different buffer components to overlap? Do we require in an vector buffer component that {\tt len} be a multiple of {\tt lenblk}? } A buffer is described by an opaque object accessed via a {\bf buffer handle}. Such object is created and destroyed via calls to {\tt MPI\_CREATE} and {\tt MPI\_FREE}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_ADD\_BLOCK}, {\tt MPI\_ADD\_VECTOR} or {\tt MPI\_ADD\_INDEX}, in order to append a component to the buffer associated with a buffer handle. A buffer object can be destroyed only if there is no pending communication operation using it. After a buffer object is destroyed the associated buffer handle is undefined. {\bf \ \\ MPI\_ADD\_BLOCK( buffer, start, len, datatype)} \\ Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_VEC( buffer, len, stride, lenblk, datatype )} \\ Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_INDEX( buffer, start, list\_of\_indices, datatype)} \\ Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[start] initial position for indexing (choice) \item[list\_of\_indices] list of relative indices of entries (array of integers) \item[IN datatype] datatype identifier (status) \end{description} Consider, for example, the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, B, 1, MPI_INT) CALL MPI_ADD_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_ADD_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_ADD\_xxx} call). Such handle is associated with an empty buffer, and a message created from it contains no data. \discuss{ The main modifications w.r.t. the proposal of Gropp and Lusk is measuring length in elements, rather than bytes. Seems more natural, since type is known. Also, object creation uses a generic function, rather than a function specific to buffer descriptors. As result, I gave up on the size parameter. This may not be such a loss: it is not clear that specifying a maximum length at buffer object creation is useful, since indices of arbitrary size may need to be stored in the object. } \subsubsection{Data Conversion} The types and the locations of the entries used to create a message is solely determined from the information in the buffer descriptor, using the storage association rules specified by the host language and its implementation; the type and the locations of these entries do not depend on the declaration for the corresponding variables in the calling program. It is not required that the data types specified in a buffer descriptor match the data types of the corresponding elements in the host program. However, in case of mismatches, the correspondence between entries in the host program and entries in a message created with the buffer descriptor may be implementation dependent. No data conversion occurs when data is moved from a sender buffer into a message. Consider the following fragment of Fortran code \begin{verbatim} REAL A(100) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, A(1), 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, A(2), 1, MPI_INT) CALL MPI_ADD_BLOCK (BH, A(3), 1, MPI_LOGICAL) CALL MPI_ADD_BLOCK (BH, A(4), 1, MPI_COMPLEX) CALL MPI_ADD_BLOCK (BH, A(6), 1, MPI_DOUBLE) CALL MPI_ADD_BLOCK(BH, A(8), 4, MPI_CHAR) \end{verbatim} A message created from this buffer will consist of a sequence containing one real, one integer, one logical, one complex, one double, and four characters. No data conversion occurs when values are copied from the sender buffer to the message. Thus, the first entry in the message is a real number equal to {\tt A(1)}; the second entry is an integer number that happens to have the same binary representation as {\tt A(2)}; the third entry is a logical value that happens to have the same binary representation as {\tt A(3)}; the fourth entry in a complex number with a binary representation identical to {\tt A(4), A(5)}; the fifth entry is a double precision value with a binary representation identical to {\tt A(6), A(7)}; and if words have four bytes then the last four entries are bytes that make the binary representation of {\tt A(8)}, in the byte order of the executing machine. The correspondence between the first seven entries of the array {\tt A} and the first five entries of the message created from this buffer is determined by the rules of Fortran 77 on storage association: Each variable of type {\tt INTEGER}, {\tt REAL}, or {\tt LOGICAL} occupy one {\em numeric storage unit}; a variable of type {\tt DOUBLE} or {\tt COMPLEX} occupy two numeric storage units. Thus, the same correspondence will hold for any implementation. However, different implementations may have different binary encodings of integer, real and logical values, so that the actual values transferred by the message may differ. The correspondence between the entry {\tt A(8)} of the array, and the last four character entries in the message is implementation dependent, since the Fortran language does not specify a correspondence between character storage units and numeric storage units (an array of characters cannot be equivalenced with an array of integers). Different results may occur in big-endians or small-endians machines, or in 32 bit or 64 bit machines. The same holds, symmetrically, when a message is received. Entries are moved from the message into the receiver memory according to the information provided by the buffer descriptor, with no regard to the way the corresponding variables are declared in the receiving program. When data is moved in a homogeneous environment between nodes having the same architecture, then no data conversion occur at any point during data transfer. Assume, in the previous example, that an identically declared buffer descriptor is used to receive data into an identically declared array at the receiving process. Then, when these two nodes communicate, the values in the first eight entries of array {\tt A} of the sender will be copied into the first eight entries of array {\tt A} at the receiver (assuming that reals occupy the same storage as four bytes). In particular, in a homogeneous environment, it is possible to communicate using only character typed messages. When data is moved into a heterogeneous environment between nodes having distinct architectures, data conversion may occur during the transfer: Each entry is converted from the data representation used on the sending node to the data representation used in the receiving node. Consider the following fragment of Fortran code. \begin{verbatim} REAL X, Y CHARACTER (4) Z INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, X, 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, Y, 4, MPI_CHAR) CALL MPI_ADD_BLOCK (BH, Z, 4, MPI_CHAR) \end{verbatim} Assume that the same arrays {\tt X, Y, Z} and buffer handle {\tt BH} are created at two processes A and B; this handle is used by A to create and send a message for B, by B to receive this message. Further assume that both A and B run on distinct nodes, with possibly different 32 bit architectures. Then {\tt X} at process B is assigned the value of {\tt X} at process A (up to rounding errors that may occur during conversion); {\tt Z} on process B is assigned the character string value of {\tt Z} on process A; if both nodes use ASCII encoding, then no conversion is required for the characters. On the other hand, variable {\tt Y} on process B may be allocated a value that differs from the value of variable {\tt Y} on process A. This may occur if the two nodes use a different byte sequence (little-endian vs big-endian), or use a different binary representation for reals (IEEE vs HEX). Thus, in order to ensure correct execution in a heterogeneous environment, it is important that the types of values in a message match the types of the corresponding values in the sending and in the receiving program. \subsection{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \subsection{Communication Mode} A sending operation can occur in one of two modes: \begin{description} \item[REGULAR] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \end{description} A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required. \discuss{I deleted the symmetric ready receive. Will revive it if there is a requirement for it.} \subsection{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such object persists only until the communication operation is completed, at which point it is destroyed. Thus, correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Communication Object Creation} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. {\bf \ \\ MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} \\ Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] handle to context of messages sent with this handle \item[IN mode] send mode (state type, with two values: {\tt MPI\_REGULAR} and {\tt MPI\_READY}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} {\bf \ \\ MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, persistence)} \\ Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or DONTCARE (integer). \item[IN tag] user tag for messages received with this handle, or DONTCARE (integer). \item[IN context] handle to context of messages received with this handle. \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. } \subsubsection{Communication Start} {\bf \ \\ MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf \ \\ MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. {\bf \ \\ MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete, In such case, the return handle points to an opaque object that contains information on the completed information. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. Implementation notes: A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. {\bf \ \\ MPI\_RETURN\_STAT( handle, len, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT len] difference between length of receive buffer and length of received message, in bytes. Thus, the value returned is zero if the received message matches the the receive buffer, positive if it is shorter (integer). \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \discuss{ I put the difference between message buffer and message length as the value returned, rather than length of received message, so that it might be easy to test for exact match. ``Accepted practice'' seems to be a byte count. The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming.} \subsubsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. {\bf \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_handle)} \\ Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} is equivalent to the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, see Section~\ref{subsec:correct}). {\tt \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_status\_handle)} \\ is \begin{verbatim} {MPI_WAIT (handle[1], return_handle); index = 0} || ... || {MPI_WAIT (handle[n], return_handle); index = n-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) {\bf \ \\ MPI\_STATUSANY ( list\_of\_handles, index, return\_handle)} \\ Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the later case, it returns a value of -1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} {\bf MPI\_WAITALL(list\_of\_handles, list\_of\_return\_handles)} \\ Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \subsection{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void or {\bf R}) indicates the start mode (regular or ready). {\bf \ \\ MPI\_SEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RSEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} {\bf Implementation note:} While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} indicates the {\tt READY} mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. {\bf \ \\ MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) \end{verbatim} \subsection{Block Sending Operations} The most frequent type of buffer used is a contiguous buffer of numeric storage units, i.e., a contiguous buffer of words that may contain either INTEGER, REAL or LOGICAL values (in FORTRAN). In a homogeneous environment such messages can be used to send arbitrary sequences of contiguous items, where each item occupies an integer number of words. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf B} in the function name, for {\tt BLOCK}. {\bf \ \\ MPI\_SENDB (start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_SEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_RSENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RSEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf MPI\_RECVB(start, len, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RECV(buffer_handle, source, tag, context, return_status_handle) \end{verbatim} {\bf \ \\ MPI\_ISENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_ISEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRSENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRSEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRECVB(handle, start, len, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRECV(handle, buffer_handle, source, tag, context) \end{verbatim} \discuss{ I use word count, rather than byte count. I believe word messages are much more prevalent than byte messages, and it's a blessing not to have to multiply by 4 for each message. Byte messages are still available the hard way. Also, word messages fit well with the Fortran numeric storage unit concept. If we are to add more functions, my next addition would be a vector send operation. } \subsection{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation or, in the case of a send, because the matching receive successfully receives another message. {\bf An enabled operation either completes successfully or becomes permanently disabled.} I.e., an enabled operation either eventually succeeds, or becomes disabled (progress); and an operation cannot be enabled infinitely many times without ever succeeding (fairness). \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVB(recvbuf, len, 1, tag, context) CALL MPI_SENDB(sendbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDB(sendbuf, len, 0, tag, context) CALL MPI_RECVB(recvbuf, len, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt len} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Tue Mar 23 22:54:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03906; Tue, 23 Mar 93 22:54:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15939; Tue, 23 Mar 93 22:53:30 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Mar 1993 22:53:29 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15931; Tue, 23 Mar 93 22:53:28 -0500 Message-Id: <9303240353.AA15931@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1909; Tue, 23 Mar 93 22:53:29 EST Date: Tue, 23 Mar 93 22:49:11 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: new draft Reply-To: SNIR@watson.ibm.com Has few changes w.r.t. previous draft, mostly typos and clarifications, or alternative proposals acknowledged in dicussion parts. Thanks to James Cownie, Jon Flower, Rik Littlefield, Eric Barszcz, Tom Henderson, and whoever else I forgot, for corrections and suggestions. From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 25 12:41:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21419; Thu, 25 Mar 93 12:41:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27041; Thu, 25 Mar 93 12:41:13 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 25 Mar 1993 12:41:12 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27031; Thu, 25 Mar 93 12:41:07 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA02886 (5.65c/IDA-1.4.4); Thu, 25 Mar 1993 12:41:03 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA18733; Thu, 25 Mar 93 17:40:59 GMT Date: Thu, 25 Mar 93 17:40:59 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303251740.AA18733@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA07372; Thu, 25 Mar 93 17:37:30 GMT To: snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu Cc: mpi-context@cs.utk.edu Subject: PT2PT draft (MARCH 23) Content-Length: 5131 Marc, [This is a reconstruction from memory of some mail I sent this morning, which seems to have disappeared... If anyone got the original they can test how good my memory is by comparing the two mails :-)] Here are some comments and queries on the current (23 March) point to point proposal. To avoid duplication of lots of text I will reference the proposal bye page number. Apologies to those of you who haven't printed it ! Page 3 : Discussion of options for Lists of Handles My preference is for option 3 (A separate length parameter). I prefer this over 1 (length as first element) because this leads to off by one errors, or a funny structure in C with an arbitrary sized array. I prefer it over 2 (a delimited list) because it avoids a whole pass over the list to work out its length when this is required. (e.g. if a copy needs to be made. cf the horrible C code char * copy = strcpy(malloc(strlen(string)+1), string); which accesses every byte of the string twice) Page 4: Discussion of named constants in FORTRAN. The F90 discussion should mention MODULEs (e.g. "... can be made available via an INCLUDE file, or MODULE.") PLEASE PLEASE PLEASE don't use character strings. This is guaranteed to be slow. I'd rather have literal values in place than strings ! Page 8 et seq: Contexts. I don't understand how to implement your context model in the way you seem to be describing. On page 8 you suggest that you can check the context at transmission, and avoid a collective agreement alogrothm at context creation. Similarly on page 10 in the implementation note you say "No communication is needed to create a new context beyond a barrier synchronisation". Surely you MUST check the context at the receiver (you seemed to agree with this in previous mail in MPI-context), therefore the sender and receiver must agree on the context name. (Or at least the sender must know the value by which the context is known at the receiver, which could be different for each receiver if you like [I don't !]) To achive this surely you need a group co-operation on context creation. Consider Process number i.e. Rank in ALL 0 1 Group 0 0 1 Group 1 1 0 The number in the table is the rank of the process in the group. Certainly when process 1 receives a message from zero he must know the group/context from which it was sent, since the result of the sender inquiry will be different depending on the group/context. Even if you only allow partitioning, I don't think you can avoid needing a cooperation, consider a scenario like this Process number Time 0 0 1 2 Time 1 0 | 1 2 Partition of all Time 2 1 | 2 Partition of subgroup Time 3 0 1 | 2 Partition of all Where the | denotes a partition of the group. Now with only a sequnce number process 0 has has been involved in 2 partitions, but process 1 (which should be in the same group) has been involved in 3. Ah, you can get the right answer by using a count of the partitions of the the parent group, and its depth in the tree. (Somehow...) Anyway it all falls apart when you let in the group construction by list (I believe) ! Page 12: I agree that we shouldn't REQUIRE dynamic process creation. Page 13: A global errno is the CLASSIC example of global data which gives threads a problem. Do we REALLY want to do it like this ? Page 14: Data types I think we should have both MPI_CHAR and MPI_BYTE. The difference being that in a heterogeneous environment MPI_CHAR would be translated according to the local character sets (e.g. ASCII -> EBCDIC), MPI_BYTE would be an 8 bit integer. What should we do about Fortran 90 KINDs ? Page 15: Data buffers. Vector buffer : Do we allow a zero or negative inter block stride ? Zero is a neat way to perform a fill operation... Page 17: Last line on the page... What is the (4,2,1) ? It's not Fortran 77. Looks a bit like an F90 array constructor ... Page 18: Data conversion You state "No data conversion occurs when data is moved from a sender buffer into a message". In a heterogeneous environment surely this is an implementation issue. The implementation should be free to translate at source, at dest, or on the moon if it likes. Page 36: You should delete "out of either process' address space." The example only requires buffering, it doesn't matter where the implementation chooses to do it. General question ================ Where is the store which is referenced by handles allocated ? At present it appears that all of this (even for ephemeral objects) is MPI system managed. I would very much like it to be possible to have the user manage this store, as she can often do it cheaper. (e.g. allocating ephemeral objects on the stack). Hmmm... came out more or less the same I think. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 10:09:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18013; Tue, 6 Apr 93 10:09:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22609; Tue, 6 Apr 93 10:07:28 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 10:07:26 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22587; Tue, 6 Apr 93 10:07:23 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA27788; Tue, 6 Apr 93 10:07:20 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA05994; Tue, 6 Apr 93 10:07:19 EDT Date: Tue, 6 Apr 93 10:07:19 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304061407.AA05994@b125.super.org> To: mpi-pt2pt@cs.utk.edu Subject: ready receive We have now met three times in Dallas and each time we have had a vote on ready receive. The votes have been: Jan: 13 Y 10 N Feb: 20 Y 12 N Apr: 10 Y 10 N Given these votes, I assume that ready receive remains in the draft by default. However, since the discussion was limited at the last meeting, I thought it would be useful to outline the arguments on this topic. That way, if we need to vote again, we can hopefully do it quickly and with the pros/cons at hand. Ready receive is a special type of send that tells the processor that an appropriate receive has already been posted for this message. On some systems this allows the sender to transmit the message without having to send system request/acknowledge packets (or significantly reduces the number). This effectively raises the bandwidth that the user achieves on the system. For example, on the Intel Delta ready receive (forced type) can increase the bandwidth by over 20%. An example of the use of ready receive is for a fan in/fan out collection/distribution of information. A standard algorithm would use a tree structure to recursively collect the information at the root node. Then the inverse tree is used to distribute the information back out to the other nodes in the tree. The modification with ready receive is to post an asynchronous receive for the fan out step at each node before the node sends the data to the next node during the fan in steps. Thus, when the fan out occurs, the sender knows that a receive is already posted and can use ready receive to decrease the overhead associated with this step. Some arguments for ready receive: - it reduces system overhead for message passing on some systems - variants of it are used on current generation systems (Intel NX) Some arguments against ready receive: - when improperly used a message can be lost. worse yet, this may only happen on a larger problem where synchronization is worse. this leads to users filling bug reports with the vendor. - it is yet another variant of send and adds more calls to MPI Now, IMHO: Lots of things in MPI will fail if used improperly by the user. Ready receive is not unique in this fashion. We left in asynchronous calls for several reasons but if the user does not properly issue a wait then he/she can get the wrong answer. I don't think we should penalize a knowledgeable user because some people may do the wrong thing. As for it being yet another call, I do have sympathize here. However, ready receive can simply do a standard send and that would comply with the MPI standard. Thus, on systems that don't care, the extra work is minimal. This is the same as the extra call for every regular call to get profiling. On the other hand, it can be a big win on some systems and there the implementor can spend the time to do it right. For these reasons I support ready receive (with a new name :-). I hope this summarizes the arguments but fear I have left something out. I welcome additions and comments. Steve From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 11:05:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19536; Tue, 6 Apr 93 11:05:10 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25029; Tue, 6 Apr 93 11:04:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 11:04:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25021; Tue, 6 Apr 93 11:04:06 -0400 Received: by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA18478; Tue, 6 Apr 93 08:04:05 PDT Date: Tue, 6 Apr 93 08:04:05 PDT From: nessett@ocfmail.ocf.llnl.gov (Danny Nessett) Message-Id: <9304061504.AA18478@ocfmail.ocf.llnl.gov> To: mpi-pt2pt@cs.utk.edu Subject: timeouts - a proposal The Problem ----------- There are a number of situations in which it is desirable/necessary to timeout communications supported by MPI. For example, if a process wishes to test for non-MPI I/O completion, such as non-blocking file I/O, keyboard/mouse interrupts or other asynchronous events, it cannot afford to block inside an MPI routine forever. Another example is when a long job (say an atmospheric code that runs for weeks), is submitted via batch and encounters either a hardware or software error. There will be no one who will be monitoring the progress of this job to terminate it. If there is no programming facility within MPI to detect such errors, it is conceivable that such a job could continue "running" consuming memory and perhaps CPU resources for many hours before corrective action is taken. In regards to the last problem, several people I have talked to have commented that it is not the job of MPI to detect and recover from either hardware errors or erroneous programs. I believe this is acceptable if we are designing a prototype or an academic toy, but in real production environments, such an approach will lead to much less acceptance of MPI as a standard programming interface. The Issues As I See Them ------------------------ There seems to be several issues in regards to timeout that are important. If I have missed some, I encourage others to point them out. The issues I see are: o People are reluctant to add a timeout parameter to every MPI call that could possibly block. Not only does it complicate the interface, it also requires those programmers who are not interested in timeout to think about it. o People want MPI to conform to standard practice. Current practice in the homogeneous MPP (NX, CMMD, etc) and workstation cluster world (PVM, P4, etc) is not to provide a timeout or to time out communications in the supporting library and crash. Current practice in the networking world (say sockets) is to provide for a timeout in the programming library. Consequently, there is a collision of approaches in current practice. o Some MPP manufacturers may find it difficult to provide timeouts. Either there is no hardware support in the machine's message passing mechanism or it is not easily used to provide the necessary high-performance message passing semantics. o Timeouts require the programmer to catch timeout errors in if statements or other conditional statements and provide the appropriate exception handling. Scientific programmers have a habit of not coding for errors. o Timeouts move the responsibility for failure atomicity to the application programmer. The underlying MPI library implementation cannot simply crash all of the parallel executables when a timeout occurs (at least in all cases). A Proposal ---------- Given the above issues, consider the following proposed change to the MPI interface to support timeouts. To each of the functions MPI_WAIT, MPI_WAITANY, and MPI_WAITALL add a timeout argument. This argument should be a double value representing the number of seconds to wait before timing out the WAIT call. If the conditions for the WAIT to return are not satisfied before the timeout occurs, the function returns a return value indicating timeout. In detail the following changes are proposed : MPI_WAIT( handle, return_status_handle, timeout) IN handle communication handle OUT return_handle handle that is associated with return status object. IN timeout timeout value (double) MPI_WAITANY( list_of_handles, index, return_handle, timeout) IN list_of_handles list of handles to communication objects OUT index index of handle for for operation that completed (integer) OUT return_handle handle that is associated with return status object. Set to return status of operation that completed. IN timeout timeout value (double) MPI_WAITALL( list_of_handles, list_of_return_handles, timeout) IN list_of_handles list of handles to communication objects OUT list_of_return_handles Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. IN timeout timeout value (double) MPI_WAIT, MPI_WAITANY and MPI_WAITALL return a status value as their return value. Thus, in order to use them properly, the programmer must check their return value in a conditional statement. A suitable value should represent a TIMEOUT and another value SUCCESS. Since supplying a timeout value of 0 transforms these wait functions into the equivalent of the MPI functions MPI_STATUS, MPI_STATUS_ANY and MPI_STATUS_ALL (NB: I didn't find the last function in the current draft, but I think it was added at the last meeting), these function calls can be eliminated from the MPI standard. Specifying a timeout value of -1 indicates a "wait forever" request (i.e., do not timeout the calls). Analysis of the proposal ------------------------ An examination of the proposal according to the issues described above follows: o Only three MPI calls take a timeout parameter. Consequently, the MPI interface may be used without knowledge of or concern with user supplied timeouts. o Use of timeouts in the WAIT family of function calls follows the specification of a timeout value in the UNIX I/O "select" call. Current practice for networked message passing is followed in this family, while current practice for MPP and cluster library message passing is followed in the other MPI calls. o Provision of a timeout in the WAIT family of calls for MPPs may or may not be problematic. Specific vendor advice is required to clarify this issue. o Scientific programmers not using the WAIT family of MPI calls (probably most) need not check return values. Use of the WAIT family is considered to be "advanced MPI programming" and so should only be used by "experts." o Failure atomicity can still be provided by the underlying MPI implementation as long as the programmer only uses blocking MPI calls. Use of non-blocking I/O is advanced use of MPI and so requires the programmer to exercise more skill in developing programs. From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 11:30:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19957; Tue, 6 Apr 93 11:30:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26307; Tue, 6 Apr 93 11:29:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 11:29:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26299; Tue, 6 Apr 93 11:29:28 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.16) id AA11026; Tue, 6 Apr 93 10:29:21 CDT Received: by teak18.cray.com id AA10963; 4.1/CRI-5.6; Tue, 6 Apr 93 10:29:20 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304061529.AA10963@teak18.cray.com> Subject: Re: ready receive To: mpi-pt2pt@cs.utk.edu Date: Tue, 6 Apr 93 10:29:16 CDT In-Reply-To: <9304061407.AA05994@b125.super.org>; from "Steve Huss-Lederman" at Apr 6, 93 10:07 am X-Mailer: ELM [version 2.3 PL11b-CRI] Steve Huss-Lederman writes: > > Ready receive is a special type of send that tells the processor that > an appropriate receive has already been posted for this message. On > some systems this allows the sender to transmit the message without > having to send system request/acknowledge packets (or significantly > reduces the number). This effectively raises the bandwidth that the > user achieves on the system. For example, on the Intel Delta ready > receive (forced type) can increase the bandwidth by over 20%. I think this is a good description of ready receive. It matches the description given in each meeting. One thing that bothers me, and that is behind the reason I vote *against* the feature, is that while the proponents say: "some systems [can benefit]" they always mention the same example -- Intel systems. No other vendor's MPP or cluster system ever gets mentioned, nor any of the workstation packages. Will ready receive be of benefit to other current or soon-to-be systems or packages? I'll start. I do not anticipate that ready receive will be of benefit to the CRAY T3D (MPP system). As we all know, defining a standard requires balancing several opposing pressures. I can live with MPI including a familiar, commonly-used performance feature even if our architecture may not be able to use it optimally. But I dislike the idea of standardizing an uncommon feature that is only beneficial to one kind of system. (Especially a competitor! ;-) Thanks. - Peter Rigsbee Cray Research, Inc. par@cray.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 12:08:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21123; Tue, 6 Apr 93 12:08:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28443; Tue, 6 Apr 93 12:07:05 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 12:07:01 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28429; Tue, 6 Apr 93 12:06:58 -0400 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA18922 (5.65c/IDA-1.4.4 for ); Tue, 6 Apr 1993 11:06:56 -0500 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05217; Tue, 6 Apr 93 11:06:55 CDT Date: Tue, 6 Apr 93 11:06:55 CDT Message-Id: <9304061606.AA05217@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu In-Reply-To: Peter Rigsbee's message of Tue, 6 Apr 93 10:29:16 CDT <9304061529.AA10963@teak18.cray.com> Subject: ready receive ... I'll start. I do not anticipate that ready receive will be of benefit to the CRAY T3D (MPP system). Why? I've heard arguments that other systems operate in what I'll call receiver-probably-ready which may (or may not) reduce the need for receiver-ready, but I have not heard anyone offer EVIDENCE that receiver-ready will not help. The only evidence that we have is that the one vendor that has implemented it does see a significant performance improvement. Because of the strong assertion that receiver-ready makes about the communication, there is a some reason to believe that any system could take advantage of the additional semantic information, though it need not do so. Does anyone care to provide some evidence that ready-receive can not provide additional performance on other systems? Bill From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 13:10:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21974; Tue, 6 Apr 93 13:10:09 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01876; Tue, 6 Apr 93 13:09:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 13:09:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01868; Tue, 6 Apr 93 13:09:05 -0400 Date: Tue, 6 Apr 93 18:08:47 BST Message-Id: <723.9304061708@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: ready receive To: par@teak.cray.com (Peter Rigsbee), mpi-pt2pt@cs.utk.edu In-Reply-To: Peter Rigsbee's message of Tue, 6 Apr 93 10:29:16 CDT Reply-To: lyndon@epcc.ed.ac.uk Peter Rigsbee writes: > Steve Huss-Lederman writes: > > > > Ready receive is a special type of send that tells the processor that > > an appropriate receive has already been posted for this message. On > > some systems this allows the sender to transmit the message without > > having to send system request/acknowledge packets (or significantly > > reduces the number). This effectively raises the bandwidth that the > > user achieves on the system. For example, on the Intel Delta ready > > receive (forced type) can increase the bandwidth by over 20%. > > I think this is a good description of ready receive. It matches the > description given in each meeting. > > One thing that bothers me, and that is behind the reason I vote *against* > the feature, is that while the proponents say: > "some systems [can benefit]" > they always mention the same example -- Intel systems. No other vendor's > MPP or cluster system ever gets mentioned, nor any of the workstation > packages. I agree with this. I want to make a strong proposal about the argument that a 20% improvement on the Delta is attainable, but more precisely about the Delta in relation to MPI. There is one Delta. There will never be another (conjecture, but true). We should place the Delta outside the set of machines which MPI does consider. It is of no future importance and is of trivial importance in the present. This means that arguments such as "You can't do XXX on the Delta" are not acceptable as arguments for excluding XXX from MPI, and that arguments such as "XXX goes ? faster than YYY on the Delta" is not acceptable as an argument for including XXX in MPI. > I'll start. I do not anticipate that ready receive will be of benefit to > the CRAY T3D (MPP system). I encourage vendors to add their statements. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 13:31:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22354; Tue, 6 Apr 93 13:31:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02816; Tue, 6 Apr 93 13:29:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 13:29:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02729; Tue, 6 Apr 93 13:28:38 -0400 Date: Tue, 6 Apr 93 18:28:33 BST Message-Id: <754.9304061728@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: interrupt driven To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Dear MPI Colleagues We have, reasonably in my opinion, decided not to provide interrupt driven receive/send (as in Intel hrecv/hsend) in MPI, or active messages. I have heard, can't recall from whom, that this carries the consequence that we should not define any feature of MPI which asks for the implementor of MPI to make use of interrupt driven receives, or active messages. This seems to me to be a bogus argument. In my mind the primary reason for not providing hrecv/hsend type facilities is that it is very difficult to standardise these facilities across different operating systems, particularly in terms of what the handler procedure is and is not allowed to do (e.g., can it do I/O, can it use MPI, ...). It seems to me clear that on many (all) machines of interest the system itself will be making use of similar facilities. Who would write MPI for CM-5 without using active messages? It seems to me that for example non blocking communications really do ask for use of interrupt/active messages. Is there agreement, disagreement, or what on this point? Please do let me know. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:01:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25987; Tue, 6 Apr 93 16:01:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11304; Tue, 6 Apr 93 15:59:27 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 15:59:26 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11296; Tue, 6 Apr 93 15:59:24 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA19208 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 6 Apr 1993 12:59:22 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA09481; Tue, 6 Apr 93 12:50:01 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA01378; Tue, 6 Apr 93 12:50:00 PDT Date: Tue, 6 Apr 93 12:50:00 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9304061950.AA01378@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Timeouts. I like the idea of a timeout too. I also spoke to a handful of people during the last meeting and got the distinct impression that this would be a useful addition. I also like the idea of collapsing the STATUS and WAIT calls together by using the timeout value. Not only does this reduce the number of routines but it also eliminates the need to remember which is which! Jon From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:07:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26079; Tue, 6 Apr 93 16:07:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11792; Tue, 6 Apr 93 16:07:20 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 16:07:18 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11784; Tue, 6 Apr 93 16:07:17 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA19865 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 6 Apr 1993 13:07:15 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA09487; Tue, 6 Apr 93 12:57:54 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA01393; Tue, 6 Apr 93 12:57:53 PDT Date: Tue, 6 Apr 93 12:57:53 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9304061957.AA01393@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Ready receive. While I tend to agree that implementing ready-receive might well improve the performance of many vendors systems I would like to make the following observations which relate to my continued voting against this idea: a) I don't believe that vendors will actually go to the trouble of implementing this. All the discussions we've had in Dallas keep coming back to the "feature" of allowing ready-receive which is that it can actually be a no-op. To my mind this implies that vendors will take this approach wherever possible. (As suggested by Peter -- thank you Peter!!) b) If the best that can be expected from this is a 20% improvement in communication speed I (personally) don't care very much. I don't think that such an improvement is worth fighting for, especially since it only means a tiny overall improvement in an "efficient" program. Of these two I think point a) is where I have the biggest problem. Why add a feature to MPI that those outside our meetings might expect to be implemented, when we know, in our hearts, that it probably isn't going to be? At best it's misleading and at worst someone will go out of their way to use it (by re-ordering their communication logic, for example) only to find that it doesn't help!. I think this is the classic case of an "optional" feature that should be pushed to the back of the manual and left in peace. Jon From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:38:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26562; Tue, 6 Apr 93 16:38:49 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13254; Tue, 6 Apr 93 16:38:15 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 16:38:14 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13246; Tue, 6 Apr 93 16:38:11 -0400 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04157; Tue, 6 Apr 93 20:38:07 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA10433; Tue, 6 Apr 93 14:36:46 MDT Date: Tue, 6 Apr 93 14:36:46 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304062036.AA10433@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Timeouts. > I like the idea of a timeout too. I also spoke to a handful of people > during the last meeting and got the distinct impression that this > would be a useful addition. > > I also like the idea of collapsing the STATUS and WAIT calls together > by using the timeout value. Not only does this reduce the number of > routines but it also eliminates the need to remember which is which! > > Jon I agree. At last, a proposal that REDUCES the number of routines! Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 21:26:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29246; Tue, 6 Apr 93 21:26:02 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26159; Tue, 6 Apr 93 21:24:11 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 21:24:10 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26151; Tue, 6 Apr 93 21:24:07 -0400 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA12606; Tue, 6 Apr 93 18:23:57 PDT Message-Id: <9304070123.AA12606@SSD.intel.com> To: lederman@b125.super.org (Steve Huss-Lederman) Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: ready receive In-Reply-To: Your message of "Tue, 06 Apr 93 10:07:19 EDT." <9304061407.AA05994@b125.super.org> Date: Tue, 06 Apr 93 18:23:57 -0700 From: prp@SSD.intel.com First a little history of "force types", which is our form of ready receive. It was first implemented on the iPSC/2 system to solve a flow control problem which has not been discussed here and might have been specific to the NX protocols. It had the side effect of being faster, and was used for that reason in applications. Because it was useful, it was propogated to the iPSC/860 and Delta systems where the performance effects were more pronounced. Since the Delta system is effectively a prototype of the Paragon system, we expect to continue to see a performance win. We would certainly implement ready receive in MPI if it is part of the standard. So, it will not be ignored in all implementations. Certainly vendor input is valuable in assessing the (near-term) benefit of including ready receive in the first version of MPI, but I think that users should make the decision to keep it in or not. After all, it is a very small burden for vendors; once implemented or ignored, it is done. The bigger question is, will it be used and provide benefit enough to make it worth having. Application writers should make the call. Thats the meat of my comments, as a postscript I will try to motivate the notion that ready receive is inherently advantageous in at least the simplest implementations. A compliant implementation of MPI should run this program. Otherwise, what is the point in having tags (I oversimplify, that is a separate argument): 1 2 async send to 2, tag=1 async send to 2, tag=2 async send to 2, tag=3 blocking receive from 1, tag=3 A simple implementation with no buffering must use a protocol where the receiver somehow lets the sender know which message can be sent. Otherwise, if the sender simply sent the first message (tag=1) and waited for a matching receive, the program would deadlock because the receiver is blocked waiting for the (tag=3) message and won't post any other receive until it comes in. For simple implementations with buffering assume the message sizes are very large (too large to buffer more than part of 1 message) and the same arguments hold. Only complex protocols which can partly buffer messages can get around the problem, and then one can simply increase the number of messages sent to defeat the buffering, although this might not fall within the spirit of the standard. So, the argument is that, in general, a compliant implementation must use a protocol where information travels from a receiving process back to the sender. This is more true for simple implementations than complex ones. Such a protocol incurs a minimum 2-trip penalty for at least some messages. Ready receive semantics eliminate the need for this step. With ready receive, the implementation does not have to use two or three trips, or does not have to figure out how many trips to use. It just sends the message, and the receiver gives an error or drops the message if there is no matching receive. Paul Pierce From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 11:25:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14072; Wed, 7 Apr 93 11:25:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07583; Wed, 7 Apr 93 11:24:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 11:24:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07575; Wed, 7 Apr 93 11:24:06 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA29865; Wed, 7 Apr 93 08:24:02 PDT Message-Id: <9304071524.AA29865@ocfmail.ocf.llnl.gov> Date: Wed, 7 Apr 1993 08:32:56 -0800 To: mpi-pt2pt@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: ready receive For those interested in understanding the reasoning behind ready receive and regular receive, note that the problems they solve were addressed over 20 years ago in a paper by David Walden in the April 1972 issue of CACM (pp. 221-230). This paper discusses (among other things) the notion of a rendezvous site for interprocess communication. Section 3 of the paper contains the relevant information. Dan From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 12:42:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17089; Wed, 7 Apr 93 12:42:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12113; Wed, 7 Apr 93 12:41:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 12:41:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12105; Wed, 7 Apr 93 12:41:55 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.17) id AA20978; Wed, 7 Apr 93 11:41:51 CDT Received: by teak18.cray.com id AA12541; 4.1/CRI-5.6; Wed, 7 Apr 93 11:41:48 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304071641.AA12541@teak18.cray.com> Subject: Re: ready receive To: mpi-pt2pt@cs.utk.edu Date: Wed, 7 Apr 93 11:41:42 CDT In-Reply-To: <9304070123.AA12606@SSD.intel.com>; from "prp@SSD.intel.com" at Apr 6, 93 6:23 pm X-Mailer: ELM [version 2.3 PL11b-CRI] prp@SSD.intel.com writes: > > Certainly vendor input is valuable in assessing the (near-term) benefit of > including ready receive in the first version of MPI, but I think that > users should make the decision to keep it in or not. After all, it is a > very small burden for vendors; once implemented or ignored, it is done. > The bigger question is, will it be used and provide benefit enough to > make it worth having. Application writers should make the call. I agree that it is trivial to write a ready-receive send that just looks like or invokes the regular send. But that may not be the only cost involved. To properly use ready receive, an application must ensure that the receive is ready before the message is sent. If the application would normally be written in such a way that this just "falls out" (that is, you can simply change a "send" to a "ready-receive send"), then I agree there is minimal impact involved to the application. But what if the application required special synchronization or was written in such a way that some additional overhead was required in order to properly use ready receive? If the cost of this extra overhead is less than the benefit from the ready receive, this would make sense. But now you have an application whose optimal performance *depends* on ready-receive send working faster than the regular send. If they port this to a system in which ready-receive send is nothing more than the regular send, then they are paying the cost of the extra overhead, but getting no benefit from it. And unfortunately, the people doing the port may not realize or remember this. So, one might argue, the vendor supplying that system ought to implement the ready-receive send! But if the underlying architecture doesn't lend itself to any benefit, what should the vendor do? Use a different mechanism so that the ready-receive send is faster than the (now slower) regular send? (This isn't hypothetical -- I've thought hard the last couple days about how I would implement a faster ready-receive on the CRAY T3D, and the only way I can think of how to do it is to use a protocol similar to what Paul describes. But I have a different protocol for the T3D that I believe is faster and just as reliable; one that is made possible by features of the T3D architecture that aren't in most other systems.) If ready-receive isn't in MPI, Intel still has the option of extending MPI to support a ready-receive send on their systems. It would be clear to a user choosing to use that extension that they are making a choice that will reduce portability. But by using ready-receive in any case, they are making this choice; this would make the decision explicit. In a sense, this is a general issue. People have argued for several MPI functions because they offer the enticing prospect of both portability and optimization. But this analysis is based on the design of the current architectures that people are familiar with, and in the long run may prove specious. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 13:49:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18831; Wed, 7 Apr 93 13:49:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15376; Wed, 7 Apr 93 13:48:39 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 13:48:38 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15367; Wed, 7 Apr 93 13:48:34 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 7 Apr 93 10:48 PDT Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06918; Wed, 7 Apr 93 10:46:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA17058; Wed, 7 Apr 93 10:46:26 PDT Date: Wed, 7 Apr 93 10:46:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: interrupt driven To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@carbon.pnl.gov, mpi-pt2pt@cs.utk.edu Message-Id: <9304071746.AA17058@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. > > I have heard, can't recall from whom, that this carries the consequence > that we should not define any feature of MPI which asks for the > implementor of MPI to make use of interrupt driven receives, or active > messages. This seems to me to be a bogus argument. I (Rik Littlefield) have regularly made arguments that are close enough to this to be worth clarifying or defending. Think of the MPI specification as being divided into two parts: 1) things you can implement with portable code, and 2) things you can't. My bias is to make the POTENTIALLY portable part of MPI as large as possible. (I emphasize "potentially" to make it clear that I am talking about interface definitions. I encourage implementors to do whatever they want internally.) As an example, consider groups and collective communication. All other factors being equal, I would strongly prefer a specification that permitted groups and collective communication, including group formation and disbanding, to be implemented using just the standardized MPI point-to-point capabilities. This sort of layerability potentially has several good effects. Three of these are 1) earlier widespread availability via porting, 2) better focusing of vendor effort on tuning the point-to-point layer, and 3) the capability to create portable MPI-plug-compatible implementations that are tuned for different requirements, e.g., debugging versus performance. Of course, all other factors are not equal. Strict layerability also has some potentially bad effects. Two of these (depending on where you draw the line between layers) are 1) the loss of integrated syntax (e.g., being able to use rank directly in the point-to-point calls), and 2) loss of functionality that is not supported by lower layers (e.g., asynchronous name servers). Whether the good outweighs the bad is a matter of opinion. The applications that I deal with now and anticipate dealing with over the lifetime of MPI-1 can be characterized as follows: . sensitive to performance, . intended to run on lots of platforms, . not sensitive to syntax, and . not needing name server support. Applications like these are best served by layerability. Others' mileage may differ. Lyndon continues: > In my mind the primary reason for not providing hrecv/hsend type > facilities is that it is very difficult to standardise these facilities > across different operating systems, particularly in terms of what the > handler procedure is and is not allowed to do (e.g., can it do I/O, can > it use MPI, ...). It seems to me clear that on many (all) machines of > interest the system itself will be making use of similar facilities. > Who would write MPI for CM-5 without using active messages? It seems to > me that for example non blocking communications really do ask for use of > interrupt/active messages. > > Is there agreement, disagreement, or what on this point? Please do let > me know. I agree that it is important to distinguish between exporting a standardized hrecv/hsend and and exporting a standardized asynchronous name server interface. The former seems hopeless; the latter is doable, but at the cost of potentially detracting from other features such as performance and prompt widespread availability. Thus, I do not agree that it is bogus to argue against requiring asynchronous servers et.al. in MPI-1. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 07:35:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08643; Thu, 8 Apr 93 07:35:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06247; Thu, 8 Apr 93 07:33:36 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 07:33:35 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06237; Thu, 8 Apr 93 07:33:32 -0400 Date: Thu, 8 Apr 93 12:33:28 BST Message-Id: <2110.9304081133@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-pt2pt: ready receiver To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI colleagues. This is the promised "ready-receiver" follow up to my "mpi-comm: various (long)" message. It is not so long :-) The first part follows through from the mentioned letter. The second part is more thoughts on ready receiver unrelated to the mentioned letter. o--------------------o I begin by describing ready-receiver communication for secure and insecure contexts. For an insecure context this is exactly as we have discussed in the meetings. For a secure context I suggest that the ready-reciever is exactly the same as an insecure-context, except that send gets an acknowledge from the receiver MPI saying whether the ready-receiver message is accepted or discarded (although I am happy to drop the suggested acknowledge for secure ready-receiver in which case it really does become less than secure.) o--------------------o Now I wish to make a more general comment about ready-receiver which is not in any way related to the secure/insecure context suggestion. My understanding of the argument for ready-receiver is that, if used in MPI user software, will provide performance benefit on some machines (where ready-receiver is implemented with essentially no protocol) and will not incur penalty on other machines (where ready-receiver is implemented as regular). I claim that on the latter machines software written with ready-receiver will actually go slower than equivalent software written without ready-receiver. The reason for this claim is quite simple, here is an example. Consider a "classic" case where ready-receiver can optimise a program, a fan-in and fan-out algorithm over a binary tree. Each node in the tree, which is not a leaf and not the root, executes the following in versions with ready-receive and without ready-receive. With Ready-receiver | Without Ready-receiver ------------------- | ---------------------- | regular receive from each child | regular receive from each child | process child messages | process child messages | start nbrecv from parent | | send to parent | send to parent | wait for nbrecv from parent | recv from parent | ready-receiver send to each child | send to each child When ready-receiver is implemented as send, the version on the left (written with ready-receiver) is slower than the version on the right (written without ready-receiver) since the node must perform more operations, and in particular may have to hit the system one more time, as the diagram clearly shows. The summary of this is that we can expect actual use of ready-receiver to sometimes make software go faster and sometimes make software go slower. I guess we beg the questions: A) How much faster is the left version than the right version when ready-receiver is not the same as regular? B) How much slower is the left version than the right version when ready-receiver is the same as regular? o--------------------o Comments, questions, (flames :-) please? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 08:57:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09502; Thu, 8 Apr 93 08:57:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10253; Thu, 8 Apr 93 08:56:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 08:56:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10245; Thu, 8 Apr 93 08:56:29 -0400 Date: Thu, 8 Apr 93 13:56:24 BST Message-Id: <2167.9304081256@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: timeouts - a proposal To: nessett@ocfmail.ocf.llnl.gov (Danny Nessett), mpi-pt2pt@cs.utk.edu In-Reply-To: Danny Nessett's message of Tue, 6 Apr 93 08:04:05 PDT Reply-To: lyndon@epcc.ed.ac.uk Regarding time-outs in the non-blocking wait calls, proposed by Danny Nesset. Some of the programmers either within, or assocaited with, EPCC have specifically asked for this capability. Exactly this facility is provided in the Meiko system CSN. I support this proposal, and agree that we need to seek vendor advice on whether they can sensible implement this, subject to a single amendment. The amendment is that the value to pass for an "infinite timeout" should be a named constant as opposed to the literal constant -1 (although I realise this is really a langauge binding issues). Vendors - please advise us. I should point out that I strongly disagree with the authors statement that nonblocking communication is advanced MPI usage that scientific programmers do not have to think about. In order for the programmer to write a portable (i.e. safe) program using the point-to-point communications the programmer will have to write programs using a mixture of blocking and nonblocking sends and receives. Of course, the programmer is free to write an unsafe program (which is potentailly simpler) using only blocking sends and receives. Let us hope that such programmers realise that such programs are not really portable. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 11:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12590; Thu, 8 Apr 93 11:46:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17500; Thu, 8 Apr 93 11:43:29 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 11:43:28 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17492; Thu, 8 Apr 93 11:43:25 -0400 Date: Thu, 8 Apr 93 16:43:20 BST Message-Id: <2373.9304081543@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-pt2pt: unsafe program? (short) To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues We have agreed that the following is an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 0 Send to Process 1 Tag 0 Receive from Process 2 Tag 0 Receive from Process 1 Tag 0 So it seems that the following is also an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 1 Receive from Process 1 Tag 2 Send to Process 2 Tag 2 Receive from Process 1 Tag 1 Does anyone disagree? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 13:11:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13793; Thu, 8 Apr 93 13:11:30 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21714; Thu, 8 Apr 93 13:09:57 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 13:09:56 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21702; Thu, 8 Apr 93 13:09:54 -0400 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA04999; Thu, 8 Apr 93 13:09:52 -0400 Date: Thu, 8 Apr 93 13:09:52 -0400 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9304081709.AA04999@msr.EPM.ORNL.GOV> To: mpi-pt2pt@CS.UTK.EDU Subject: Re:mpi-pt2pt: unsafe program? (short) Lyndon "the prolific" writes: So it seems that the following is also an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 1 Receive from Process 1 Tag 2 Send to Process 2 Tag 2 Receive from Process 1 Tag 1 Does anyone disagree? >We will not serve our purpose if we declare every program >that requires buffering to be unsafe. >The vast majority of message-passing programs are written now >assuming some buffering. >Many programs suffer a big performance hit if they have to be >written with synchronous communication exclusivily. (to be "safe") Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 14:54:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15960; Thu, 8 Apr 93 14:54:19 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27575; Thu, 8 Apr 93 14:53:43 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 14:53:42 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27567; Thu, 8 Apr 93 14:53:39 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA23393; Thu, 8 Apr 93 14:53:37 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06629; Thu, 8 Apr 93 14:53:37 EDT Date: Thu, 8 Apr 93 14:53:37 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304081853.AA06629@b125.super.org> To: mpi-pt2pt@cs.utk.edu Subject: ready receive I am glad to see that ready receive is being discussed. I agree with the comments that if a lot of machines don't implement a separate ready receive then there is the chance that users will write programs assuming it is there and then possible get slower results. I can see this especially happening if ones adds synchronizations that would not otherwise be present. Lyndon raised the issue of the extra expense in the example of fan-in/fan-out if one uses forced type semantics but only gets a regular receive. I wrote a program to test this on the Delta since one can get ready receive (forced type) on that machine. All tests had these things in common. They did 10 trials to reduce the noise. The program was run twice to make sure it was reproducible. I gsync() before each trial and test to make sure each node starts at about the same time. After that I also send another long blocking message to really make sure they started about the same time. I think the code is correct :-) There were only 2 nodes to keep the case simple and clean. Node 0 can be thought of as the leaf and node 1 as the parent in Lyndon's example. Node 0 sends its information to node 1 who then returns it to node 0. With more nodes, node 1 would pass the data on and receive it again before it sends it back to node 0. The difference in the 3 cases is whether you use ready receive semantics and if you actually do the special type send. For the pseudo code below irecv is non-blocking, crecv is blocking and csend is blocking. forced (ready receive): node 0 node 1 ------ ------ irecv (tag=2) crecv(tag=1) csend(tag=1, reg) csend(tag=2, forced) wait(tag=2) regular: node 0 node 1 ------ ------ csend(tag=1, reg) crecv(tag=1) crecv (tag=2) csend(tag=2, regular) forced/regular (ready receive semantics without the special send): node 0 node 1 ------ ------ irecv (tag=2) crecv(tag=1) csend(tag=1, reg) csend(tag=2, regular) wait(tag=2) The times in seconds for various message sizes is: node # bytes forced regular reg/forced 1 100 2.946200e-04 2.283100e-04 2.955100e-04 0 100 4.382000e-04 3.820900e-04 4.548500e-04 1 500 4.092900e-04 4.174500e-04 4.086200e-04 0 500 6.818600e-04 6.904600e-04 6.711400e-04 1 2500 8.961900e-04 8.948600e-04 8.995300e-04 0 2500 1.155420e-03 1.288320e-03 1.223410e-03 1 12500 3.110890e-03 3.712720e-03 3.702740e-03 0 12500 3.494100e-03 4.035370e-03 3.953160e-03 1 62500 1.482880e-02 1.787127e-02 1.755866e-02 0 62500 1.542180e-02 1.819615e-02 1.786163e-02 1 312500 7.446586e-02 8.802909e-02 8.647874e-02 0 312500 7.508341e-02 8.834550e-02 8.674206e-02 1 1562500 3.713712e-01 4.369691e-01 4.320925e-01 0 1562500 3.719815e-01 4.372853e-01 4.323565e-01 I note a few general things about these results. Node 0 is generally slower because it is receiving the second message. There is a greater difference in time between the two nodes for smaller messages. Several general conclusions can be drawn: 1) forced type is faster than regular once the message is several thousand bytes long. The times are about 17% more for regular than forced for long messages. This is as advertised (remember that only one of the two messages was forced type). 2) Doing the irecv/wait is the same or faster than the "regular" send semantics even without doing a forced type send. This is the opposite of what was probably expected. With the caveat that these results are for the Delta, it seems that ready receive (forced type) semantics are better in general. Thus, if you don't have to add synchronization to your code and you are sending messages of sufficient length, then using ready receive is a win. If you add a synchronization to achieve ready receive then it could clearly slow your code down. I hope this answers the question Lyndon raised. There is no penalty (on the Delta) for solely using ready receive semantics. I clearly read all of his messages carefully and even spent several hours to answer one of his many questions. I wonder if I really get the beer :-) From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 15:24:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16483; Thu, 8 Apr 93 15:24:27 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28843; Thu, 8 Apr 93 15:23:20 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 15:23:19 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28835; Thu, 8 Apr 93 15:23:13 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 12:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08178; Thu, 8 Apr 93 12:11:40 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA19332; Thu, 8 Apr 93 12:11:37 PDT Date: Thu, 8 Apr 93 12:11:37 PDT From: rj_littlefield@pnlg.pnl.gov Subject: buffering proposal to mpi-envir To: geist@msr.EPM.ORNL.GOV, gropp@mcs.anl.gov, mpi-envir@CS.UTK.EDU, mpi-pt2pt@CS.UTK.EDU Cc: d39135@carbon.pnl.gov Message-Id: <9304081911.AA19332@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@CS.UTK.EDU, mpi-envir@CS.UTK.EDU Al Geist writes: > We will not serve our purpose if we declare every program > that requires buffering to be unsafe. > The vast majority of message-passing programs are written now > assuming some buffering. > Many programs suffer a big performance hit if they have to be > written with synchronous communication exclusivily. (to be "safe") I agree, and I have previously informally proposed a way to get around this problem in a system-independent fashion. This seems like a point-to-point issue, but the minutes of the Feb.17-19 meeting say that it was deferred to the Environment subcommittee. So, I hereby officially offer this proposal to the Environment subcommittee, with cross-posting to both groups. PROPOSAL TO SUPPORT MPI MESSAGE BUFFERING IN USER-PROVIDED SPACE 1. INTERFACE An application program can optionally provide buffer space for MPI's use: e.g. MPI_USER_PROVIDES_BUFFER (len,buffer) where IN len is the length of 'buffer', in bytes. OUT buffer is a scratch array of len bytes for MPI's use in buffering messages 2. FUNCTIONALITY If the above routine is called, then for subsequent sends, MPI must *act as if* message data is being buffered by the sender. That is, user-provided buffer space may be consumed by outgoing messages, but not by incoming ones. 3. POSSIBLE IMPLEMENTATION One approach is for MPI to simply transform all blocking sends into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) No doubt many optimizations within MPI are possible -- the proposal just requires that MPI act as described here. 4. DISCUSSION The intended usage is that an application will provide buffer space once, immediately after MPI initialization. This is a minimalist proposal. An application may not be able to compute tight bounds on how much buffer space is actually required, but at least the proposed interface provides a single point-of-modification to aid in porting. No mechanism is proposed for canceling MPI access to the user-provided buffer. No mechanism is proposed to support multiple buffer spaces, although this might be convenient for some libraries. Allowing user-provided buffer space to be bound to a particular context (per Lyndon's recent suggestion regarding 'secure' communications) might meet both of these needs, but further study of that extension would be required. --Rik Littlefield ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 19:38:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20014; Thu, 8 Apr 93 19:38:57 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11689; Thu, 8 Apr 93 19:36:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 19:36:21 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11681; Thu, 8 Apr 93 19:36:18 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 16:27 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08549; Thu, 8 Apr 93 16:25:04 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20464; Thu, 8 Apr 93 16:25:00 PDT Date: Thu, 8 Apr 93 16:25:00 PDT From: rj_littlefield@pnlg.pnl.gov Subject: receiver-ready send, performance data To: lederman@b125.super.org, mpi-pt2pt@cs.utk.edu Cc: d39135@carbon.pnl.gov Message-Id: <9304082325.AA20464@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Steve Lederman writes: > Several general conclusions can be drawn: > > 1) forced type is faster than regular once the message is several > thousand bytes long. The times are about 17% more for regular than > forced for long messages. This is as advertised (remember that only > one of the two messages was forced type). > > 2) Doing the irecv/wait is the same or faster than the "regular" send > semantics even without doing a forced type send. This is the opposite > of what was probably expected. I think it's a bit optimistic to say that these conclusions are "general". The situation that Steve outlines is highly asymmetric -- process 0 does basically all the receiving while 1 does the sending. I've run literally hundreds of tests under symmetric conditions, both two-party and three-party transfers (i.e., exchange and shift), and I get results that are different from Steve's. I agree that forced types have higher bandwidth. However, the amount of the increase depends on whether you're doing two- or three-party transfers, as well as what machine you're on. On the Delta, 20% is a typical number; in some situations it's closer to 2X. On the iPSC/860, things are even more confusing -- overall, using forced types can be about the same or 2X faster than unforced, or something in between on average, and it's not even deterministic. Oh, and on the iPSC/860, breakeven for forced versus unforced is at 100 bytes even with the extra sync, versus several thousand bytes on the Delta. (A quick summary and discussion of my data is available in "Characterising and Tuning Communications Performance on the Touchstone DELTA and iPSC/860 (extended abstract)", available via anonymous ftp from delilah.ccsf.caltech.edu as file delta/documents/PNL_papers/IUGpaper.ps) I disagree with Steve's conclusion about irecv/wait being the same or faster than crecv. While that's usually true in his tests (but not always -- see bytes=100), I have consistently found the opposite. Steve continues: > ... If > you add a synchronization to achieve ready receive then it could > clearly slow your code down. > > ... There is no penalty > (on the Delta) for solely using ready receive semantics. These two statements seem contradictory. In the general case, you do have to add synchronization to use receiver-ready semantics, and that can slow you down if the messages are short. There ain't no such thing as a free lunch. My specific conclusion about timing is that On Intel systems, long messages are faster if forced and short messages aren't, and if you really care, your code had better be prepared to adapt. My general conclusion about timing is that General conclusions about timing are really dangerous to make. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 19:45:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20045; Thu, 8 Apr 93 19:45:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11921; Thu, 8 Apr 93 19:44:45 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 19:44:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11913; Thu, 8 Apr 93 19:44:42 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 16:41 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08563; Thu, 8 Apr 93 16:39:21 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20547; Thu, 8 Apr 93 16:39:16 PDT Date: Thu, 8 Apr 93 16:39:16 PDT From: rj_littlefield@pnlg.pnl.gov Subject: receiver-ready send, opinion To: lederman@b125.super.org, mpi-pt2pt@cs.utk.edu Cc: d39135@carbon.pnl.gov Message-Id: <9304082339.AA20547@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu I confess to having conflicting opinions about receiver-ready send. On the one hand, I like it a lot because it runs faster on some machines and it isn't much harder to use than regular send. If it's guaranteed safe with no extra sync, then I just use it. Otherwise I write adaptive codes that figure out whether to use it or not based on the message length. That strategy ports fine to systems where receiver-ready send doesn't help. On the other hand, I don't like the concept as it appears in MPI. I see the basic issue as Should MPI support a variety of protocols with different performance characteristics? In the specific case of receiver-ready send, we (the MPI committee) have answered "yes". But in the general case, we have answered "no" by specifying that any form of send can match with any form of receive. For example, an MPI implementation effectively can't use a "two-trip receiver pulls" (2TRP) protocol for exact-match messages, even though this would naturally handle many of the situations for which I now use receiver-ready send. This is because under the MPI spec, the sender doesn't know whether the receiver is going to do exact-match or wildcard, and using 2TRP with wildcard receive would be prohibitively expensive. It's true that the semantics of receiver-ready send allows MPI to use a more efficient protocol. But I suspect that an even greater improvement could be obtained if we required matching of send and receive modes, which we already voted is not acceptable. Perhaps receiver-ready send is just one of those borderline questions that won't be resolved except by some random process like flipping a coin or taking an MPI vote in the afternoon. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 10:28:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28034; Fri, 9 Apr 93 10:28:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21283; Fri, 9 Apr 93 10:25:49 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 10:25:47 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21268; Fri, 9 Apr 93 10:25:45 -0400 Date: Fri, 9 Apr 93 15:25:41 BST Message-Id: <3089.9304091425@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re:mpi-pt2pt: unsafe program? (short) To: geist@msr.EPM.ORNL.GOV (Al Geist), mpi-pt2pt@CS.UTK.EDU In-Reply-To: Al Geist's message of Thu, 8 Apr 93 13:09:52 -0400 Reply-To: lyndon@epcc.ed.ac.uk Al writes: > > Lyndon "the prolific" writes: > So it seems that the following is also an unsafe program. > > > Process 1 Process 2 > --------- --------- > > Send to Process 2 Tag 1 Receive from Process 1 Tag 2 > Send to Process 2 Tag 2 Receive from Process 1 Tag 1 > > Does anyone disagree? > > >We will not serve our purpose if we declare every program > >that requires buffering to be unsafe. I do not with to dispute your point. I asked a simple question, as I was unclear, and a colleague at EPCC asked me the question. Do I take your sentence as agreement that the program is unsafe? > >The vast majority of message-passing programs are written now > >assuming some buffering. I do not with to dispute your point. > >Many programs suffer a big performance hit if they have to be > >written with synchronous communication exclusivily. (to be "safe") > This seems to be a rather strong conjecture. I would be happy to receive significant justification. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:27:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29032; Fri, 9 Apr 93 11:27:53 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23903; Fri, 9 Apr 93 11:27:21 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:27:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23895; Fri, 9 Apr 93 11:27:18 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA24572; Fri, 9 Apr 93 08:27:01 PDT Message-Id: <9304091527.AA24572@ocfmail.ocf.llnl.gov> Date: Fri, 9 Apr 1993 08:36:04 -0800 To: lyndon@epcc.ed.ac.uk From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: timeouts - a proposal Cc: mpi-pt2pt@CS.UTK.EDU I agree with Lyndon's suggestion that MPI should use a named constant for representing an "infinite timeout". Whether or not non-blocking communication is advanced MPI usage rests on a subjective opinion of what is elementary and what is advanced. Dan From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:29:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29039; Fri, 9 Apr 93 11:29:22 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23929; Fri, 9 Apr 93 11:28:55 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:28:54 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23921; Fri, 9 Apr 93 11:28:52 -0400 Date: Fri, 9 Apr 93 16:28:49 BST Message-Id: <3185.9304091528@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: timeouts - a proposal To: nessett@ocfmail.ocf.llnl.gov (Dan Nessett), lyndon@epcc.ed.ac.uk In-Reply-To: Dan Nessett's message of Fri, 9 Apr 1993 08:36:04 -0800 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@CS.UTK.EDU Dan writes: > I agree with Lyndon's suggestion that MPI should use a named constant for > representing an "infinite timeout". Whether or not non-blocking > communication is advanced MPI usage rests on a subjective opinion of what > is elementary and what is advanced. > I concur! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:44:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29311; Fri, 9 Apr 93 11:44:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24508; Fri, 9 Apr 93 11:43:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:43:32 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24500; Fri, 9 Apr 93 11:43:30 -0400 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA02495; Fri, 9 Apr 93 09:43:28 MDT Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA01706; Fri, 9 Apr 93 09:44:23 -0600 Message-Id: <9304091544.AA01706@newton.sandia.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: mpi-pt2pt: unsafe program? (short) In-Reply-To: Your message of Thu, 08 Apr 93 16:43:20 -0000. <2373.9304081543@subnode.epcc.ed.ac.uk> Date: Fri, 09 Apr 93 09:44:22 MST From: mpsears@newton.cs.sandia.gov Lyndon asks if the program Process 1 Process 2 send(2, tag=1) recv(1, tag=2) send(2, tag=2) recv(1, tag=1) is unsafe. The answer is yes. With no buffering, process 2 cannot complete the first recv, since process 1 has not sent the first msg yet and process 1 cannot complete the first send, since process 2 has not posted a receive for it. Even if buffering is available, the program can still hang with large enough messages. The only safe way to write this program is for one of the processes to post its first operation in unblocked mode. The basic reason for the strong requirement on a safe program is that there is no good theoretical description of what a safe program is in the presence of buffering. One possible weakening would be to allow an MPI programmer to assume that when a message is sent by one process to another, the second process can detect if the message envelope has arrived (via probe), without blocking the first process. Therefore, zero length messages would always be safe. Al Geist's comments are appropriate. MPI is defined so that if you want to write a safe program you must assume no buffering, which requires a LOT of programmer effort, and like the ready receiver argument this programmer effort may indeed be counterproductive on machines which do have significant buffering. The existing constraint, while theoretically attractive, may be too limiting. We can legislate whatever we want in meetings and build a useless specification. Further opinions: 1) time-outs are a very bad idea. They add unlimited ways for things to go wrong. Remember that a timeout is a fixed threshold (unless you want to be able to change the timeout dynamically) and will always be chosen wrong in some situation. Your program will fail just when your boss or your customer is looking over your shoulder. There are usually better ways to accomplish the same thing without timeouts. For example, suppose the program has a process waiting for a task and there are no more tasks. Rather than using a time-out, a better way is to send a task request message that tells the waiting process to quit waiting for tasks. Timeouts are useful in network software which utilizes unreliable protocols, because they are the only way to detect certain classes of errors. MPI assumes reliable delivery of messages; therefore timeouts are inappropriate. 2) ready-receiver is a moderately bad idea, (this is my own position) for reasons already posted by others on the net. Mark Sears Sandia National Laboratories - ------- End of Unsent Draft ------- End of Forwarded Message From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 12:01:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AB29589; Fri, 9 Apr 93 12:01:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25338; Fri, 9 Apr 93 12:00:05 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 12:00:04 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25311; Fri, 9 Apr 93 12:00:01 -0400 Date: Fri, 9 Apr 93 16:59:58 BST Message-Id: <3212.9304091559@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-pt2pt: unsafe program? (short) To: mpsears@newton.cs.sandia.gov, mpi-pt2pt@cs.utk.edu In-Reply-To: mpsears@newton.cs.sandia.gov's message of Fri, 09 Apr 93 09:44:22 MST Reply-To: lyndon@epcc.ed.ac.uk Mark writes: > Al Geist's comments are appropriate. This point is well taken. > MPI is defined so that > if you want to write a safe program you must assume no buffering, > which requires a LOT of programmer effort, This is another subjective thing. Please believe me that I know and work with a lot of programmers with varying levels of experience and scientific backgrounds who find this way of programming is LITTLE effort indeed. > and like the ready > receiver argument this programmer effort may indeed be counterproductive > on machines which do have significant buffering. The existing > constraint, while theoretically attractive, may be too limiting. > We can legislate whatever we want in meetings and build a useless > specification. Understood. I am trying to do things to separate the secure communication I propose from the regular communication which provides buffering. Please do see my long message to mpi-comm of April 8 "Subject: mpi-comm: various (long)". I strongly encourage the inclusion of secure mode of communication in MPI. I also strongly encourage the inclusion of buffered mode and effort to allow the user management of how much buffering and where. I believe they are two ways of programming both of which MPI must support, rather than mandate one at the expense of the other, and that this is critical to the long term success of MPI as a de facto standard. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 12:24:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29791; Fri, 9 Apr 93 12:24:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26324; Fri, 9 Apr 93 12:24:14 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 12:24:13 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26316; Fri, 9 Apr 93 12:24:11 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA11445; Fri, 9 Apr 93 12:24:06 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06800; Fri, 9 Apr 93 12:24:06 EDT Date: Fri, 9 Apr 93 12:24:06 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304091624.AA06800@b125.super.org> To: rj_littlefield@pnlg.pnl.gov Cc: mpi-pt2pt@cs.utk.edu, d39135@carbon.pnl.gov In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Thu, 8 Apr 93 16:39:16 PDT <9304082339.AA20547@sodium.pnl.gov> Subject: receiver-ready send, opinion I agree with Rik's comments that the performance does vary with the circumstance that you use forced type on the Delta. I mostly wrote that code to check Lyndon's question of the performance difference in a blocking receive vs. a non-blocking receive followed by a wait. It was not great. So it often comes down to (as Rik noted) the point that ready receive requires user skill if you want to be sure you get better performance and this will not easily port to other machines in many circumstances. The flip side is that "us" performance seekers want the extra performance and have routinely been willing to pay for it in more complex code. We do lots of ifs on message size, type of communication, etc. if we want optimal performance. I have concerns about typical users and ready receive. I have even thought about only making it available with message buffers but this may reduce performance. I would like to see ready receive but would be pleased by a better overall proposal on how to fit it into MPI. Steve From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 13:01:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00631; Fri, 9 Apr 93 13:01:48 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28117; Fri, 9 Apr 93 13:01:03 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 13:01:01 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28093; Fri, 9 Apr 93 13:00:56 -0400 Received: by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA25915; Fri, 9 Apr 93 10:00:54 PDT Date: Fri, 9 Apr 93 10:00:54 PDT From: nessett@ocfmail.ocf.llnl.gov (Danny Nessett) Message-Id: <9304091700.AA25915@ocfmail.ocf.llnl.gov> To: mpi-lang@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: cross-language support - a proposal (long) Cc: nessett@ocfmail.ocf.llnl.gov I am cross posting this message to the point-to-point list, since it contains a proposed modification to the point-to-point proposal. Those interested in point-to-point, but not language binding issues should skip to section 3, and evaluate whether the changes are acceptable. 1. The Problem -------------- At the recent MPI meeting in Dallas (31 March - 2 April), the language binding subcommittee proposed that the MPI standard make no provision for interoperability between MPI-based programs written in different programming languages. I objected to that suggestion and volunteered to develop a proposal that would allow such language interoperability. My objections were and are based on the following considerations : o MPI is a mulit-use standard, being developed for homogeneous multi- processors, homogeneous computer clusters and heterogeneous distributed systems. The objective is to provide a message passing interface that can be used in all of these environments. In addition, there is an objective to promote portability of MPI-based codes from one environment to another. Correct operation of MPI-based codes in a heterogeneous environment requires the MPI implementation to accommodate different formats for the communicated data. As a side effect, programs written in different programming languages will interoperate as long as the data types are conceptually the same. For example, REALs in Fortran and floats in C are conceptually the same. Unless there is a requirement that MPI support cross-language interoperability, some programs will work in heterogeneous environments, but not work in homogeneous environments. This will give the appearance that MPI is poorly designed. o MPI can be used in at least two different ways. It can be used for intercommunication between application executables that are aware (from the programmer's point of view) of each other's internal data structures and algorithms. It can also be used to provide services, much like a library provides services, in which the internal data structures and algorithms are not visible to the service client. In this second class of use, MPI acts as a generic service interface, supporting a wide variety of applications. Commonly, this generic service interface is known as client/server. Traditionally, libraries define their interface data structures and procedure prototypes in a specific programming language. Some libraries provide multiple interface definitions, one for each particular programming language they support. However, when using a generic service interface to access what is the equivalent of library services, this approach becomes problematic. The MPI interface is itself specified in terms of one or more programming language bindings. There is no useful way to specify another language binding for the client/server interface. Consequently, another approach is used for the client/server interface, which is programming language independent. Generally, server writers specify an interface in terms of a message parameter that represents the function or procedure to execute and in conjunction with a particular value of that parameter, specify a set of parameters that are the function's or procedure's arguments. Note that with a message passing based generic service interface, such as MPI (as opposed to a remote procedure call generic service interface), the service call is not constrained to obey function or procedure semantics, since the client is not obliged to block while the server provides the service. However, that is an aside. Since the server's interface is specified in terms of MPI data types, rather than in terms of the data types of the programming language used to write the server, there is a complete decoupling of the programming language used to write a server from the programming language used to write the client (and vice versa). The server can use one MPI language binding to offer service and the client can use another MPI language binding to access those services. An example may clarify this somewhat. Suppose I wish to write a server that provides a matrix manipulation service. I define my interface as follows : first parameter : operation to be performed (integer) - 1 = matrix sum 2 = matrix multiply 3 = matrix inversion 4 = matrix transposition 5 = return result second parameter: number of rows (integer) in matrix (or matrices) third parameter : number of columns (integer) in matrix (or matrices) fourth parameter: first matrix for all operations (single precision real array, row major order) fifth parameter : second matrix for operations 1 and 2 (single precision real array, row major order) For this interface, the operation and parameters will be specified as MPI data types. For a C or C++ language binding of MPI, these data types will be "int" and "float". For a Fortran language binding of MPI, these data types will be "INTEGER" and "REAL". This means there really is no way to prevent cross language interactions, other than to provide a "caveat emptor" warning to the programmer. Some programmers will not read the manual that closely; will implement a cross language service interface; will use it on various machines with no problems; and finally will curse the MPI implementors when it doesn't work on other machines. This will reduce confidence in MPI as a standard message passing interface. Note the property described in the previous bullet. In a heterogeneous environment, even if both client and server are written in the same language, the MPI implementation must accommodate conversion between different ranges of values for a particular type. For example, if one machine represents integers as 32 bit quantities, while another represents them as 64 bit quantities, sending an integer parameter from the second to the first requires a check to ensure the 64 bit value can be represented as a 32 bit value (e.g., that 100,000 represented as a 64 bit number fits into a 32 bit representation). This check also allows a client written in one language (say, C) to call a server written in another language (say, Fortran). 2. My view of the issues ------------------------ Below I summarize my view of the issues in regards to cross-language MPI service. I am sure that there are other issues that I have not thought of and welcome others to contribute them to the discussion. o As I have stated above, MPI is a multi-use standard being targeted for homogeneous multi-processors, homogeneous computer clusters and heterogeneous distributed systems. Vendor and user communities representing each of these environments are participating in the MPI standardization process. This invariably leads to a conflict of objectives. For example, the community most interested in homogeneous multi-processors does not want to sacrifice the performance of parallel programs running on these machines in order to accommodate heterogenous distributed processing or cross-language MPI support. Alternatively, those interested in heterogenous distributed systems don't want to constrain the MPI standard in such a way that severly limits its applicability in those environments. Those interested in homogeneous computer clusters are probably somewhere in between with their objectives. Consequently, if the MPI standard is to provide cross-language support, it should do so in a way that doesn't penalize the performance of programs running on homogeneous multi-processors, while at the same time minimizing the implementation effort for cross-language and heterogenous distributed system support. Also, programmers who intend to use only one language should not have to think about cross-language issues. o Cross-language support raises the question of translation between data values that are of a fundamentally different type (as opposed to being of a different "kind" in the Fortran 90 sense). For example, should the MPI standard provide support that would allow a Fortran program to send a "complex" value to a program written in C? (By this I mean support the transmission of the Fortran "complex" type to a C program, not the transmission of a "complex" value represented in some user specified way, like two real values). A cross-language facility may or may not allow this, depending on the degree of automation the standard is designed to provide. It is conceivable for a cross-language standard to support the communication of only those data types that are common to all of the target programming languages. There is also the issue of type coercion of transmitted values within a single language. For example C allows you to coerce a real value into an integer value when you assign a real value to an integer variable. Taking this approach would imply a sender could specify a real value in an MPI_ADD_BLOCK call, while the receiver specified an integer variable and expect MPI to perform the conversion. Allowing such coercion in the MPI interface would significantly increase its complexity, since the programmer would now have to specify not only the type being sent or received, but also the type being coerced. For this reason, I believe we have decided not to support such type coercion in MPI. o Translation between values of the same type (but perhaps different "kind") across languages requires knowledge of the mapping between a particular language type and its machine representation. For example, in Fortran 77 a REAL might map to an IEEE 32 and a DOUBLE PRECISION to an IEEE 64. In C a float might map to an IEEE 32, a double to an IEEE 64 and a long double to an IEEE 64. In Fortran 90 a REAL with a given kind parameter might map to either an IEEE 32 or an IEEE 64. Similar mapping of int, long, INTEGER (with different kind parameters in Fortran 90), char, CHARACTER, LOGICAL and COMPLEX to underlying representations is required to properly support cross-language interoperability. Some language types that have been suggested as MPI types would require definition in certain languages. For example, LOGICAL in Fortran has no direct type analog in C. Similarly, there is no COMPLEX data type in C. It would be possible to simply declare use of these MPI types in a program written in an incompatible programming language as erroneous or the standard could specify a mapping between these type and non-native types in the appropriate languages (e.g., LOGICAL could map to unsigned char and COMPLEX to a structure containing two reals). In a heterogeneous environment the mapping information can be more complex. On a SUN Workstation REALs, DOUBLE PRECISIONS, floats, doubles and long doubles may map as specified above. On a CRAY, however, a REAL may map to a CRAY 64, a DOUBLE PRECISION to a CRAY 64, a float to a CRAY 64, a double to a CRAY 64 and a long double to a CRAY 128. o Cross-language interoperability may require an MPI implementation that supports a particular language binding to be aware in some way of the existence of other language bindings. This may cause a problem when new MPI language bindings are developed. The features of MPI that support cross-language interoperability should allow the graceful integration of new language bindings into the standard. 3. A Proposal ------------- In order to support cross-language use of MPI, I propose the following modification to the point-to-point draft. The proposal is in two completely independent parts. One can be accepted without accepting the other. 3.1 First Part Modify the datatype parameter values of the MPI_ADD_... functions so that they come from the following set : MPI_REAL MPI_DOUBLEPRECISION MPI_COMPLEX MPI_INTEGER MPI_LOGICAL MPI_CHARACTER MPI_FLOAT MPI_DOUBLE MPI_LONGDOUBLE MPI_SHORT MPI_INT MPI_LONG MPI_CHAR (a character array) MPI_UCHAR MPI_BYTE These values for the datatype parameter are allowed in any MPI implementation irrespective of its language binding. They specify the type of the data that is being sent/received. Notice that the first group (i.e., MPI_REAL,..., MPI_CHARACTER) are Fortran types; the second group (MPI_FLOAT,...,MPI_UCHAR) are C (C++) types and MPI_BYTE is a language independent type. The use of a type from the first group in a Fortran program indicates that the data being sent/received is in Fortran format and consequently does not require translation. The use of a type from the first group in a C (C++) program indicates that the data being sent/received is in Fortran format and so may require translaton. Similarly, the use of a type from the second group in a Fortran program indicates the data being sent/received is in C (C++) format and so may require translation. The use of a type from the second group in a C program indicates the data being sent/received is in C (C++) format and so need not be translated. A client/server interface would specify its interface in terms of these MPI types. When a MPI_ADD_... function is called, the specified type would be used as the datatype parameter irregardless of the MPI language binding. Thus, if a parameter is specified as MPI_REAL, that datatype would be specified both in Fortran and C programs using the client/server interface. To properly pass messages, both the client and server must use the same datatype for the parameter. In order for the MPI implementation to take advantage of the provided type information in a cross-language communication, it must know which Fortran types map into which C types. Therefore, the standard should specify this information. Let me make the following strawman proposal : MPI_REAL maps to MPI_FLOAT; MPI_DOUBLEPRECISION maps to MPI_DOUBLE; MPI_COMPLEX maps either to a C struct or has no mapping, which would cause an error in a cross-language communication; MPI_INTEGER maps to MPI_INT; MPI_LOGICAL maps to MPI_UCHAR or has no mapping, which would cause an error; and MPI_CHARACTER maps to MPI_CHAR. There are three C types that have no natural analogs in Fortran : MPI_SHORT, MPI_LONG, and MPI_LONGDOUBLE. MPI_SHORT and MPI_LONG probably should map to MPI_INTEGER, since it is the only integer type available in Fortran 77. MPI_LONGDOUBLE probably should map to MPI_DOUBLEPRECISION. However, that means that MPI_DOUBLE and MPI_LONGDOUBLE map to the same Fortran data type. An alternate mapping would map MPI_DOUBLE to MPI_REAL and MPI_LONGDOUBLE to MPI_DOUBLEPRECISION. This requires discussion. In addition to a standard mapping between programming language types, a particular MPI implementation must know how the supported language types map into underlying machine representations. For example, it must know that REAL maps into IEEE 32, int maps into a 32 bit 2's complement value, etc. An MPI implementation would operate as follows. I categorize the implementations by environment in order to demonstrate properties of the previously discussed issues. 3.1.1 Homogeneous Mulit-processors and Computer Clusters -------------------------------------------------------- I describe the behavior of an MPI implementation with a Fortran language binding. The corresponding actions of an MPI implemenation with a C language binding should be obvious. 3.1.1.1 Send If the datatype parameter is in the Fortran set, block copy the data using the underlying system network. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type and use the per implementation machine representation mapping to decide the underlying representation for both the Fortran and C type. If they are the same, block copy the data using the underlying system network. If not, convert the data to the appropriate underlying representation for the C type and send the converted data. 3.1.1.2 Receive If the datatype parameter is in the Fortran set, the buffer in which the data arrived is in the proper format. Make it available to the MPI caller. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type and use the per implementation machine representation maping to decide the underlying representation for both the Fortran and C type. If they are the same, the buffer in which the data arrived is in the proper format. Make it available to the MPI caller. Otherwise, convert the buffer to the appropriate underlying representation and make it available to the MPI caller. A comment on implementation strategy. It is possible for both a send and receive operation to know before hand whether it needs to translate the buffer data or not (i.e., the above algorithms can be run before the actual machine transmission is sent or received). Consequently, in those situations in which translation is necessary, the implementation can supply an intermediate buffer in which to translate or from which to translate the data. By knowing both the Fortran and C types as well as their underlying representations, the implementation can precalculate the size of the translation buffer. 3.1.2 Heterogeneous Distributed Systems --------------------------------------- Supporting MPI communications in a heterogeneous distributed system is more complicated than in a homogeneous environment. Not only must differences in programming language data types be accommodated, differences in underlying machine represenations are also a concern. The exact algorithms to use when sending and receiving depend on the particular presentation-level protocol employed. Protocols like XDR and ASN.1 use a intermediate representation of data for transmission purposes. A protocol like NDR (used in OSF DCE) transmits the data in the format of the sender, placing the burden on the receiver to translate it. In the following discussion I finesse the protocol issue by using the generic description "call the off-machine protocol translation module" to mean execute the appropriate presentation protocol algorithm. Some implementations will combine the protocol translation activity with sending or receiving the data. 3.1.2.1 Send If the datatype parameter is in the Fortran set, determine the underlying machine represenation and call the off-machine protocol translation module to put it into the proper format. Send the result to the receiver. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type. Determine the Fortran type's underlying representation and call the off-machine protocol translation module to put it into the proper format. Send the result to the receiver. 3.1.2.2 Receive If the datatype parameter is in the Fortran set, determine the underlying machine representation and call the off-machine protocol translation module to put it into the proper format. Return this result to the MPI caller. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type. Determine the Fortran type's underlying representation and call the off-machine protocol translation module to put it into the proper format. Return this result to the MPI caller. 3.2 Second Part Cross-language interoperability raises the issue of handling data types in one programming language that have no exact analog in another programming language. With the current suggested language bindings for MPI (i.e., Fortran 77, Fortran 90, C and C++), I think the following types fall into this category, one way or another : Fortran 77 LOGICAL COMPLEX Fortran 90 REAL(SELECTED_REAL_KIND(--,--)) INTEGER(SELECTED_INT_KIND(--)) LOGICAL COMPLEX COMPLEX(SELECTED_REAL_KIND(--,--)) C and C++ types included in the MPI standard have natural analogs in Fortran 77 and Fortran 90. There is at least two ways to handle these type mismatches. The simplest strategy is to generate an error when these types are referenced in an inappropriate language. This approach has the advantage of being simple to implement. It has the disadvantage that MPI-based programs written in one language without thought of using it from a program written in another language will likely use inappropriate types. The second strategy is to define analogs in each language for the non-common types. For example, LOGICAL in both Fortran 77 and Fortran 90 could map to unsigned char in C. COMPLEX in Fortran 77 and Fortran 90 could map to a struct with two float members in C. However, REAL, INT and COMPLEX with KIND parameter information cannot be handled in this way, since the "length" of the type is a machine dependent quantity. For example, on some machines INTEGER(SELECTED_INT_KIND(4)) might be equivalent to an int, while on other machines it is equivalent to a long (and not an int). Similar "length" problems exist with the REAL(SELECTED_REAL_KIND(--,--)) type. With these points in mind I propose the following. Map LOGICAL into unsigned char, COMPLEX into a C struct with two floats and disallow specification of the Fortran 90 types that specify a KIND parameter. Since Fortran 90 supports a KIND function call that the programmer can use to determine when INTEGER and INTERGER(SELECTED_INT_KIND(--)), REAL and REAL(SELECTED_REAL_KIND(--,--)), and COMPLEX and COMPLEX(SELECTED_REAL_KIND(--,--)) are equivalent, there is a work around for most cases. 4. Analysis of the Proposal --------------------------- Following is an analysis of the proposal according to the issues specified in section 2. o As described in section 3.1.1 the proposal can be implemented in such a way that it does not adversely impact the performance of either homogeneous multi-processors or homogeneous computer clusters. The programmer who writes programs in one language need never consider cross- language issues. Furthermore, the MPI types he/she would use would be natural for the programming language in which he/she is working. o The issue of mapping data types not found in one programming language into data types found in another is addressed by the second part, which covers mapping of data types by defining analagous types in the programming language from which a type is missing. Type coercion is not supported between types, since it is simple to first coerce the type (e.g., float to int) before sending it. It is not simple to support this kind of type coercion from within the MPI implementation. o The proposal handles translation of values of the same type by requiring the programmer to specify the type of data in the datatype parameter of the MPI_ADD_... functions. There is a limited amount of translation between types of different kinds due to the mapping required to associate a type in one language with a type in another language. This has the advantage of automating much of the work required to communicate values in a cross-language situation. However, there is also a possible disadvantage. In a heterogeneous distributed system the mapping could lead to the loss of information or to an error. For example, the suggested mapping associates a float with a REAL. On a CRAY a REAL is a 64-bit floating point number, while on a SUN a float is a 32-bit floating point. Transfering a REAL on a CRAY to a float on a SUN causes loss of precision and potentially could result in a translation error because the value on the CRAY cannot be represented by a 32-bit floating point. This problem also occurs on a single machine if a REAL maps to, say, an IEEE 64 and a float to an IEEE 32 (which doesn't seem likely). o Adding a new language binding to the MPI standard requires the definition of new MPI_ values and the mapping between these new values and the existing MPI_ values. Introduction of MPI implementations with support for the new language bindings can be accomplished gradually, since they will support the old MPI_, eventhough the old implementations do not support the new MPI_. As vendors upgrade their MPI implementations to conform to the new language binding standards, the new MPI_s can be used more and more until all useful implementations support them. Thus, the proposal supports the gradual introduction of new language bindings without requiring all implementations to immediately support them. From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 22:57:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11822; Fri, 9 Apr 93 22:57:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20891; Fri, 9 Apr 93 22:56:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 22:56:56 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20883; Fri, 9 Apr 93 22:56:53 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 9 Apr 93 19:45 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA09922; Fri, 9 Apr 93 19:43:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA22459; Fri, 9 Apr 93 19:43:26 PDT Date: Fri, 9 Apr 93 19:43:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: proposal -- context and tag limits To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, mpsears@newton.cs.sandia.gov Cc: d39135@carbon.pnl.gov, gropp@mcs.anl.gov, mpi-collcomm@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Message-Id: <9304100243.AA22459@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Lyndon et.al. write: > ... This seems to say that the bit > length of the envelope is fixed to some number of bits and the more > fields we want to cram into the envelope the shorter the bit lengths of > fields must be. Is there a good reason why the bit length of the > envelope shoud be fixed in this fashion, or perhaps are you arguing > that the bit length of the envelope should be as short as possible? > > > This is a question vendors might answer: how many > > context values and tag values are you willing to support on future > > platforms and how many are you willing to back fit on existing ones? > > Yes, this would be a good question for the vendors indeed. > > VENDORS - PLEASE PLEASE PLEASE DO ADVISE US ON THIS ONE. I wonder what kind of useful advice vendors could really give us. Hardware support boils down to a question of getting faster performance in exchange for some relatively small resource limit. But in almost every case I can think of, such limits are made functionally transparent to the user by automatic fallback to some slower mechanism without the resource limit. Thus we have.. fixed size register sets with compilers that spill to memory, fixed size caches with automatic flush/reload from main memory, fixed size TLB's with cpu traps for TLB reload, fixed size physical memory with virtual memory support, and so on. The only counterexample that pops to mind is fixed-length numeric values, for which reasonably well established conventions exist. No such conventions currently exist regarding tag and context values. ============ PROPOSAL TO ENVIRONMENT COMMITTEE ============== The MPI specification should 1. require that all MPI implementations provide functional support for specified generous limits (e.g., 32 bits) on tag and context values, and 2. suggest that vendors provide a system-specific mechanism by which the user can optionally specify tag and context limits that the program agrees to abide by. Even the form of these limits should remain unspecified since they may vary from system to system. ======================== END PROPOSAL ======================== Further discussion... If a vendor wishes to provide hardware support to enhance performance for some stricter limits, and if some people are able and willing to write programs within those limits, that's great. Those people on those machines will be lark happy. If the performance increase is substantial, and I'm on one of those machines, and my program is simple enough, I'll probably be one of those people. However, I am not aware of any system on which generous limits could not be supported, albeit with some loss of performance compared to staying within the (currently hypothetical) hardware-supported limits. Everyone I know would MUCH prefer suboptimal performance over HAVING to rewrite applications to conform to varying and inconsistent hard limits. Yes, I recall the many arguments against mandating specific limits. But, I claim that those arguments are misdirected. They are based on analogy to things like word length and memory size, which I again note are subject to well established conventions and principles. (You can't run big programs on small machines, and we pretty much agree about what "big" and "small" mean.) In the case of context and tag values, such conventions do not exist, and a very wide range of conflicting limits have been discussed at various times and places. I believe that we will not meet our goal of portability if we do not specify usable limits on tag and context values. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Mon Apr 12 14:54:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20748; Mon, 12 Apr 93 14:54:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28670; Mon, 12 Apr 93 14:53:50 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 12 Apr 1993 14:53:49 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28662; Mon, 12 Apr 93 14:53:45 -0400 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA02194; Mon, 12 Apr 93 18:53:21 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA19563; Mon, 12 Apr 93 12:54:38 MDT Date: Mon, 12 Apr 93 12:54:38 MDT From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9304121854.AA19563@nipmuc.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: mpi-pt2pt: unsafe program? (short) > Subject: Re: mpi-pt2pt: unsafe program? (short) > Date: Fri, 09 Apr 93 09:44:22 MST > From: mpsears@newton.cs.sandia.gov > > Lyndon asks if the program > > Process 1 Process 2 > > send(2, tag=1) recv(1, tag=2) > send(2, tag=2) recv(1, tag=1) > > is unsafe. The answer is yes. With no buffering, process 2 cannot > complete the first recv, since process 1 has not sent the first msg yet > and process 1 cannot complete the first send, since process 2 has > not posted a receive for it. Even if buffering is available, the program > can still hang with large enough messages. The only safe way to > write this program is for one of the processes to post its first > operation in unblocked mode. I have a similar question to Lyndon's. Given that I don't think MPI has yet specified a lower limit on the number of outstanding requests that may exist, is the unblocked version safe. Suppose you can only have one outstanding request. Again, clearly only one outstanding request is too few, but how many is enough? I would personally expect "most" programs written with non-blocking receives to work, but they may well be unsafe. In an algorithm where many boundary exchanges are happening there may be a lot of pending non-blocking receives and this is the problem I think of when the question of safe versus unsafe arises. Regards, Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-pt2pt@CS.UTK.EDU Mon Apr 12 16:26:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22237; Mon, 12 Apr 93 16:26:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05506; Mon, 12 Apr 93 16:25:17 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 12 Apr 1993 16:25:16 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05497; Mon, 12 Apr 93 16:25:14 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.17) id AA25777; Mon, 12 Apr 93 15:25:11 CDT Received: by teak18.cray.com id AA18699; 4.1/CRI-5.6; Mon, 12 Apr 93 15:25:08 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304122025.AA18699@teak18.cray.com> Subject: Re: mpi-pt2pt: unsafe program? (short) To: mpi-pt2pt@CS.UTK.EDU Date: Mon, 12 Apr 93 15:25:03 CDT In-Reply-To: <9304121854.AA19563@nipmuc.fsl.noaa.gov>; from "Leslie Hart" at Apr 12, 93 12:54 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Leslie Hart writes: > > > Subject: Re: mpi-pt2pt: unsafe program? (short) > > Date: Fri, 09 Apr 93 09:44:22 MST > > From: mpsears@newton.cs.sandia.gov > > > > Lyndon asks if the program > > > > Process 1 Process 2 > > > > send(2, tag=1) recv(1, tag=2) > > send(2, tag=2) recv(1, tag=1) > > > > is unsafe. The answer is yes. With no buffering, process 2 cannot > > complete the first recv, since process 1 has not sent the first msg yet > > and process 1 cannot complete the first send, since process 2 has > > not posted a receive for it. Even if buffering is available, the program > > can still hang with large enough messages. The only safe way to > > write this program is for one of the processes to post its first > > operation in unblocked mode. > > I have a similar question to Lyndon's. Given that I don't think MPI has > yet specified a lower limit on the number of outstanding requests that may > exist, is the unblocked version safe. Suppose you can only have one > outstanding request. Again, clearly only one outstanding request is too > few, but how many is enough? I would personally expect "most" programs > written with non-blocking receives to work, but they may well be unsafe. > In an algorithm where many boundary exchanges are happening there may be > a lot of pending non-blocking receives and this is the problem I think of > when the question of safe versus unsafe arises. I find these discussions quite frustrating and mostly theoretical. I'd suggest we note these as issues and move on. Although opposed to recent political trends, I'd suggest we take the approach of letting the marketplace settle issues like this. The Fortran 77 standard does not specify a minimum number of COMMON blocks that a compiler must support. There is probably an implicit minimum of two (blank common and a named common block). So anyone could write and market a standard-conforming Fortran compiler that only lets you have two common blocks per compilation unit. How many users worry about this? How many users jam all their data into one named COMMON block so it will be compilable by such a brain-dead compiler? Not many. Why? Because such compilers aren't going to survive in the marketplace -- they will either be fixed (by increasing the limit), or they won't have any users. So someone trying to write portable Fortran uses a "reasonable" number of COMMON blocks (i.e., a number that is supported by all the systems of immediate interest) and figures he'll wait in case he hits a compiler limit with some new machine before worrying about it further. The same thing will be true of MPI. Who cares if someone writes an MPI implementation that supports at most one active send and receive in the entire system? If there is an official MPI test suite, the test suite will probably fail. Even if there isn't, there won't be any MPI applications that will run. Either that implementation will be fixed, or it will die on its own. The MPI standard doesn't have to solve this problem, the marketplace will. Taking a less extreme example, assume a new version of MPI is developed that can't run a code that runs on every other MPI system. I suggest that the developer of that code will point this out to the new MPI vendor. Ideally, the end result of this discussion is that the final application runs everywhere. Maybe this means the new MPI version is changed, or the application is changed, or both are changed. There may also be cases where the result is that the application is not ported to the new version of MPI, perhaps to the detriment of both vendor and developer, but maybe not. (For example, the new system may simply be too small to run that particular application. In this case, the presence of the smaller limit helped identify this situation.) The major problem with setting bounds for these things is one of coming to agreement. Using Leslie's question as an example, I suggest that we'd all agree that only one outstanding request is too few. And we'd probably agree that having 1,000,000 is more than enough. But trying to agree on a single minimum value (or even minimum order of magnitude) between 1 and 1,000,000 will take forever. And whatever we agreed on would be wrong for some situations, discouraging MPI's implementation. The other problem is that of units. I think this came up in the discussion of enquiry functions. IMO, we don't want MPI forcing a particular implementation. Let us consider the question of how do you define the maximum number of outstanding requests. One MPP vendor might want to define a value which means "number per CPU". Another would want to define his as "number per thread". A third uses a shared memory pool and wants to have a value which means "number for system as a whole". A vendor with a network implementation might want to support a different value for each machine in the network. Which unit(s) do you use for the definition? And how will this effect the other (and future) implementations? In summary, I think questions about the minimum maximum number of outstanding requests, or how many contexts and/or groups should be supported, or about whether or not a particular code sequence is safe for all sizes of data and all possible implementation styles are interesting questions that vendors and users should think about. But if MPI tries to choose and dictate the answers, it will take forever to finish and be less widely accepted and implemented. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 14 18:20:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23206; Wed, 14 Apr 93 18:20:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00352; Wed, 14 Apr 93 18:19:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 14 Apr 1993 18:19:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00344; Wed, 14 Apr 93 18:19:11 -0400 Message-Id: <9304142219.AA00344@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1697; Wed, 14 Apr 93 18:19:05 EDT Date: Wed, 14 Apr 93 18:19:01 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V2.DVI.* %%Pages: 30 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 8 56 df34 DI50 DI<12E0B3B3B3B1 EAFFFCA30E4A73811C>I<131CB3B3B3B1EAFFFCA30E4A80811C>I<12E0B3A9031B73801C>I<12 E0B3A9031B75801C>I E /Fb 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EA E738EA0700A30D0E7E8E12>3 D E /Fc 2 36 df<1506A28116801501ED00E0B712F816F0C912 E0ED0180150316001506A2250E7E902A>33 D<1203B3ABEAC30CEAF33CEA3B70EA1FE0EA0780A2 EA03007E0E257D9C15>35 D E /Fd 2 63 df<127012F8A3127005057C840D>58 D<12E01278121EEA0780EA01E0EA0078131C130FEB03C0EB00F0143C140FEC03C0A2EC0F00143C 14F0EB03C0010FC7FC131C1378EA01E0EA0780001EC8FC127812E01A1A7C9723>62 D E /Fe 4 107 df0 D15 D<12C012F0123C120FEA03C0EA00F0133C130E6D7EEB 01E0EB0078141EEC0780EC01C0EC0780EC1E001478EB01E0EB0780010EC7FC133C13F0EA03C000 0FC8FC123C12F012C0C9FCA7B612C0A21A247C9C23>21 D<12C0B3B3A9022D7BA10D>106 D E /Ff 32 122 df<1238127C127EA2123E120E121E123C127C12F81260070B798416>44 D<127012F8A312700505788416>46 D48 DI<13E0487EA2 13B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C0 13197F9816>65 DI<3801F180EA07FF5AEA1F0FEA 3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F01119 7E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFC A3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>III73 D76 D<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3 EA38E3A21303A538FE0FE0A313197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313 C7A2121CA213E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>III82 DI<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I< 387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F8151980 9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803A313E3EA39F3A213B300191300 A61313EA1B1BEA0F1EA2EA0E0E13197F9816>87 D<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>89 D91 D93 D97 D101 D105 D110 D112 D<12035AA4EA7FFFB5FCA20007C7 FCA75BEB0380A2130713873803FF005BEA00F811177F9616>116 D<387F1FC038FF9FE0387F1F C0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279127BEA7F80 6CC7FC123C131B7F9116>121 D E /Fg 53 122 df<90380FE0FE90387FF3FF9039F87F8F8039 01E0FF1F000313FEEA07C091387E0F00023EC7FCA5B612F0A22607C03EC7FCB0393FF9FFE0A221 20809F1E>11 D13 D<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA2 5AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114 >40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01 E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F12 3B1203A21206A2120E120C12181270122008107C860F>44 DI<123812 7C12FEA3127C123807077C860F>I63 D<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F 48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E9E28>65 DI<903807FC0290 383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406127EA200FE1400 A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383FFF80903807FC00 1F1F7D9E26>IIII<903807FC0290383FFF0E9038FE03DE3903 F000FE4848133E4848131E485A48C7120EA2481406127EA200FE91C7FCA591387FFFE0A2007E90 3800FE00A2127F7EA26C7E6C7E6C7E3803F0013900FE03BE90383FFF1E903807FC06231F7D9E29 >III76 DIIIIII<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC 13F86CB4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813 F038FE01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140C A200E0140E00C01406A400001400B10007B512C0A21F1E7E9D24>IIIIII97 DIIIII<3801FC3C38 07FFFE380F07DEEA1E03003E13E0A5001E13C0380F0780EBFF00EA19FC0018C7FCA2121C381FFF 8014F06C13F8003F13FC387C007C0070133E00F0131EA30078133CA2383F01F8380FFFE0000113 00171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217EA00E>I107 DI<3AFE0FE0 3F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8FFEA227147D 932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D>I<48B4FC00 0713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F03807FFC000 01130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA6143E147E147C EB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I<39FFE07F80 A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13FF6DC7FCA2133EA2131C A219147F931C>I<38FFE1FFA2380F80706C6C5A6D5A3803E180EA01F36CB4C7FC137E133E133F 497E136FEBC7C0380183E0380381F0380701F8380E00FC39FF81FF80A219147F931C>120 D<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13FF6DC7FC A2133EA2131CA21318A2EA783012FC5BEAC0E0EAE1C0EA7F80001EC8FC191D7F931C>I E /Fh 28 86 df<1230127812F81278127005057C840D>46 D<130C131C13FCEA0FF813381200 A41370A613E0A6EA01C0A61203EA7FFE12FF0F1E7C9D17>49 D<133FEBFFC03801C1E0380300F0 00061378A2000F137C1380A2EB00781206C712F814F0130114E0EB03C0EB0780EB0F00131C5B5B 13C0485A380300601206001C13C05AEA7FFFB51280A2161E7E9D17>I<137F3801FFC0380383E0 EA0701EB00F05A1301A2000013E0A2EB03C0EB0780EB0F0013FE13F8130E7F1480EB03C0A3EA30 07127812F8A238F00F8000C01300EA601EEA383CEA1FF8EA07E0141F7D9D17>II<00031370EBFFF014E01480EBFE000006C7FC A6EA0C7CEA0DFF380F8780EA0E03380C01C0A2000013E0A4387003C012F0A21480EAC00700E013 00EA600EEA383CEA1FF0EA0FC0141F7D9D17>I<133FEBFF803801C1E0EA038038070070120EA3 14E0120FEB81C0EBC3803807F700EA03FC120148B4FC380F3F80EA1C0F383807C01303EA7001A2 12E0A3EB0380007013005BEA3C1CEA1FF8EA07E0141F7D9D17>56 D<137E48B4FC380383803807 01C0120E121E001C13E0123CA21278A41303A2383807C0A2EA1C0FEA1FFB380FE380EA0007A214 00130E1260EAF01C5B485A5BEA7FC06CC7FC131F7C9D17>I<14181438A21478147C14FCA2EB01 BCA2EB033C143EEB061EA2130CA21318141F497EA21360EB7FFF90B5FCEBC00F3901800780A2EA 0300A21206A2001F14C039FFC07FFCA21E207E9F22>65 D<0007B5FC15C039003C01E015F09038 7800F8A515F0EBF001EC03E0EC07C0EC0F809038FFFE00ECFF803901E007C0EC03E0A21401A215 F0D803C013E01403A2EC07C0A2EC0F800007EB3F00387FFFFEB512F01D1F7E9E20>I<903803F8 0890380FFE1890383F073890387801F83801F000D803C013F0000714705B48C7FC5A121E003E14 60123C007C1400A45AA415C01278EC0180127C003CEB0300A26C13065C6C6C5A3807E0703801FF C06C6CC7FC1D217B9F21>I<0007B5FC15E039003C01F0EC00F849137C153C151EA3151F5BA648 48131E153EA3153C157C4848137815F0A2EC01E0EC03C0EC0F800007EB3F00387FFFFCB512E020 1F7E9E23>I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801 E0701430A2151815301400485A1560A215E0EC01C014030007130F007FB51280B6FC1D1F7E9E1F >I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801E0701430 A491C7FC485AA61207EA7FFE12FF1D1F7E9E1E>I<903801FC0490380FFF0C90383F039C903878 00FC5BD803E013785B4848133848C7FC5A121E003E1430A2481400A45AA2EC7FFCA2EC01E01278 EC03C0127C123CA27E6C13073907800F803803E0393801FFF039003F80001E217B9F24>I<3A07 FFC7FFC0A23A003C007800A2495BA649485AA490B5FCA23901E003C0A64848485AA60007130F39 7FFCFFF8485A221F7E9E22>I<3807FFE0A238003C00A25BA65BA6485AA6485AA61207EAFFFCA2 131F7F9E10>II<3A07FFE0FFE0A23A003C003E001538495B5DEC 01804AC7FC14065C495A5C5CEBF1F013F3EBF778EA01EC13F8497E13E080A248487EA36E7EA26E 7E0007497E397FFC1FFC00FF133F231F7E9E23>I<3807FFF0A2D8003CC7FCA25BA65BA6485AA3 EC0180A2EC0300EA03C0A25C1406140E141E0007137E387FFFFCB5FC191F7E9E1C>II<3A07FC03FFC0A23A003E007C001538016F1330A3EB6780 A2EB63C001C35BA2EBC1E0A2EBC0F0A2D801805B1478A2143CA33903001F80A2140FA3481307D8 0F8090C7FC387FF00312FF221F7E9E22>II<0007 B5FC15C039003C03E0EC01F0EB780015F8A59038F001F0A215E0EC03C0EC0F809038FFFE004813 F801E0C7FCA5485AA61207EA7FFC12FF1D1F7E9E1F>I<3807FFFC14FF39003C07C0EC03E0EB78 0115F0A415E0EBF00315C0EC0780EC1F00EBFFFC14F03801E038143C141CA2141EA23803C03EA4 1506A20007140C397FFC1F1800FFEB0FF8C7EA03E01F207E9E21>82 DI< 001FB512F8A2381E03C000381438EB078012300070141812601538153038C00F0000001400A513 1EA65BA6137C381FFFF05A1D1F7B9E21>I<39FFFC7FF8A23907800F80EC0700380F0006A6001E 5BA6485BA600385BA35C003C5B121C381E0180D80F07C7FCEA07FCEA01F81D20799E22>I E /Fi 73 126 df<127012F8B012701200A5127012F8A31270051C779B18>33 D38 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812781270A212F05AA87E1270 A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C24799F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A2120113C012 0313801207EA0F00121E5A5A5A12600C247C9F18>II<136013 F0A7387FFFC0B512E0A26C13C03800F000A7136013147E9718>I<123C127E127FA3123F120F12 0E121E127C12F81270080C788518>II<127812FCA41278 0606778518>I48 DIII<131F5B5B137713F7EA01E713 C71203EA07871307120E121E123C1238127812F0B512F8A338000700A6EB7FF0EBFFF8EB7FF015 1C7F9B18>I<381FFF805AA20038C7FCA8EA3BFCEA3FFE7F383E0780381803C0380001E01300A2 126012F0130100E013C0EAF003387C0F80383FFF006C5AEA07F8131C7E9B18>I<137E48B4FC00 071380380FC3C0EA1F03123C383801800078C7FC1270A2EAF3F8EAEFFEB5FC38FE0F8038F803C0 EAF00114E01300A312701301007813C0EA3C03381E0F80380FFF006C5AEA03F8131C7E9B18>I< 12E0B512E0A338E003C0EB078038000F00130E131E131C133C5B1370A213F05BA212015BA31203 5BA7131D7E9C18>III<123C127EA4123C1200A81238127C127EA3123E120E121E123C12 7812F01260071A789318>59 D<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II<387F07 F038FF87F8387F07F0381C03C0EB078014005B131E5B133813785B121D7F121F13BC131CEA1E1E 130EEA1C0F7F1480130314C01301387F03F038FF87F8387F03F0151C7F9B18>75 DI<38FC01F8EAFE03A238 3B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<38 7E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D 130DA3EA7F07EAFF87EA7F03151C7F9B18>IIIII<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A290C7FC7E1278123FEA1FF0EA0FFE EA01FF38001F80EB03C0EB01E01300A2126012E0130100F013C0EAFC07B512801400EAE7FC131C 7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>I<38FF83FEA3381C 0070B2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA007C171C809B18>I<38FF07F8 A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA018C13DCA3EA00D813F8A21370151C 7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39DCA2001913C0A3138CEA1D8DA4000D 13801305EA0F07A2EA0E03151C7F9B18>I<387F8FE0139F138F380E0700120FEA070E138EEA03 9C13DCEA01F8A26C5AA2137013F07F120113DCEA039E138EEA070F7F000E13801303001E13C038 7F07F038FF8FF8387F07F0151C7F9B18>I<38FF07F8A3381C01C0EA1E03000E1380EA0F070007 1300A2EA038EA2EA01DCA213FC6C5AA21370A9EA01FC487E6C5A151C7F9B18>I91 D93 D95 D97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB00701478000E1338A5120F1478 1470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30383C0F00EA3807A5EA3C0F EA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801F038F00078481338A36C13 78007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<127E12FE127E120EA5133FEB FF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3FC171C809B18>II<12FEA3120EA5EB3F F0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB0780130314C038FFC7F8A3 151C7F9B18>107 DI<387DF1 F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBFBF80397F1F1F001914819318 >IIII<387F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8 EA7FFEB5FC6C5A15147F9318>114 DI<487E 1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E0013197F9818 >I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13FE3801FCFC1714809318>I<38 7F8FF000FF13F8007F13F0381E03C0000E1380A338070700A3EA038EA4EA01DCA3EA00F8A21370 15147F9318>I<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>120 D<387F8FF000FF13F8007F13F0380E01C0EB0380A21207EB0700A2EA03871386138EEA01CEA2EA 00CCA213DC1378A31370A313F05B1279EA7BC0EA7F806CC7FC121E151E7F9318>I 123 D<126012F0B3B012600424769F18>I<127CB47E7FEA07E01200AB7FEB7FC0EB3FE0A2EB7F C0EBF0005BAB1207B45A5B007CC7FC13247E9F18>I E /Fj 23 122 df<3801FFF0A238001E00 A35BA45BA45BA4485AA4485AA4485AA4EAFFF8A2141F7D9E12>73 D<48B4ECFFC0A2D8001F9038 01F800A2ED037801375C1506150CA20167EB19E0EB63801531A201C3495A15C3A2EC8183D80183 EB8780EC8307A214862703038C0FC7FCA21498EB01D80006EBF01EA214E0000E13C03AFFE1C3FF E0D9C1835B2A1F7D9E29>77 D<48B5128015E039001E00F01578153849133CA4491378A2157015 E0EBF001EC078090B5120014F8D801E0C7FCA4485AA4485AA4EAFFF85B1E1F7D9E1F>80 D97 D<137E48B4FC38038380EA0F07121E001C1300EA3C02 48C7FCA35AA5EA70021307EA381EEA1FF8EA07E011147C9315>99 D<1478EB03F814F0EB0070A3 14E0A4EB01C0A213F1EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133C EA707C3878FCC0EA3FCF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB03 00EA780EEA7FFC13E000F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I<14F8 EB01FCEB03BC143CEB07181400A2130EA53801FFE0A238001C00A35BA55BA65BA4485AA35B1233 127B00F3C7FC127E123C1629829F0E>III<136013F0A213E01300A7120FEA1F80123113C0EA6380A212C3EA0700A3120EA3EA1C30 1360A2EA38C01218EA1F80EA0F000C1F7D9E0E>I108 D<391E07C0F8393F1FE1FC3933 B8730E3963E07607EBC03CEB803800C7EB780E38070070A3000E495AA3ED3860261C01C013C0A2 ED718015313A3803803F00D81801131E23147D9325>I<381E07C0383F1FE03833B870EA63E0EB C038138000C71370EA0700A3000E13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018 147D931A>I<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB078014 00EA700F131EEA3838EA1FF0EA07C013147C9317>I<3803C1E03807E7F838067E3C380C7C1CEB 781E1370EA18E01200A33801C03CA314781203147014E0EBE3C038077F80EB1E0090C7FCA2120E A45AEAFFC0A2171D809317>I114 D<13FCEA03FEEA0707EA0E0FA2130EEA1E00EA0F8013 F0EA07F8EA03FCEA003E131E1270EAF01CA2EAE038EA6070EA3FE0EA1F8010147D9313>II<000F1360381F8070003113E013C0EA6380A238C381C0EA0701A3380E0380A314 8CEB0718A2130FEB1F303807F3F03803E1E016147D9318>I<380F01C0EA1F83003113E013C1EA 6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D9315>I< 38078780380FCFC03818F8E0EA3070EA6071A238C0E1C03800E000A3485AA30071136038F380C0 A238E3818038C7C300EA7CFEEA387C13147D9315>120 D<000F1360381F8070003113E013C0EA 6380A238C381C0EA0701A3380E0380A4EB0700A25B5BEA07FEEA03EEEA000EA25B12785BEA7070 EA60E0EA3FC06CC7FC141D7D9316>I E /Fk 40 122 df<1238127C12FEA3127C123807077C86 10>46 D<13181378EA01F812FFA21201B3A7387FFFE0A213207C9F1C>49 DII<14E013011303A21307130F131FA2 1337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2380007E0A7EBFF FEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018C7FCA4EA19FC EA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8386007E0003013 C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00F05A1478123C12 3E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF8383C1FFCEA7807EB 01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC00001130017207E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F 140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A >65 D67 DIII72 D77 DIII<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400 B4FC13E06CB4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA3 6C131EA26C133C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F 00781407007014030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127 >I97 DI< EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127EA2003E13306C13 60380FC0E03807FF803800FE0014167E9519>II< 13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E 003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B51280 380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC12 1CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB4 5A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C 247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78 FCEA7FF0EA1FC0102E83A311>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0 A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2 1A207E951F>I114 DI<13C0A41201A212031207120F12 1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF 14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E000713 0CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E >I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB 7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B20 7F951E>121 D E /Fl 66 123 df11 D<133FEBFF803803C1C0EA0703A2380E 018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C17>II<90383F03F09038FFCFF83903C0FC1C3907 01F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA030012 06120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C12 0C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380 A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<1270 12F012F8A212781218A31230A2127012601240050D7D840C>44 DI<12 7012F8A3127005057D840C>I<1303A213071306A2130E130C131C1318A213381330A213701360 A213E013C0A21201138012031300A25A1206A2120E120CA2121C1218A21238123012701260A212 E05AA210297E9E15>II<12035A123FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>I< EA07C0EA1FF0EA3878EA603C131E12F0EAF80FA312701200130E131E131C133C1378137013E0EA 01C0EA0380EA0700EA0E03120C1218EA3006EA7FFE12FFA2101C7E9B15>II<1260387FFF80A21400EA6003EAC006 5BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15>55 D57 D<127012F8A312701200A8127012F8A3127005 127D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012601240051A 7D910C>I63 D<1306130FA3497EA4EB33C0A3EB61E0A3EB C0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C1D7F9C1F>65 DI<90381F8080EBFFE13803F0333807801B380F000F00 1E1307001C1303123C5A1401127012F091C7FCA70070EB01801278A27E001CEB0300121E6C1306 6C6C5A3803F0383800FFF0EB1F80191E7E9C1E>IIII<90381F8080EBFFE13803F0333807801B380F000F001E 1307001C1303123C5A1401127012F091C7FCA5ECFFF0A20070EB07801278A27E121C121E7E3807 800F3803F0393800FFF090381FC0001C1E7E9C21>I<39FFF3FFC0A2390F003C00AAEBFFFCA2EB 003CAC39FFF3FFC0A21A1C7E9B1F>II77 DIIIII<3807E080EA1FF9EA3C1FEA7007 130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FFC61380130FEB03C0A2130112C0A3 00E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C17>I<007FB512C0A238780F03007013 010060130000E014E000C01460A400001400B03803FFFCA21B1C7F9B1E>I<3AFFE0FFE1FFA23A 1F001E007C6C1530143FA20180147000079038678060A32603C0E713C0ECC3C0A2D801E0EBC180 9038E181E1A3D800F3EBF3001400A2017B13F6017E137EA3013C133CA3011C133801181318281D 7F9B2B>87 D92 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C000 1C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFF E0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB 38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C 0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F 130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF 9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA 1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00 A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA 03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA2 6C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713 CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01 DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I< 38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A2 00C7C7FC127E123C131A7F9116>II E /Fm 18 118 df<1238127C12FEA3127C12381200A41238127C12FEA3127C123807127D910D>58 D63 D68 D73 D77 D97 D99 D101 D<3803F0F0380FFFF8383E1F 38383C0F30007C1380A4003C1300EA3E1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E048 13F0387801F838F00078A3007813F0383E03E0381FFFC03803FE00151B7F9118>103 D<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>105 D108 D<39FF1FC0FE90387FE3FF3A1FE1F70F809039 80FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE0381FE1F0EB80F8A213 00AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8 130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFFE0A2171A7F911B> I115 D<1203A35AA25AA2123FEAFFFCA2EA1F00A9130CA4 EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03F818127F 911B>I E /Fn 43 122 df<903901FF81FF011F01EF13C0903A7F80FF87E0D9FE01EB0FF03903 FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712F8A32707F001FCC7FCB3A33A7FFF1FFFE0 A32C2A7FA928>11 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB 007F13FEA317277BA622>III<14075C5C5C5C5CA25B 5B497E130F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C713 00A7017F13F8A31D277EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6 EB7FC0380FFFF8EB80FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300 007E137F007814806CEBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E0 4814C01580A215003870001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213 FEA41201A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE0701 0FEBFF8F90393FF007FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B 007F1503A3484891C7FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C 6C6C13F890393FF007F0010FB512C0010391C7FC9038003FF829297CA832>II73 D77 D III82 D<90387F80603903FFF0E0000F13FF381F807F383F001F003E1307007E1303127C00FC1301A214 007E7E6D130013F8EBFF806C13F814FE6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F 1407A200E013031401A37E15F06C13036C14E0B413079038E01FC090B5120000E05B38C01FF01D 297CA826>I<007FB712C0A39039807FC03FD87C00140700781503A20070150100F016E0A24815 00A5C71500B3A490B612E0A32B287EA730>I<48B47E000F13F0381F81FC486C7E147FA2EC3F80 A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F 83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<90 38FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80 FE001F5B380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14 F0121F397E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038 007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3 A3EAFFFEA30F2B7DAA14>105 DIII<3BFFC07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F90 3ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FF C07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B 7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A2 5DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26> I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C 13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FFC000C11300161B 7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438A73803F8700001 13F03800FFE0EB1F8015267FA51B>I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F0 3BFF803801FFF338003FC3211B7D9A26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEB FC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D 5A211B7F9A24>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485A A2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C 90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>121 D E /Fo 73 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7B612 80A23907007000B0387FE3FFA21D20809F1B>11 DII<90380F80F890387FE7FE9038E06E063901C0FC 0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0AC1260A21270A3 12301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C012607E7E121C12 0C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A213005A12065A121C12 185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A21238123012601240060E7C840D> 44 DI<127012F8A3127005057C840D>I<1303A213071306A2130E130C A2131C1318A213381330A213701360A213E013C0A212011380A312031300A25A1206A2120E120C A2121C1218A212381230A212701260A212E05AA2102D7DA117>IIIII<130EA2131E133EA2136E 13EE13CEEA018E1203130E1206120E120C121812381230126012E0B512F0A238000E00A7EBFFE0 A2141E7F9D17>II<137CEA01FEEA0783380E0380EA0C07121C3838030090C7FC12781270A2EAF3F8EAF7FEEA FC0E487EEB0380A200F013C0A51270A214801238EB0700121CEA0E1EEA07FCEA01F0121F7E9D17 >I<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207 A76CC7FC121F7D9D17>III<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A3 12701200AA127012F8A312781218A41230A21260A21240051D7C930D>I<007FB512E0B612F0C9 FCA8B612F06C14E01C0C7D9023>61 D65 DI<90381FC04090387FF0C03801F8393803C00D 38078007380F0003121E003E1301123C127C1400127812F81500A8007814C0127CA2123C003EEB 0180121E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D9F21>IIII<90380FC02090387FF8603901F81CE03803E00638078003380F0001121E14005A127C 1560127812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E38078003EA03E03901F80E6039 007FFC2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90B5FCA2EB800FAD39FFF8 FFF8A21D1F7E9E22>II76 DI<39FF807FF813C00007EB07809038E00300A2EA06F0A21378 133CA2131EA2130FA2EB078314C31303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0 801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA6003 12E01301A36CC7FCA21278127FEA3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A2130112C0A4 00E01380EAF00338F80700EAFE0EEACFFCEA81F812217D9F19>I<007FB512E0A238780F010070 130000601460A200E0147000C01430A400001400B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A2 3907800780EC0300B3A300031302EBC006A200015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22> I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800FC006A3913819E00ED803C0140CA214 393A01E030F018A33A00F0607830A3ECE07C903978C03C60A390393D801EC0A390383F000F6D5C A3010E6DC7FCA32C207F9E2F>87 D<397FF83FF8A23907C00F800003EB06003801E00EEBF00C00 005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C13078080130DEB1DF0EB18F8EB3878EB307CEB 603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FFE01FFEA21F1F7F9E22>I92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00E0000E13 F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>I III<133C13 FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FCA216207F 9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I<13E0EA01 F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C28829E0E >I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E1C131E13 0E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20809F0C>I< 390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0A223147F93 26>IIII<3803E180EA0FF9EA 1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA0003A7EB3FF8 A2151D7E9318>III<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318A5EA073013E0 EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01F116147F9319> I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370 A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14E00007EBE180EB 186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E147F9321>I<387F C7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA01C71387380303 80380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E13C01480A23807 0300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3007FC7FC123C15 1D7F9318>III E /Fp 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815FE489039001F FF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F01 3FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB 01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF00101 07903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC0113C0 EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5BA212FF AA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB3F8090 3A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fq 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D67 D97 D<903801FFC0011F13F8017F 13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03 FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB01 F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0 5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13 FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07 FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F 13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80 91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D A529>114 D116 D E /Fr 36 119 df38 D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA214371477146714C7A2EB01 87A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA0300A21206A2 121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E150FA24913 0E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A1538153CA2 48481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B512F015FC90 380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A216005D4848 130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68 D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748 C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C 130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D73 D<903807FFC04913809038003C00 A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F 801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14 0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E 7EA28139FFF80FFEA226227DA125>III<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001 311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C 1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378 4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0 15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I< 90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512 005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0 1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030 131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4 5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F 000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03 F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F 5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C 0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87 D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001 5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124> 89 D97 D<137E48B4FC3803C380EA0703 EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB 0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860 A21440EB18C038E01F8038600F0014237DA216>107 DI<381E0780383F1F E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0 123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803 C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863 B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70 07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2 EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13 30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2 130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3 80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I E /Fs 54 122 df35 D<127012F812FCA2127C12 0CA41218A21230A212601240060F7C840E>44 DI<127012F8A3127005 057C840E>I48 DIII<00101380EA1C07 381FFF005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012 F0A214C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C180380701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEA F60738FC038000F813C0130112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07 FCEA01F013227EA018>I56 DI<497E497EA3497EA3 497E130CA2EB1CF8EB1878A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A300 06EB01E0A3000EEB00F0001F130139FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038 F81C703801E0063903C003F03807800148C7FC121E003E1470123C127C15301278A212F81500A7 00781430A2127CA2003C1460123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB 0FE01C247DA223>II70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807 800048C71278121E15385AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27E A27E6C7E6C6C13F83801F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFF A239078001E0AD90B5FCA2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6 127012F8A3EAF01EEA601CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F0 15C05D4AC7FC14065C5C14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E138080 80A26E7E8114036E7EA26E7E4A7E3AFFFC07FF80A221227EA126>III<39FF800FFF13C00007EB01F89038E000607F1206 1378A27F133E131E7FA2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A2 14031401A2381F8000EAFFF0156020227EA125>III82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A3 6C1300A21278127FEA3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C 136014E06C13C0EAF80138EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C078000 70143800601418A200E0141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03F FC07FEA23B0F0007C001F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701 E00C781300A33A00F0183C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3 010F14E0EC8003A36D486C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F8 4813F0EB01E0EAC00314C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C0 12071380000F13071300001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E> 90 D97 D<120E12FEA2121E120EAAEB1F80EB7FE038 0FC0F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017 237FA21B>II<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C 01EA38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>I< EA01FCEA07FF380F0780381C03C0EA3801007813E0EA7000B5FCA200F0C7FCA5127814607E6C13 C0380F83803807FF00EA00FC13157F9416>I<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA 0700B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380 A5001C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F03870 0070481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA212 1E120EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E 123E121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>II<120E12FE A2121E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480 EB03C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237F A20E>I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFF E3FF8FFEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FF A218157F941B>II<380E1F8038 FE7FE038FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F00 90C7FCA8EAFFE0A2171F7F941B>I114 DI<1206A5120EA3121E123EEA FFF8A2EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E 00000E1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E 1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8F F9FFA2391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C 3901D81980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E 1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0 C012F1EAF380007FC7FC123E171F7F941A>121 D E /Ft 20 118 df 45 D68 D 73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8 A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27 07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38> 109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E 3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0 38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813 705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2 141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312 03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB 19>II E /Fu 5 85 df<1630167016F0A21501A21503A2150715 0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038 3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314 7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201 1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7 121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713 0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213 7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092 380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201 3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0 C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13 0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31 74B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 695 a Fu(D)26 b(R)g(A)f(F)h(T)225 787 y Ft(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685 981 y Fs(Scott)c(Berryman,)e Fr(Y)l(ale)19 b(Univ)700 1039 y Fs(James)c(Co)o(wnie,)h Fr(Meiko)i(Ltd)474 1098 y Fs(Jac)o(k)e(Dongarra,)i Fr(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1156 y Fs(Al)f(Geist,)f Fr(ORNL)795 1214 y Fs(Bill)f(Gropp,)j Fr(ANL)767 1272 y Fs(Rolf)f(Hemp)q(el,)e Fr(GMD)762 1330 y Fs(Bob)i(Knigh)o(ten,)g Fr(Intel)786 1388 y Fs(Rust)o(y)g(Lusk,)h Fr(ANL)614 1446 y Fs(Stev)o(e)e(Otto,)h Fr(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578 1504 y Fs(T)l(on)o(y)g(Skjellum,)c Fr(Missisippi)j(State)j(Univ)653 1562 y Fs(Marc)d(Snir,)g Fr(IBM)h(T.)g(J.)g(Watson)744 1621 y Fs(Da)o(vid)f(W)l(alk)o(er,)f Fr(ORNL)626 1679 y Fs(Stev)o(e)g(Zenith,)g Fr(Kuck)k(&)f(Asso)n(ciates)811 1803 y Fs(Marc)o(h)e(25,)g(1993)87 1861 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g (under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192 1919 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology) f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1977 y(Agreemen)o(t)f(No.)21 b(CCR-8809615.)p eop %%Page: 1 2 bop 75 357 a Fq(Chapter)32 b(1)75 565 y Fp(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)876 788 y Fo(Marc)14 b(Snir)656 845 y(William)j(Gropp)e (and)g(Ewing)h(Lusk)75 992 y Fn(1.1)70 b(In)n(tro)r(duction)75 1095 y Fo(This)19 b(section)f(is)h(a)f(draft)f(of)h(the)g(curren)o(t)g(prop)q (osal)g(for)g(p)q(oin)o(t-to-p)q(oin)o(t)g(comm)o(unication.)30 b(It)18 b(do)q(es)75 1151 y(not)d(y)o(et)g(include)i(a)e(description)i(of)d (the)i(F)l(ortran)e(77)g(and)i(C)f(bindings.)166 1208 y(I)g(ha)o(v)o(e)f (tried)i(to)e(indicate,)i(wherev)o(er)e(appropriate,)h(gaps)f(and)h(unresolv) o(ed)h(issues,)f(using)g(small)75 1265 y(t)o(yp)q(e.)166 1398 y Fm(Discussion:)45 b Fl(The)17 b(follo)o(wing)d(sections)k(of)e(the)h(in)o (tro)q(duction)g(con)o(tain)f(general)h(information)d(on)i(the)75 1448 y(design)g(of)f(MPI)h(pro)q(cedures.)27 b(The)16 b(material)e(should)h (b)q(e)i(mo)o(v)o(ed)d(to)i(a)g(general)g(in)o(tro)q(duction)f(for)h(the)g (en)o(tire)75 1497 y(do)q(cumen)o(t.)21 b(The)15 b(actual)g(binding)f(of)g (these)i(ob)r(jects)g(in)f(F)m(ortran)g(and)f(C)h(will)f(b)q(e)h(discussed)i (in)d(the)i(language)75 1547 y(binding)d(sub)q(committee.)75 1777 y Fn(1.2)70 b(Data)23 b(T)n(yp)r(es)75 1881 y Fk(1.2.1)55 b(Handle)75 1968 y Fo(MPI)16 b(pro)q(cedures)h(use)f(at)f(v)m(arious)h (places)h Fj(hand)r(les)p Fo(.)22 b(Handles)17 b(are)e(used)i(to)e(access)h (opaque)g(ob)s(jects.)75 2025 y(Suc)o(h)j(ob)s(ject)e(can)h(b)q(e)h(created,) g(up)q(dated)f(and)h(destro)o(y)o(ed)e(only)i(b)o(y)f(b)o(y)g(calling)i (suitable)f(MPI)f(pro-)75 2081 y(cedures,)i(and)f(pro)o(viding)g(the)g (handle)h(as)e(parameter.)29 b(Opaque)20 b(ob)s(jects)e(hide)i(from)d(the)i (user)g(the)75 2138 y(in)o(ternal)e(represen)o(tation)e(used)i(for)e(v)m (arious)h(MPI)f(ob)s(jects,)g(th)o(us)h(allo)o(wing)g(to)f(ha)o(v)o(e)h (similar)g(calls)h(in)75 2194 y(C)i(and)g(F)l(ortran,)g(allo)o(wing)h(to)f(o) o(v)o(ercome)f(problems)i(with)f(the)g(t)o(yping)h(rules)g(in)g(these)f (languages,)75 2251 y(and)c(allo)o(wing)g(for)g(future)f(extension)i(of)e (their)h(functionalit)o(y)l(.)21 b(The)15 b(mec)o(hanism)h(for)e(opaque)h(ob) s(jects)75 2307 y(used)h(here)f(lo)q(osely)i(follo)o(ws)e(the)g(POSIX)h(F)l (ortran)e(binding)k(standard.)166 2364 y(An)h(opaque)h(ob)s(ject)e(can)i(b)q (e)f Fj(p)n(ersistent)f Fo(or)h Fj(ephemer)n(al)p Fo(.)32 b(A)19 b(p)q(ersisten)o(t)h(ob)s(ject)e(p)q(ersists)i(un)o(til)75 2421 y(destro)o(y)o(ed)c(b)o(y)h(an)g(explicit)i(op)q(eration.)25 b(An)17 b(ephemeral)h(ob)s(ject)f(is)g(go)q(o)q(d)g(for)f(a)g(single)j(use;)e (th)o(us)g(an)75 2477 y(ephemeral)c(ob)s(ject)d(asso)q(ciated)i(with)g(a)f (comm)o(unication)h(op)q(eration)g(disapp)q(ears)g(once)g(this)g(op)q (eration)75 2534 y(is)g(completed)g(\(or)f(once)g(this)h(ob)s(ject)f(is)h (not)f(needed)i(an)o(ymore)d(for)h(the)h(completion)g(of)f(the)g(op)q (eration\).)166 2591 y(Opaque)21 b(ob)s(ject)e(are)h(created)g(b)o(y)g(calls) h(that)f(are)f(sp)q(eci\014c)j(to)e(eac)o(h)g(ob)s(ject)f(t)o(yp)q(e.)35 b(They)20 b(are)75 2647 y(destro)o(y)o(ede)g(b)o(y)h(a)g(call)g(to)g Fi(MPI)p 642 2647 15 2 v 16 w(FREE)p Fo(.)f(Additional)j(MPI)d(functions)i (are)e(a)o(v)m(ailable)j(to)d(access)h(and)75 2704 y(up)q(date)16 b(sp)q(eci\014c)h(opaque)e(ob)s(jects.)964 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fo(2)631 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)166 45 y Fg(MPI)p 275 45 16 2 v 18 w(FREE\(handle\))75 187 y(IN)h(handle)24 b Fo(handle)16 b(to)f(ob)s(ject)166 294 y(Mark)e(the)g(handle)i(for)e(destruction.)20 b(The)14 b(ob)s(ject)f(can)h (destro)o(y)o(ed)f(as)g(so)q(on)g(as)h(there)f(there)h(is)g(no)75 351 y(p)q(ending)k(op)q(eration)e(that)f(is)h(using)h(this)f(ob)s(ject.)22 b(This)16 b(is)g(equiv)m(alen)o(t)i(to)d(c)o(hanging)i(the)e(p)q(ersistence) 75 407 y(of)h(the)h(handle)h(from)e Fi(persistent)f Fo(to)h Fi(ephemeral)p Fo(.)23 b(F)l(or)16 b(example,)i(if)f(a)f(comm)o(unication)h (ob)s(ject)f(is)75 463 y(freed,)e(after)e(the)i(comm)o(unication)g(started)f (but)g(b)q(efore)h(it)g(completed,)g(then)g(the)g(p)q(ending)h(op)q(eration) 75 520 y(is)i(not)e(a\013ected;)h(one)g(can)h(use)f(the)g(handle)i(in)f (calls)g(to)e Fi(MPI)p 1147 520 15 2 v 17 w(WAIT)g Fo(or)h Fi(MPI)p 1404 520 V 17 w(STATUS)p Fo(.)f(Ho)o(w)o(ev)o(er,)g(once)75 576 y(the)e(op)q(eration)h(completes,)g(it)g(is)g(erroneous)f(to)g(reuse)g (the)h(freed)g(handle)g(for)f(a)g(new)h(comm)o(unication.)166 712 y Fm(Discussion:)166 765 y Fl(An)f(alternativ)o(e)f(design)g(p)q(oin)o(t) g(w)o(as)h(discussed,)h(whereb)o(y)f(it)f(is)h(the)g(user)g(resp)q(onsibilit) o(y)f(to)h(free)g(a)f(handle)75 815 y(only)e(if)h(there)h(are)g(no)f(p)q (ending)g(op)q(erations)h(on)f(this)g(handle.)17 b(The)12 b(curren)o(t)g (design)g(w)o(as)f(deemed)g(more)f(elegan)o(t.)75 865 y(Note)17 b(that)g(it)f(is)h(not)f(necessary)j(to)d(destro)o(y)i(the)f(handle)f(as)h (so)q(on)g(as)f(it)h(is)f(not)h(needed)h(an)o(ymore.)25 b(A)17 b(v)n(alid)75 915 y(implemen)o(tatio)o(n)11 b(ma)o(y)h(just)i(mark)f(freed)h (handles)h(and)e(garbage)h(collect)g(them)f(p)q(erio)q(dically)m(.)166 968 y(On)18 b(the)h(other)g(hand,)g(some)e(ob)r(jects,)j(suc)o(h)f(as)f (bu\013er)i(descriptor)f(ob)r(jects,)h(can)f(b)q(e)f(used)h(b)o(y)f(sev)o (eral)75 1017 y(distinct)c(concurren)o(t)i(comm)o(unicati)o(on)11 b(op)q(erations.)18 b(Th)o(us,)c(to)g(\014nd)g(out)g(whether)h(suc)o(h)f(ob)r (ject)h(can)f(b)q(e)h(freed,)75 1067 y(one)e(w)o(ould)f(need)i(a)e(reference) k(coun)o(t)d(of)f(p)q(ending)h(op)q(erations.)18 b(This)12 b(complication)f(w)o(as)i(not)f(observ)o(ed)i(at)f(our)75 1117 y(last)h(meeting,)e(and)i(ma)o(y)e(b)q(e)i(a)g(go)q(o)q(d)f(argumen)o(t)g(to) h(reconsider)h(the)g(curren)o(t)g(de\014nition)f(of)f Ff(MPI)p 1635 1117 14 2 v 15 w(FREE)p Fl(.)166 1295 y Fg(MPI)p 275 1295 16 2 v 18 w(ASSOCIA)l(TED\(handle,)18 b(t)o(yp)q(e\))75 1437 y(IN)f(handle)24 b Fo(handle)16 b(to)f(ob)s(ject)75 1544 y Fg(OUT)j(t)o(yp)q(e)23 b Fo(state)166 1651 y(Returns)17 b(the)g(t)o(yp)q(e)g (of)f(the)h(ob)s(ject)g(the)g(handle)h(is)f(curren)o(tly)g(asso)q(ciated)g (with,)h(if)f(suc)o(h)g(exists.)75 1707 y(Returns)d(the)f(sp)q(ecial)i(t)o (yp)q(e)e Fi(MPI)p 643 1707 15 2 v 17 w(NULL)f Fo(if)i(the)f(handle)h(is)g (not)f(curren)o(tly)g(asso)q(ciated)h(with)f(an)o(y)g(ob)s(ject.)166 1767 y(MPI)19 b(ma)o(y)g(pro)o(vide)h(prede\014ned)h(opaque)f(ob)s(jects)f (and)g(prede\014ned,)j(static)d(handles)i(to)e(these)75 1823 y(ob)s(jects.)g(Suc)o(h)d(ob)s(jects)f(ma)o(y)f(not)h(b)q(e)h(destro)o(y)o (ed.)75 1963 y Fk(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 2055 y Fo(An)j(MPI)f(call)i(ma)o(y)e(need)i(a)e(parameter)g(that)g(is)h(an)f Fj(arr)n(ay)i(of)f(hand)r(les)p Fo(.)39 b(The)22 b(arra)o(y-of-handles)75 2112 y(ob)s(ject)16 b(is)g(not)g(self-delimitin)q(g;)j(it)d(do)q(es)h(not)f (carry)f(information)i(on)f(the)g(n)o(um)o(b)q(er)g(of)g(en)o(tries)h(in)g (the)75 2168 y(arra)o(y)l(.)29 b(This)19 b(information)g(has)f(to)g(b)q(e)h (pro)o(vided)h(b)o(y)e(an)h(additional)h Fi(len)e Fo(parameter,)g(whenev)o (er)h(a)75 2225 y(list-of-handles)e(parameter)e(is)g(used.)166 2361 y Fm(Discussion:)166 2414 y Fl(The)i(reason)h(for)f(not)g(ha)o(ving)f (self-delimiting)e(arra)o(ys)k(is)f(that)g(records)h(are)g(not)f (\014rst-class)h(citizens)g(in)75 2463 y(F)m(ortran)13 b(and)g(that)g(ha)o (ving)f(an)h(arra)o(y)g(where)h(one)f(en)o(try)h(enco)q(des)h(length,)e (whereas)h(the)g(other)f(en)o(tries)i(enco)q(de)75 2513 y(handles)j(w)o(as)f (deemed)g(to)g(b)q(e)h(to)q(o)f(error-prone.)30 b(This)17 b(line)g(of)g (argumen)o(tation)e(really)i(implies)e(that)j(w)o(e)f(do)75 2563 y(not)e(view)g(arra)o(y-of-handles)g(as)g(an)g(opaque)g(ob)r(ject,)h (and)f(that)h(w)o(e)f(exp)q(ect)i(the)f(user)g(to)g(build)e(these)j(ob)r (jects)75 2613 y(explicitly)m(,)12 b(not)i(via)f(MPI)h(calls.)p eop %%Page: 3 4 bop 75 -100 a Fh(1.3.)34 b(PR)o(OCESSES)1403 b Fo(3)75 45 y Fk(1.2.3)55 b(State)75 133 y Fo(MPI)15 b(pro)q(cedures)h(use)g(at)e(v)m (arious)i(places)g(argumen)o(ts)f(with)g Fj(state)g Fo(t)o(yp)q(es.)20 b(The)c(v)m(alues)g(of)f(suc)o(h)h(data)75 190 y(t)o(yp)q(e)i(are)h(all)g (iden)o(ti\014ed)h(b)o(y)f(names,)f(and)h(no)f(op)q(eration)h(is)g(de\014ned) h(on)e(them.)29 b(F)l(or)18 b(example,)i(the)75 246 y Fi(MPI)p 150 246 15 2 v 17 w(APPEND)14 b Fo(routine)i(has)f(a)g(state)f(t)o(yp)q(e)h (parameter)g(with)g(v)m(alues)i Fi(MPI)p 1343 246 V 16 w(INT,)24 b(MPI)p 1551 246 V 17 w(REAL,)14 b Fo(etc.)75 376 y Fk(1.2.4)55 b(Named)18 b(constan)n(ts)75 464 y Fo(MPI)11 b(pro)q(cedures)h(sometimes)f (assign)g(a)g(sp)q(ecial)i(meaning)f(to)e(a)h(sp)q(ecial)i(v)m(alue)f(of)f(a) g(basic)g(t)o(yp)q(e)g(param-)75 520 y(eter;)17 b(e.g.)24 b Fi(tag)17 b Fo(is)g(an)g(in)o(teger)g(v)m(alued)h(parameter)e(of)g(p)q(oin)o (t-to-p)q(oin)o(t)i(comm)o(unication)f(op)q(erations,)75 577 y(with)f(a)f(sp)q(ecial)i Fi(DONTCARE)d Fo(v)m(alue.)22 b(Suc)o(h)16 b(parameters)e(will)j(ha)o(v)o(e)e(a)g(range)g(of)g(regular)h(v)m(alues,)g (whic)o(h)75 633 y(is)h(a)f(prop)q(er)h(subrange)f(of)g(the)h(range)f(of)g(v) m(alues)h(of)f(the)h(corresp)q(onding)g(basic)g(t)o(yp)q(e;)g(sp)q(ecial)h(v) m(alues)75 690 y(\(suc)o(h)e(as)g(DONTCARE\))g(will)i(b)q(e)f(outside)g(the)g (regular)f(range.)23 b(The)16 b(range)g(of)g(regular)g(v)m(alues)i(can)75 746 y(b)q(e)d(queried,)g(and)f(sometimes)g(set,)g(using)h(en)o(vironmen)o(t)f (inquiry)i(or)e(en)o(vironmen)o(t)g(setting)g(functions)75 803 y(\(Section)i Fg(??)p Fo(\).)166 937 y Fm(Discussion:)166 988 y Fl(Need)f(to)f(agree)g(on)g(a)f(language)g(mec)o(hanism)f(for)h(named)g (constan)o(ts.)166 1039 y(Implemen)o(ters)h(should)g(detect,)j(whenev)o(er)f (p)q(ossible,)f(illegal)e(uses)j(of)e(\\sp)q(ecial)h(v)n(alues".)21 b(Th)o(us,)15 b(the)g(use)75 1089 y(of)e(the)i Ff(DONTCARE)d Fl(v)n(alue)h(to)h(tag)f(a)h(message)g(sen)o(t)g(will)f(b)q(e)h(\015agged)g (as)g(an)g(error.)75 1301 y Fk(1.2.5)55 b(Choice)75 1389 y Fo(MPI)16 b(functions)g(sometimes)g(use)g(parameters)f(with)h(a)g Fj(choic)n(e)f Fo(\(or)g(union\))i(data)e(t)o(yp)q(e.)21 b(I.e.,)16 b(distinct)75 1445 y(calls)i(to)e(the)h(same)f(routine)h(ma)o(y)f(pass)h(b)o (y)f(reference)i(actual)e(parameters)g(of)g(di\013eren)o(t)h(t)o(yp)q(es.)25 b(The)75 1502 y(mec)o(hanism)16 b(for)e(pro)o(viding)j(suc)o(h)e(parameters)f (will)j(di\013er)f(from)e(language)i(to)e(language.)166 1636 y Fm(Discussion:)166 1687 y Fl(The)j(F)m(ortran)g(77)g(standard)g(sp)q (eci\014es)i(that)e(the)h(t)o(yp)q(e)f(of)f(actual)h(argumen)o(ts)f(need)i (to)f(agree)h(with)e(the)75 1737 y(t)o(yp)q(e)f(of)f(dumm)o(y)d(argumen)o (ts;)i(no)i(construct)h(equiv)n(alen)o(t)d(to)i(C)f(p)q(oin)o(ters)h(is)f(a)o (v)n(ailable.)j(Th)o(us,)e(it)f(w)o(ould)f(seem)75 1786 y(that)19 b(there)i(is)e(no)g(standard)h(conforming)d(mec)o(hanism)g(to)i(supp)q(ort)h (c)o(hoice)f(parameters.)35 b(Ho)o(w)o(ev)o(er,)20 b(most)75 1836 y(F)m(ortran)g(compiler)e(either)j(don't)e(c)o(hec)o(k)i(t)o(yp)q(e)f (consistency)h(of)e(calls)h(to)f(external)i(routines,)g(or)f(supp)q(ort)g(a) 75 1886 y(sp)q(ecial)d(mec)o(hanism)d(to)j(link)e(foreign)h(\(e.g.,)g(C\))h (routines.)27 b(I)17 b(suggest)g(that)g(w)o(e)g(accept)g(this)g(nonconformit) o(y)75 1936 y(with)12 b(F)m(ortran)h(77)f(standard.)19 b(I.e.,)12 b(w)o(e)h(accept)h(that)f(the)g(same)f(routine)h(ma)o(y)e(b)q(e)i(passed)h (an)f(actual)f(parameter)75 1986 y(of)h(a)h(di\013eren)o(t)h(t)o(yp)q(e)f(at) g(distinct)g(calls.)166 2037 y(Generic)j(routines)g(can)f(b)q(e)h(used)h(in)d (F)m(ortran)i(90)e(to)h(pro)o(vide)h(a)f(standard)g(conforming)f(solution.)24 b(This)75 2087 y(solution)13 b(will)f(b)q(e)j(consisten)o(t)g(with)f(our)f (nonstandard)i(conforming)c(F)m(ortran)j(77)f(solution.)75 2320 y Fn(1.3)70 b(Pro)r(cesses)75 2507 y Fm(Discussion:)15 b Fl(This)f(material)e(b)q(elongs)i(to)g(an)f(in)o(tro)q(duction)h(or)g(en)o (vironmen)o(t)f(section)166 2647 y Fo(An)i(MPI)f(program)g(is)h(executed)h(b) o(y)e(sev)o(eral)h(autonomous)f(pro)q(cesses)h(that)e(execute)j(eac)o(h)e (their)75 2704 y(o)o(wn)19 b(co)q(de,)h(in)g(a)e(MIMD)h(st)o(yle.)31 b(The)19 b(co)q(des)h(executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)g(need)h(not)f (b)q(e)g(iden)o(tical.)p eop %%Page: 4 5 bop 75 -100 a Fo(4)631 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 45 y Fo(The)k(pro)q(cesses)g(comm)o(unicate)g(via)g (calls)g(to)f(MPI)h(comm)o(unication)g(primitiv)o(es.)35 b(T)o(ypically)l(,) 22 b(eac)o(h)75 102 y(pro)q(cessor)17 b(executes)i(in)f(its)g(o)o(wn)f (address)g(space,)h(although)g(shared-memory)g(implemen)o(tations)g(of)75 158 y(MPI)12 b(are)g(p)q(ossible.)21 b(This)13 b(do)q(cumen)o(t)f(sp)q (eci\014es)i(the)f(b)q(eha)o(vior)f(of)g(a)g(parallel)i(program)d(assuming)i (that)75 214 y(only)h(MPI)g(calls)h(are)e(used)i(for)e(comm)o(unication.)20 b(The)14 b(in)o(teraction)g(of)f(an)h(MPI)g(program)e(with)i(other)75 271 y(p)q(ossible)i(means)e(of)g(comm)o(unication)g(\(e.g.,)f(shared)h (memory\))g(is)g(not)g(sp)q(eci\014ed.)22 b(In)14 b(particular,)h(it)f(is)75 327 y(assumed)h(that)g(message)g(bu\013ers)g(at)f(distinct)j(pro)q(cessors)e (are)f(disjoin)o(t.)166 386 y(MPI)d(do)q(es)g(not)g(sp)q(ecify)h(the)f (execution)h(mo)q(del)g(for)e(eac)o(h)h(pro)q(cess.)19 b(A)11 b(pro)q(cess)g(can)h(b)q(e)f(sequen)o(tial,)75 442 y(or)h(can)g(b)q(e)h(m)o (ultithreaded,)g(with)g(threads)f(p)q(ossibly)h(executing)h(concurren)o(tly)l (.)19 b(Care)12 b(has)g(b)q(een)h(tak)o(en)75 499 y(to)i(mak)o(e)f(MPI)h (\\thread-safe",)f(b)o(y)i(a)o(v)o(oiding)f(the)g(use)h(of)f(implicit)i (global)f(states.)166 557 y(The)d(initial)h(allo)q(cation)g(of)e(pro)q (cesses)g(to)g(an)g(MPI)h(computation)f(and)h(their)g(binding)h(to)e(ph)o (ysical)75 614 y(pro)q(cessors)17 b(is)i(not)e(sp)q(eci\014ed)j(b)o(y)d(the)h (program)f(itself.)28 b(It)18 b(is)g(exp)q(ected)h(that)e(v)o(endors)g(will)i (pro)o(vide)75 670 y(mec)o(hanisms)g(to)g(do)f(so)h(either)h(at)e(load)h (time)g(or)f(at)h(run)g(time.)31 b(Suc)o(h)20 b(mec)o(hanisms)f(will)i(allo)o (w)e(to)75 727 y(sp)q(ecify)i(the)f(initial)i(n)o(um)o(b)q(er)e(of)f (required)i(pro)q(cesses,)g(the)f(co)q(de)g(to)f(b)q(e)i(executed)f(b)o(y)g (eac)o(h)g(initial)75 783 y(pro)q(cess,)j(and)f(the)g(allo)q(cation)g(of)g (pro)q(cesses)g(to)f(pro)q(cessors.)38 b(Also,)24 b(the)d(curren)o(t)h(prop)q (osal)g(do)q(es)75 839 y(not)16 b(pro)o(vide)h(for)f(dynamic)h(creation)f(or) g(deletion)i(of)e(pro)q(cesses)h(during)g(program)e(execution)j(\(total)75 896 y(n)o(um)o(b)q(er)e(of)e(pro)q(cesses)i(is)g(\014xed\),)f(although)g(it)h (is)g(in)o(tended)g(to)f(b)q(e)h(consisten)o(t)f(with)h(suc)o(h)g(extension.) 75 952 y(Finally)l(,)i(the)e(curren)o(t)g(prop)q(osal)g(do)q(es)g(not)g(sp)q (ecify)h(a)f(naming)g(sc)o(heme)h(for)e(pro)q(cesses.)23 b(W)l(e)16 b(prop)q(ose)75 1009 y(to)g(alw)o(a)o(ys)g(iden)o(tify)i(pro)q(cesses)f (according)g(to)f(their)h(relativ)o(e)g(rank)f(in)i(a)e(con)o(text)g (\(group\),)g(so)g(that,)75 1065 y(e\013ectiv)o(ely)l(,)g(pro)q(cesses)g(are) e(iden)o(ti\014ed)k(b)o(y)d(consecutiv)o(e)h(in)o(tegers.)75 1198 y Fk(1.3.1)55 b(Error)18 b(Handling)75 1371 y Fm(Discussion:)51 b Fl(Material)18 b(of)f(this)i(section)g(should)f(b)q(e)g(mo)o(v)o(ed)f(to)h (a)g(general)g(in)o(tro)q(duction)g(section.)32 b(This)75 1427 y(material)12 b(has)i(not)g(y)o(et)g(b)q(een)h(app)q(oriv)o(ed)f(b)o(y)f (MPIF)166 1568 y Fo(MPI)18 b(pro)o(vides)g(the)h(user)f(with)g(reliable)i (message)e(transmission:)26 b(A)18 b(message)f(sen)o(t)h(is)h(alw)o(a)o(ys)75 1625 y(receiv)o(ed)f(correctly)l(,)f(and)h(the)f(user)g(do)q(es)g(not)f(need) i(to)e(c)o(hec)o(k)i(for)e(transmission)h(errors,)f(time-outs,)75 1681 y(or)i(other)g(error)f(conditions.)31 b(In)19 b(other)f(w)o(ords,)g(MPI) g(do)q(es)h(not)f(pro)o(vide)h(mec)o(hanisms)g(for)f(dealing)75 1738 y(with)i(failures)h(in)g(the)f(comm)o(unication)h(system.)34 b(Where)20 b(the)g(MPI)g(implemen)o(tation)h(is)g(built)g(on)75 1794 y(an)h(unreliable)i(underlying)g(mec)o(hanism,)f(then)g(it)f(is)g(the)g (job)f(of)h(the)g(implemen)o(ter)h(of)e(the)h(MPI)75 1851 y(subsystem)16 b(to)g(insulate)i(the)e(user)h(from)f(this)h(unreliabilit)o(y)l(,)i(or)d(to)g (re\015ect)h(unreco)o(v)o(erable)g(errors)e(as)75 1907 y(global)20 b(program)e(failures.)34 b(Similarly)21 b(MPI)f(itself)g(pro)o(vides)g(no)f (mec)o(hanisms)h(for)f(handling)i(no)q(de)75 1964 y(failures.)166 2022 y(Of)16 b(course,)h(MPI)f(programs)f(ma)o(y)h(still)i(b)q(e)f (erroneous.)23 b(A)16 b Fg(program)i(error)d Fo(can)i(o)q(ccur)f(when)75 2079 y(an)i(MPI)f(call)i(is)f(called)h(with)f(an)g(incorrect)g(parameter)f (\(non-existing)h(destination)h(in)f(a)g(send)g(op-)75 2135 y(eration,)i(bu\013er)g(to)q(o)f(small)h(in)h(a)e(receiv)o(e)i(op)q(eration,) f(etc.\))33 b(This)20 b(t)o(yp)q(e)g(of)f(error)g(w)o(ould)h(o)q(ccur)g(in)75 2191 y(an)o(y)e(implemen)o(tation.)29 b(In)19 b(addition,)g(a)f Fg(resource)i(error)c Fo(ma)o(y)i(o)q(ccur)g(when)g(a)g(program)f(exceeds)75 2248 y(the)k(amoun)o(t)f(of)g(a)o(v)m(ailable)i(system)f(resources)f(\(n)o (um)o(b)q(er)h(of)f(p)q(ending)j(messages,)e(system)f(bu\013ers,)75 2304 y(etc.\).)29 b(The)19 b(o)q(ccurrence)g(of)f(this)h(t)o(yp)q(e)g(of)f (error)f(dep)q(ends)j(on)f(the)f(amoun)o(t)g(of)g(a)o(v)m(ailable)i (resources)75 2361 y(in)d(the)f(system)g(and)g(the)g(resource)g(allo)q (cation)h(mec)o(hanism)g(used;)g(this)f(ma)o(y)f(di\013er)i(from)e(system)h (to)75 2417 y(system.)j(The)13 b(recommended)i(implemen)o(tation)f(pro\014le) h(pro)o(vides)e(sev)o(eral)h(mec)o(hanisms)g(to)f(alleviate)75 2474 y(the)19 b(p)q(ortabilit)o(y)h(problem)g(this)f(represen)o(ts.)31 b(One)20 b(can)f(also)g(write)g Fg(safe)g Fo(programs,)f(that)h(are)f(not)75 2530 y(sub)s(ject)d(to)g(resource)g(errors.)166 2589 y(All)f(MPI)f(pro)q (cedure)h(calls)g(return)f(an)g(error)f(parameter)g(that)g(indicates)j (successful)f(completion)75 2645 y(of)h(the)g(op)q(eration,)g(or)g(the)g (error)g(condition)h(that)f(o)q(ccurred,)g(otherwise.)166 2704 y(The)21 b(recommended)h(implemen)o(tation)g(pro\014le)g(in)g(a)f(POSIX)h(en) o(vironmen)o(t)f(is)h(for)e(an)o(y)h(MPI)p eop %%Page: 5 6 bop 75 -100 a Fh(1.4.)34 b(MESSA)o(GES)1423 b Fo(5)75 45 y(routine)14 b(that)f(encoun)o(ters)h(a)g(reco)o(v)o(erable)f(error)g(to)h(generate)f(an)h Fj(MPI)g(err)n(or)h(signal)p Fo(,)d(using)j(a)e(sp)q(ecial)75 102 y(signal)20 b(v)m(alue.)31 b(The)19 b(default)h(handler)g(for)e(this)h (signal)h(terminates)e(the)h(execution)h(of)e(all)i(in)o(v)o(olv)o(ed)75 158 y(pro)q(cesses,)i(with)f(a)f(suitable)i(error)e(message)g(b)q(eing)i (returned)f(to)f(the)g(user.)37 b(Ho)o(w)o(ev)o(er,)20 b(the)h(user)75 214 y(can)d(pro)o(vide)h(his)g(or)f(her)h(o)o(wn)f(signal)h(handling)h (routine.)30 b(In)19 b(particular,)g(the)g(user)f(can)h(sp)q(ecify)g(a)75 271 y(\\no)q(op")d(signal)i(handler,)g(th)o(us)f(relegating)g(all)h(error)e (handling)j(to)d(the)h(user)g(co)q(de,)g(using)h(the)f(error)75 327 y(parameters)d(returned)i(b)o(y)f(the)g(MPI)h(calls.)166 384 y(MPI)d(calls)h(ma)o(y)f(initiate)h(op)q(erations)g(that)e(con)o(tin)o (ue)i(async)o(hronously)g(after)e(the)i(call)g(returned.)75 440 y(Th)o(us,)22 b(the)g(op)q(eration)f(ma)o(y)g(return)g(with)h(a)f(co)q (de)h(indicating)h(successful)g(completion,)g(y)o(et)e(later)75 497 y(cause)15 b(an)g(error)g(exception)h(to)e(b)q(e)i(raised.)k(If)15 b(there)g(is)h(a)f(subsequen)o(t)g(call)h(that)f(relates)g(to)f(the)h(same)75 553 y(op)q(eration)f(\(e.g.,)f(a)g(call)i(that)e(v)o(eri\014es)i(that)e(an)h (async)o(hronous)f(op)q(eration)h(has)g(completed\))h(then)f(the)75 610 y(error)f(parameter)g(asso)q(ciated)h(with)g(this)g(call)h(will)g(b)q(e)g (used)f(to)f(indicate)i(the)f(nature)g(of)f(the)h(error.)k(In)75 666 y(a)c(few)h(cases,)f(the)g(error)g(ma)o(y)g(o)q(ccur)h(after)f(all)h (calls)h(that)d(relate)i(to)f(the)h(op)q(eration)f(ha)o(v)o(e)g(completed,)75 723 y(so)g(that)g(no)g(error)g(parameter)g(can)g(b)q(e)i(used)f(to)e (indicate)k(the)d(nature)g(of)g(the)h(error)f(\(e.g.,)f(an)h(error)g(in)75 779 y(a)e(send)i(with)f(the)g(ready)g(mo)q(de\).)19 b(In)13 b(suc)o(h)g(cases,)g(an)g(error)f(will)i(b)q(e)g(undetected,)g(if)f(the)g (user)g(disabled)75 835 y(the)i(MPI)g(error)g(signal.)166 968 y Fm(Discussion:)34 b Fl(The)14 b(alternativ)o(e)g(c)o(hoice)g(is)g(to)g(ha)o (v)o(e)f(fatal)g(and)h(non-fatal)e(signals.)166 1018 y(One)j(migh)o(t)c(w)o (an)o(t)j(di\013eren)o(t)h(signals)e(for)g(di\013eren)o(t)i(mo)q(dules.)166 1074 y(The)f(details)g(of)f(suc)o(h)i(prop)q(osal)e(need)i(b)q(e)g(elab)q (orated)f(in)f(an)h(appropriate)g(\\pro\014le")f(sub)q(committee.)75 1299 y Fn(1.4)70 b(Messages)75 1401 y Fo(A)15 b(message)g(consists)g(of)g(an) g Fj(envelop)n(e)f Fo(and)i Fj(data)p Fo(.)75 1521 y Fk(1.4.1)55 b(Data)75 1607 y Fo(The)19 b(data)f(part)g(of)g(a)h(message)f(consists)h(of)f (a)h(sequence)g(of)g(v)m(alues,)h(eac)o(h)f(of)f(a)h(basic)g(datat)o(yp)q(e)f (in)75 1664 y(the)d(host)f(language.)20 b(Th)o(us,)14 b(in)i(F)l(ortran)d (77,)h(a)h(message)f(consists)h(of)f(a)g(sequence)i(of)e(v)m(alues)i(that)e (are)75 1720 y(eac)o(h)20 b(of)g(t)o(yp)q(e)h Fi(INTEGER)p Fo(,)e Fi(REAL)p Fo(,)g Fi(DOUBLE)k(PRECISION)p Fo(,)c Fi(COMPLEX)p Fo(,)g Fi(DOUBLE)k(PRECISION)f(COMPLEX)p Fo(,)75 1776 y Fi(LOGICAL)p Fo(,)12 b(or)h(\(length)h(1\))f Fi(CHARACTER)p Fo(.)e(A)j(message)f(ma)o(y)f (also)i(con)o(tain)f(a)h(v)m(alue)g(of)f(t)o(yp)q(e)h Fi(BYTE)p Fo(,)e(whic)o(h)75 1833 y(consists)19 b(of)g(8)f(binary)i(digits.)31 b(A)19 b(b)o(yte)g(is)g(di\013eren)o(t)g(from)f(a)h(c)o(haracter.)30 b(Diferen)o(t)19 b(mac)o(hines)h(ma)o(y)75 1889 y(ha)o(v)o(e)e(di\013eren)o (t)h(represen)o(tations)g(for)f(the)h(same)f(c)o(haracter,)h(or)f(ma)o(y)g (use)h(more)g(than)f(one)h(b)o(yte)g(to)75 1946 y(reprensen)o(t)j(a)g(c)o (haracter.)40 b(On)23 b(the)f(other)g(hand,)i(a)d(b)o(yte)h(has)g(the)g(same) g(binary)h(v)m(alue)g(on)f(an)o(y)75 2002 y(mac)o(hine.)f(A)15 b(message)g(ma)o(y)f(mix)i(v)m(alues)g(of)f(di\013eren)o(t)g(t)o(yp)q(es.)166 2135 y Fm(Missing:)166 2185 y Fl(Need)e(to)f(agree)h(on)e(the)i(C)f(t)o(yp)q (es)h(\(including)e(handling)g(of)h(signed/unsigned\),)g(and)g(on)g(F)m (ortran)g(90)f(t)o(yp)q(es)75 2234 y(\(including)i(handling)g(of)g(kinds\).) 75 2438 y Fk(1.4.2)55 b(En)n(v)n(elop)r(e)75 2524 y Fo(The)15 b(follo)o(wing)h(information)g(is)f(asso)q(ciated)h(with)f(eac)o(h)h (message:)75 2612 y Fg(source)22 b Fo(The)16 b(rank)f(the)g(sending)h(pro)q (cess)75 2704 y Fg(destination)25 b Fo(The)15 b(rank)g(of)g(the)g(receiving)i (pro)q(cess)p eop %%Page: 6 7 bop 75 -100 a Fo(6)631 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 45 y Fg(tag)24 b Fo(User)15 b(de\014ned)75 154 y Fg(con)o(text)23 b Fo(handle)166 263 y(The)14 b(range)g(of)g(v)m(alid)h (v)m(alues)h(for)d(the)h Fg(source)g Fo(and)g Fg(destination)i Fo(\014elds)g(is)e Fi(0)24 b(...)47 b(n-1)p Fo(,)14 b(where)75 320 y Fi(n)g Fo(is)h(the)f(n)o(um)o(b)q(er)h(of)f(pro)q(cesses)g(in)h(the)g (sp)q(eci\014ed)h(con)o(text.)j Fi(source)k(=)h(destination)13 b Fo(is)h(allo)o(w)o(ed:)20 b(a)75 376 y(pro)q(cess)15 b(can)h(send)g(a)e (message)h(to)g(itself.)166 437 y(The)h(ranges)f(of)h(v)m(alid)h(v)m(alues)g (for)e Fi(tag)h Fo(is)g(implemen)o(tation)h(dep)q(enden)o(t,)g(and)f(can)g(b) q(e)h(found)f(b)o(y)75 493 y(calling)h(a)e(suitable)h(query)g(function,)f(as) g(describ)q(ed)i(in)f(Section)g Fg(??)p Fo(.)166 553 y(The)f Fi(tag)g Fo(\014eld)h(can)f(b)q(e)h(arbitrarily)g(set)f(b)o(y)g(the)g (application,)h(and)g(can)f(b)q(e)h(used)f(to)g(distinguish)75 610 y(di\013eren)o(t)g(messages.)166 670 y Fi(Context)f Fo(should)i(b)q(e)g (a)f(con)o(text)g(shared)g(b)o(y)g(b)q(oth)g(source)h(and)f(destination.)166 730 y(The)e(actual)h(mec)o(hanism)g(used)f(to)g(asso)q(ciate)g(an)g(en)o(v)o (elop)q(e)i(with)e(a)g(message)g(is)h(implemen)o(tation)75 787 y(dep)q(enden)o(t;)k(some)e(of)g(the)g(information)g(\(e.g.,)f Fg(sender)g Fo(or)h Fg(receiv)o(er)p Fo(\))f(ma)o(y)g(b)q(e)i(implicit,)i (and)d(need)75 843 y(not)f(b)q(e)h(explicitly)i(carried)d(b)o(y)h(a)e (message.)75 1008 y Fn(1.5)70 b(Comm)n(unication)20 b(Bu\013ers)75 1117 y Fo(The)f(basic)h(p)q(oin)o(t)g(to)e(p)q(oin)o(t)i(comm)o(unication)g (op)q(erations)f(are)g Fg(send)g Fo(and)g Fg(receiv)o(e)p Fo(.)31 b(A)19 b Fg(send)g Fo(op-)75 1174 y(eration)g(creates)g(a)f(message;)i(the)g (message)e(data)g(is)i(tak)o(en)f(from)f(the)h Fg(send)i(bu\013er)p Fo(.)32 b(A)19 b Fg(receiv)o(e)75 1230 y Fo(op)q(eration)i(consumes)h(a)f (message;)i(the)e(message)g(data)f(is)i(put)f(in)o(to)g(the)g Fg(receiv)o(e)j(bu\013er)p Fo(.)38 b(The)75 1287 y(sp)q(eci\014cation)17 b(of)e(send)h(or)e(receiv)o(e)i(bu\013ers)g(uses)f(the)g(same)g(syn)o(tax.) 166 1347 y(A)j(bu\013er)g(consists)h(of)e(a)h(sequence)h Fg(bu\013er)i(comp)q (onen)o(ts)p Fo(.)29 b(Eac)o(h)18 b(comp)q(onen)o(t)g(consists)h(of)e(a)75 1403 y(sequence)h(of)f(\(not)g(necessarily)i(distinct\))f(v)m(ariables)g(of)f (the)h(same)e(basic)i(datat)o(yp)q(e.)26 b(The)17 b(datat)o(yp)q(e)75 1460 y(of)k(suc)o(h)h(v)m(ariable)h(is)f(sp)q(eci\014ed)i(b)o(y)e(a)f(state)g (parameter.)39 b(The)21 b(p)q(ossible)j(v)m(alues)f(for)e(F)l(ortran)f(are)75 1516 y Fi(MPI)p 150 1516 15 2 v 17 w(INT)p Fo(,)e Fi(MPI)p 342 1516 V 17 w(REAL)p Fo(,)h Fi(MPI)p 559 1516 V 16 w(DOUBLE)p Fo(,)g Fi(MPI)p 823 1516 V 16 w(COMPLEX)p Fo(,)g Fi(MPI)p 1111 1516 V 16 w(DCOMPLEX)p Fo(,)f Fi(MPI)p 1422 1516 V 17 w(LOGICAL)p Fo(,)g(MPI)p 1727 1516 14 2 v 17 w(CHAR)75 1573 y(and)k Fi(MPI)p 245 1573 15 2 v 17 w(BYTE)p Fo(,)f(corrsp)q(onding)i(to)f(datat)o(yp)q(es)g Fi(INTEGER)p Fo(,)e Fi(REAL)p Fo(,)i Fi(DOUBLE)h(PRECISION)p Fo(,)d Fi(COMPLEX)p Fo(,)75 1629 y Fi(DOUBLE)j(PRECISION)g(COMPLEX)p Fo(,)13 b Fi(LOGICAL)p Fo(,)g Fi(CHARACTER)g Fo(and)i(to)f(un)o(t)o(yp)q(ed)h Fi(BYTE)p Fo(.)f(The)h(p)q(ossible)h(v)m(al-)75 1686 y(ues)10 b(for)g(C)g(are)g Fi(MPI)p 402 1686 V 17 w(CHAR)p Fo(,)f Fi(MPI)p 609 1686 V 16 w(SHORT)p Fo(,)h Fi(MPI)p 840 1686 V 16 w(INT)p Fo(,)g Fi(MPI)p 1023 1686 V 16 w(LONG)p Fo(,)g Fi(MPI)p 1230 1686 V 16 w(FLOAT)p Fo(,)f Fi(MPI)p 1460 1686 V 17 w(DOUBLE)g Fo(and)i Fi(MPI)p 1786 1686 V 17 w(BYTE)p Fo(.)166 1822 y Fm(Discussion:)166 1876 y Fl(Ma)o(y)17 b(need)i(to)f(tak)o(e)g(care)h(of)e(kinds)h(in)g(F)m (ortran)f(90)h(and)g(of)f(signed/unsigned)h(in)g(C.)f(More)i(elab)q(orate)75 1926 y(de\014nitions)14 b(b)q(elong)f(to)h(the)h(language)e(binding.)166 2069 y Fo(There)i(are)g(\014v)o(e)h(kinds)g(of)f(bu\013er)g(comp)q(onen)o (ts:)75 2210 y Fg(Con)o(tiguous)46 b Fo(A)15 b(sequence)h(of)f(con)o(tiguous) g(v)m(alues)i(of)e(the)g(same)g(basic)h(t)o(yp)q(e,)e(sp)q(eci\014ed)k(b)o(y) 75 2320 y Fg(start)23 b Fo(Initial)17 b(elemen)o(t)75 2429 y Fg(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\()p Fi(len)11 b Fe(\025)i Fi(0)p Fo(\))75 2538 y Fg(datat)o(yp)q(e)24 b Fo(T)o(yp)q(e)15 b(of)g(elemen)o(ts)166 2647 y(Th)o(us,)e(if)h Fi(A)e Fo(is)i(an)f(arra)o(y)f(of)h(in)o(tegers,)g(then)g Fi(A\(1\),)24 b(4,)f(MPI)p 1210 2647 V 17 w(INT)13 b Fo(is)g(the)g(bu\013er)g(comp)q(onen)o (t)h(with)75 2704 y(en)o(tries)i Fi(A\(1\),)23 b(A\(2\),)g(A\(3\),)g(A\(4\))p Fo(.)p eop %%Page: 7 8 bop 75 -100 a Fh(1.5.)34 b(COMMUNICA)l(TION)16 b(BUFFERS)1023 b Fo(7)75 45 y Fg(V)l(ector)45 b Fo(A)19 b(sequence)h(of)f(equally)h(spaced)g (and)f(equally)i(sized)f(blo)q(c)o(ks)g(of)f(elemen)o(ts)g(of)g(the)g(same)75 102 y(basic)d(t)o(yp)q(e,)f(sp)q(eci\014ed)i(b)o(y)75 196 y Fg(start)23 b Fo(Initial)17 b(elemen)o(t)75 291 y Fg(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\()p Fi(len)11 b Fe(\025)i Fi(0)p Fo(\))75 386 y Fg(stride)23 b Fo(Num)o(b)q(er)15 b(of)g(elemen)o(ts)h(b)q(et)o(w)o(een)f(the)h(start)e(of)g(eac)o(h)i(blo)q(c) o(k)75 481 y Fg(len)o(blk)23 b Fo(Num)o(b)q(er)16 b(of)e(elemen)o(ts)i(in)g (eac)o(h)g(blo)q(c)o(k)f(\()p Fi(lenblk)d Fd(>)h Fi(0)p Fo(\))75 576 y Fg(datat)o(yp)q(e)24 b Fo(T)o(yp)q(e)15 b(of)g(elemen)o(ts)166 671 y(Assume)c(that)f(an)h(elemen)o(t)h(of)e(t)o(yp)q(e)h Fi(datatype)f Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)18 b(Then,)12 b(in)g(a)e(b)o(yte)h (addressable)75 727 y(mac)o(hine,)k(the)g(bu\013er)g(comp)q(onen)o(t)g (describ)q(ed)i(b)o(y)e(these)g(\014v)o(e)g(parameters)f(consists)h(of)f(the) h Fi(len)f Fo(v)m(ari-)75 783 y(ables)i(at)e(addresses)i Fi(start,)23 b(start+m,)g(...)47 b(,)24 b(start+m*\(lenblk-1\),)d(start+m*stride,)75 840 y(start+m*\(stride+1\),)g(...,)j(start+m*\(stride+lenblk-)o(1\),)d(...)p Fo(.)h(F)l(or)16 b(example,)h(if)f Fi(A)g Fo(is)h(an)75 896 y(arra)o(y)d(of)h(in)o(tegers,)g(then)143 991 y Fe(\017)23 b Fi(A\(1\),)g(5,)g(3,)h(2,)g(MPI)p 622 991 15 2 v 16 w(INT)17 b Fo(is)g(the)g(bu\013er)g(comp)q(onen)o(t)g(with)g(en)o(tries)g Fi(A\(1\),)24 b(A\(2\),)f(A\(4\),)189 1048 y(A\(5\),)g(A\(7\))p Fo(.)143 1142 y Fe(\017)g Fi(A\(1\),)g(5,)g(0,)h(2,)g(MPI)p 622 1142 V 16 w(INT)17 b Fo(is)g(the)g(bu\013er)g(comp)q(onen)o(t)g(with)g (en)o(tries)g Fi(A\(1\),)24 b(A\(2\),)f(A\(1\),)189 1199 y(A\(2\),)g(A\(1\))p Fo(.)143 1294 y Fe(\017)g Fi(A\(10\),)g(5,)g(-1,)h(2,)f(MPI)p 669 1294 V 17 w(INT)g Fo(is)h(the)g(bu\013er)g(comp)q(onen)o(t)g(with)g(en)o (tries)g Fi(A\(10\),)f(A\(11\),)189 1350 y(A\(9\),)g(A\(10\),)g(A\(8\))p Fo(.)166 1445 y Fi(Contiguous)18 b Fo(is)h(a)g(sp)q(ecial)i(case)e(of)g Fi(vector)p Fo(.)30 b(A)19 b(v)o(ector)g(bu\013er)g(comp)q(onen)o(t)g(can)g (b)q(e)h(used)g(to)75 1502 y(extract)14 b(an)i(arbitrary)e(submatrix)h(of)g (a)g(t)o(w)o(o-dimensional)h(matrix.)75 1623 y Fg(Indexed)45 b Fo(A)15 b(sequence)i(of)d(elemen)o(ts)i(of)f(the)g(same)g(basic)h(t)o(yp)q (e,)f(sp)q(eci\014ed)i(b)o(y)75 1718 y Fg(start)23 b Fo(initial)17 b(elemen)o(t)75 1813 y Fg(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h (\()p Fi(len)11 b Fe(\025)i Fi(0)p Fo(\))75 1908 y Fg(arra)o(y)p 199 1908 16 2 v 18 w(of)p 259 1908 V 19 w(indices)23 b Fo(Arra)o(y)11 b(of)g(displacemen)o(ts)i(of)e(the)h(elemen)o(ts)g(in)g(the)g(bu\013er)f (comp)q(onen)o(ts,)h(relativ)o(e)189 1964 y(to)i(the)h(initial)j(elemen)o(t.) 75 2059 y Fg(datat)o(yp)q(e)24 b Fo(T)o(yp)q(e)15 b(of)g(elemen)o(ts)166 2154 y(F)l(or)20 b(example,)i(if)f Fi(A)f Fo(is)h(an)g(arra)o(y)e(of)h(in)o (tegers,)h(and)g Fi(I)f Fo(is)h(an)g(arra)o(y)e(with)i(v)m(alues)g Fi(2,)j(-1,)f(2,)75 2210 y(1)c Fo(then)h Fi(A\(3\),)j(4,)h(I,)f(MPI)p 587 2210 15 2 v 17 w(INT)c Fo(is)h(the)f(bu\013er)h(comp)q(on)o(ten)o(t)f (with)h(en)o(tries)f Fi(A\(5\),)24 b(A\(2\),)f(A\(5\),)75 2267 y(A\(4\))p Fo(.)166 2323 y Fi(Vector)14 b Fo(is)i(a)f(sp)q(ecial)i(case)e(of) g Fi(indexed)p Fo(.)75 2445 y Fg(Heterogeneous)25 b(V)l(ector)g(\(h)o(v)o (ector\))45 b Fo(Same)22 b(as)g Fi(vector)p Fo(,)h(except)g(that)e Fi(stride)h Fo(is)g(giv)o(en)h(in)75 2501 y(m)o(ultiple)17 b(of)e(b)o(ytes,)f(rather)h(than)g(m)o(ultiples)i(of)e(elemen)o(ts:)75 2609 y Fg(start)23 b Fo(Initial)17 b(elemen)o(t)75 2704 y Fg(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\()p Fi(len)11 b Fe(\025)i Fi(0)p Fo(\))p eop %%Page: 8 9 bop 75 -100 a Fo(8)631 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 45 y Fg(stride)23 b Fo(Num)o(b)q(er)15 b(of)g(b)o(ytes)g(b)q(et)o(w)o(een)h(the)f(start)f(of)h(eac)o(h)g(blo)q(c)o (k)75 138 y Fg(len)o(blk)23 b Fo(Num)o(b)q(er)16 b(of)e(elemen)o(ts)i(in)g (eac)o(h)g(blo)q(c)o(k)f(\()p Fi(lenblk)d Fd(>)h Fi(0)p Fo(\))75 231 y Fg(datat)o(yp)q(e)24 b Fo(T)o(yp)q(e)15 b(of)g(elemen)o(ts)166 334 y(Assume)h(that)f(an)h(elemen)o(t)g(of)f(t)o(yp)q(e)h Fi(datatype)f Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)21 b(Then,)16 b(in)h(a)e(b)o(yte)h (address-)75 390 y(able)g(mac)o(hine,)g(the)g(bu\013er)f(comp)q(onen)o(t)h (describ)q(ed)i(b)o(y)d(these)h(\014v)o(e)f(parameters)g(consists)h(of)f(the) g Fi(len)75 447 y Fo(v)m(ariables)23 b(at)e(addresses)h Fi(start,)h(start+m,) f(...)48 b(,)24 b(start+m*\(lenblk-1\),)d(start+stride,)75 503 y(start+stride+m,)h(...,)h(start+stride+m*\(lenblk-1\),)d(...)p Fo(.)166 559 y(Consider)c(a)f(C)g(arra)o(y)e(declared)k(as)75 662 y Fi(struct)23 b({)147 719 y(char)g(info[3];)147 775 y(short)g(number;) 147 832 y(double)g(val;)242 888 y(})71 b(a[100])166 991 y Fo(Assume)11 b(that)f(all)i(structure)e(comp)q(onen)o(ts)h(are)f(b)o(yte)h(aligned,)i(a)d Fi(short)g Fo(o)q(ccupies)i(t)o(w)o(o)e(b)o(ytes,)h(and)75 1047 y(a)h Fi(double)g Fo(o)q(ccupies)i(eigh)o(t)e(b)o(ytes.)19 b(Then)13 b(the)f(bu\013er)h(comp)q(onen)o(t)f Fi(&a,)24 b(100,)f(13,)g(2,)h (MPI)p 1694 1047 15 2 v 17 w(CHAR)12 b Fo(ex-)75 1104 y(tracts)f(the)h (\014rst)g(t)o(w)o(o)f(c)o(haracters)g(of)h(eac)o(h)g(structure.)19 b(The)12 b(bu\013er)g(comp)q(onen)o(t)g Fi(&\(a.number\),)23 b(100,)75 1160 y(13,)g(1,)h(MPI)p 317 1160 V 17 w(SHORT)19 b Fo(extracts)f(the)i Fi(short)f Fo(\014eld)h(of)f(eac)o(h)h(structure;)h (and)f(the)f(bu\013er)h(comp)q(onen)o(t)75 1217 y Fi(&\(a.val\),)j(100,)g (13,)g(1)18 b Fo(extracts)e(the)i Fi(double)f Fo(\014eld)i(of)e(eac)o(h)g (structure.)27 b(Th)o(us,)18 b(heterogeneous)75 1273 y(v)o(ectors)c(can)i(b)q (e)g(used)f(to)g(extract)f(v)o(ectors)h(of)g(homogeneous)g(v)m(alues)h(from)e (heterogenous)i(data.)166 1330 y Fi(Vector)e Fo(is)i(a)f(sp)q(ecial)i(case)e (of)g Fi(heterogenous)22 b(vector)p Fo(.)75 1449 y Fg(Heterogeneous)d (indexed)g(\(hindexed\))46 b Fo(Same)16 b(as)g(indexed,)j(except)e(that)e (displacemen)o(ts)j(are)75 1506 y(in)e(b)o(ytes,)f(rather)f(than)h(m)o (ultiples)i(of)e(elemen)o(t)h(size.)75 1609 y Fg(start)23 b Fo(initial)17 b(elemen)o(t)75 1701 y Fg(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\()p Fi(len)11 b Fe(\025)i Fi(0)p Fo(\))75 1794 y Fg(arra)o(y)p 199 1794 16 2 v 18 w(of)p 259 1794 V 19 w(indices)23 b Fo(Arra)o(y)17 b(of)g(b)o(yte)h(displacemen)o(ts)h(of)e(the)h (elemen)o(ts)g(in)h(the)f(bu\013er)f(comp)q(onen)o(ts,)189 1851 y(relativ)o(e)e(to)g(the)g(initial)j(elemen)o(t.)75 1944 y Fg(datat)o(yp)q(e)24 b Fo(T)o(yp)q(e)15 b(of)g(elemen)o(ts)166 2046 y(F)l(or)21 b(example,)k(if)d Fi(a)g Fo(is)h(the)f(C)g(arra)o(y)e(of)i (the)g(previous)h(example,)h(and)e Fi(I)g Fo(is)h(an)f(arra)o(y)f(with)75 2103 y(v)m(alues)14 b Fi(0,)24 b(13,)f(13,)h(27,)f(26)13 b Fo(then)h Fi(&a,)23 b(5,)h(I,)f(MPI)p 1042 2103 15 2 v 17 w(CHAR)13 b Fo(is)g(the)h(bu\013er)f(comp)q(onen)o(t)g(with)h(en)o(tries)75 2159 y Fi(a[0].info[0],)22 b(a[1].info[0],)g(a[1].info[0],)g(a[1].info[1],)g (a[1].info[0])p Fo(.)166 2216 y(Lik)o(e)d Fi(heterogenous)j(vector)p Fo(,)17 b Fi(heterogenous)22 b(indexed)17 b Fo(extends)i(the)f Fi(indexed)e Fo(facilit)o(y)j(so)75 2272 y(as)c(to)f(allo)o(w)i(the)f (extraction)g(of)g(homogeneous)g(data)f(from)h(a)g(heterogenous)g(la)o(y)o (out.)166 2405 y Fm(Discussion:)166 2455 y Fl(The)10 b(use)g(of)f (displacemen)o(ts)g(in)g(indexed)h(bu\013er)h(comp)q(onen)o(ts)e(has)h(the)g (same)f(e\013ect)i(as)e(indirect)h(addressing)75 2504 y(in)o(to)j(an)g(arra)o (y)h(with)f(initial)e(index)j(zero.)19 b(I.e.,)12 b(if)h Ff(a)h Fl(is)f(an)g(arra)o(y)h(with)f(initial)f(index)h(zero,)h(then)g(a)g(bu\013er) g(com-)75 2554 y(p)q(onen)o(t)j(de\014ned)h(b)o(y)e(parameters)g Ff(a[0],)21 b(N,)h(i,)f(type)16 b Fl(extract)h(the)g(elemen)o(ts)g Ff(a[i[0]],)j(a[i[1]],)g(...,)75 2604 y(a[i[n]])p Fl(.)25 b(This)16 b(\014ts)h(w)o(ell)f(C)h(usage)g(but)f(ill-\014ts)g(F)m(ortran)g(usage)h (\(initial)e(index)i(one\).)26 b(W)m(e)16 b(to)q(ok)g(a)h(decision)75 2654 y(of)d(principle)h(to)f(force)h(\\start)g(from)e(zero")i(index)g (arithmetic)e(on)i(F)m(ortran)f(users,)i(rather)f(than)g(ha)o(v)o(e)g (diferen)o(t)75 2704 y(index)f(arithmetic)f(in)g(the)i(C)e(and)h(F)m(ortran)g (binding)f(of)g(MPI.)p eop %%Page: 9 10 bop 75 -100 a Fh(1.5.)34 b(COMMUNICA)l(TION)16 b(BUFFERS)1023 b Fo(9)166 45 y Fl(The)15 b(use)h(of)e(b)q(oth)h(bu\013er)h(comp)q(onen)o(ts) f(with)g(b)o(yte)g(and)g(elemen)o(t)f(size)i(displacemen)o(ts)e(is)h(motiv)n (ated)e(b)o(y)75 95 y(the)j(observ)n(ation)f(that)g(some)g(applications)f (require)i(the)g(more)f(general)g(b)o(yte)h(displacemen)o(t,)e(but)i(that)f (most)75 145 y(applications)d(w)o(ould)h(use)i(elemen)o(t)e(size)h (displacemen)o(t,)e(whic)o(h)i(is)f(m)o(uc)o(h)f(more)h(natural)g(and)g(con)o (v)o(enien)o(t)h(\(and)75 195 y(mac)o(hine)i(indep)q(enden)o(t\).)29 b(Th)o(us,)18 b(rather)g(than)f(forcing)f(all)g(users)j(to)e(do)f(b)o(yte)i (arithemetic,)f(w)o(e)g(decided)h(to)75 244 y(pro)o(vide)c(separately)g(the)h (more)e(adv)n(anced)h(function)f(to)h(\\exp)q(ert)h(programmers".)166 384 y Fo(A)g(bu\013er)h(is)g(describ)q(ed)h(b)o(y)e(an)h(opaque)f(ob)s(ject)g (accessed)h(via)g(a)f Fg(bu\013er)j(handle)p Fo(.)j(Suc)o(h)c(ob)s(ject)75 440 y(is)k(created)g(b)o(y)f Fi(MPI)p 434 440 15 2 v 17 w(CREATE)p 595 440 V 16 w(BUFFER)p Fo(.)f(It)i(is)g(asso)q(ciated)g(with)f(successiv)o (e)i(bu\013er)e(comp)q(onen)o(ts)h(b)o(y)75 497 y(calling)c(in)g(succession)f (one)g(of)f(the)g(functions)i Fi(MPI)p 974 497 V 16 w(APPEND)p 1134 497 V 17 w(xxx)p Fo(,)d(where)i Fi(xxx)f Fo(iden)o(ti\014es)i(the)f(t)o (yp)q(e)f(of)75 553 y(bu\013er)i(comp)q(onen)o(t)h(app)q(ended)h(to)d(the)i (bu\013er.)26 b(It)18 b(b)q(ecomes)g(a)o(v)m(ailable)h(for)d(use)i(in)g(comm) o(unication)75 610 y(op)q(erations)h(once)h(committed)f(b)o(y)g(the)g (function)h Fi(MPI)p 1045 610 V 17 w(COMMIT)p Fo(.)d(A)i(bu\013er)g (descriptor)h(is)f(destro)o(y)o(ed)75 666 y(after)13 b(the)h(completion)h(of) f(the)g(\014rst)f(comm)o(unication)i(op)q(eration)f(that)f(uses)h(it,)h(if)f (it)g(is)h(ephemeral,)f(or)75 723 y(after)h(it)g(is)h(freed)g(b)o(y)f(a)g (call)i(to)d Fi(MPI)p 702 723 V 17 w(FREE)h Fo(and)g(an)o(y)g(p)q(ending)i (comm)o(unication)f(op)q(eration)g(that)f(uses)75 779 y(it)i(has)f (completed,)h(if)g(it)f(is)h(p)q(ersisten)o(t.)24 b(After)16 b(the)g(bu\013er)h(descriptor)f(ob)s(ject)g(has)g(b)q(een)i(destro)o(y)o(ed,) 75 836 y(the)d(handle)i(is)e(unde\014ned.)166 928 y Fg(MPI)p 275 928 16 2 v 18 w(CREA)l(TE)p 510 928 V 19 w(BUFFER\()j(bu\013er,)f(p)q (ersistence)h(\))166 1020 y Fo(Create)d(a)f(new)i(bu\013er)f(descriptor)h(ob) s(ject.)j(The)c(parameters)g(are:)75 1116 y Fg(OUT)j(bu\013er)23 b Fo(bu\013er)15 b(handle)75 1212 y Fg(IN)i(p)q(ersistence)23 b Fo(bu\013er)g(p)q(ersistence)h(\(state)e(t)o(yp)q(e)h(with)g(p)q(ossible)h (v)m(alues)g Fi(MPI)p 1559 1212 15 2 v 17 w(PERSISTENT)d Fo(or)189 1269 y Fi(MPI)p 264 1269 V 16 w(EPHEMERAL)p Fo(\).)166 1441 y Fm(Discussion:)166 1491 y Fl(Ma)o(y)d(w)o(an)o(t)g(to)h(pro)o(vide)f (information)e(on)i(maxim)n(um)c(ob)r(ject)20 b(size,)g(and)e(p)q(erhaps)i (user)g(space)f(for)f(the)75 1541 y(ob)r(ject.)166 1716 y Fg(MPI)p 275 1716 16 2 v 18 w(APPEND)p 520 1716 V 18 w(CONTIGUOUS\()h(bu\013er,)e (start,)g(len,)g(datat)o(yp)q(e\))166 1808 y Fo(App)q(end)g(a)e(blo)q(c)o(k)g (comp)q(onen)o(t)h(to)e(bu\013er.)20 b(The)c(parameters)e(are:)75 1904 y Fg(INOUT)k(bu\013er)23 b Fo(bu\013er)15 b(handle)75 2000 y Fg(IN)i(start)23 b Fo(bu\013er)15 b(comp)q(onen)o(t)h(initial)h (elemen)o(t)f(\(c)o(hoice\))75 2096 y Fg(IN)h(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 2192 y Fg(IN)h(datat)o(yp)q(e)24 b Fo(datat)o(yp)q(e)15 b(iden)o(ti\014er)h(\(status\))166 2323 y Fg(MPI)p 275 2323 V 18 w(APPEND)p 520 2323 V 18 w(VEC\()h(bu\013er,)g(len,) h(stride,)f(len)o(blk,)g(datat)o(yp)q(e)i(\))166 2416 y Fo(App)q(end)e(a)e(v) o(ector)f(bu\013er)h(comp)q(onen)o(t)g(to)g(bu\013er.)20 b(The)15 b(parameters)g(are:)75 2512 y Fg(INOUT)j(bu\013er)23 b Fo(bu\013er)15 b(handle)75 2608 y Fg(IN)i(start)23 b Fo(bu\013er)15 b(comp)q(onen)o(t)h (initial)h(elemen)o(t)f(\(c)o(hoice\))75 2704 y Fg(IN)h(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\(in)o(teger\))p eop %%Page: 10 11 bop 75 -100 a Fo(10)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fg(IN)h(stride)23 b Fo(Num)o(b)q(er)16 b(of)e(elemen)o(ts)i(b)q(et)o(w)o(een)g(the)f(start)f(of)h (eac)o(h)g(blo)q(c)o(k)h(\(in)o(teger\))75 136 y Fg(IN)h(len)o(blk)23 b Fo(Num)o(b)q(er)16 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g(\(in)o (teger\))75 228 y Fg(IN)h(datat)o(yp)q(e)24 b Fo(datat)o(yp)q(e)15 b(iden)o(ti\014er)h(\(status\))166 350 y Fg(MPI)p 275 350 16 2 v 18 w(APPEND)p 520 350 V 18 w(INDEXED\()i(bu\013er,)f(start,)g(len,)h (arra)o(y)p 1352 350 V 17 w(of)p 1411 350 V 19 w(indices,)g(datat)o(yp)q(e\)) 166 442 y Fo(App)q(end)f(an)e(indexed)i(bu\013er)e(comp)q(onen)o(t)g(to)g (bu\013er.)20 b(The)15 b(parameters)f(are:)75 530 y Fg(INOUT)k(bu\013er)23 b Fo(bu\013er)15 b(handle)75 621 y Fg(IN)i(start)23 b Fo(initial)17 b(p)q(osition)g(for)d(indexing)j(\(c)o(hoice\))75 712 y Fg(IN)g(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 803 y Fg(IN)h(arra)o(y)p 277 803 V 18 w(of)p 337 803 V 19 w(indices)24 b Fo(arra)o(y)14 b(of)g(displacemen)o(ts)j(of)e(en)o(tries)g(relativ)o(e)h (to)f(start)f(\(arra)o(y)f(of)i(in)o(tegers\))75 894 y Fg(IN)i(datat)o(yp)q (e)24 b Fo(datat)o(yp)q(e)15 b(iden)o(ti\014er)h(\(status\))166 1017 y Fg(MPI)p 275 1017 V 18 w(APPEND)p 520 1017 V 18 w(HVEC\()h(bu\013er,)g (len,)h(stride,)f(len)o(blk,)g(datat)o(yp)q(e)h(\))166 1109 y Fo(App)q(end)f(a)e(heterogenous)g(v)o(ector)f(bu\013er)h(comp)q(onen)o(t)h (to)e(bu\013er.)20 b(The)15 b(parameters)g(are:)75 1196 y Fg(INOUT)j (bu\013er)23 b Fo(bu\013er)15 b(handle)75 1288 y Fg(IN)i(start)23 b Fo(bu\013er)15 b(comp)q(onen)o(t)h(initial)h(elemen)o(t)f(\(c)o(hoice\))75 1379 y Fg(IN)h(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 1470 y Fg(IN)h(stride)23 b Fo(Num)o(b)q(er)16 b(of)e(b)o(ytes)h (b)q(et)o(w)o(een)h(the)f(start)f(of)h(eac)o(h)g(blo)q(c)o(k)h(\(in)o (teger\))75 1561 y Fg(IN)h(len)o(blk)23 b Fo(Num)o(b)q(er)16 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g(\(in)o(teger\))75 1653 y Fg(IN)h(datat)o(yp)q(e)24 b Fo(datat)o(yp)q(e)15 b(iden)o(ti\014er)h (\(status\))166 1775 y Fg(MPI)p 275 1775 V 18 w(APPEND)p 520 1775 V 18 w(HINDEXED\()h(bu\013er,)g(start,)g(len,)g(arra)o(y)p 1391 1775 V 18 w(of)p 1451 1775 V 19 w(indices,)h(datat)o(yp)q(e\))166 1924 y Fo(App)q(end)f(an)e(heterogenous)g(indexed)i(bu\013er)e(comp)q(onen)o (t)g(to)g(bu\013er.)20 b(The)15 b(parameters)f(are:)75 2011 y Fg(INOUT)k(bu\013er)23 b Fo(bu\013er)15 b(handle)75 2102 y Fg(IN)i(start)23 b Fo(initial)17 b(p)q(osition)g(for)d(indexing)j(\(c)o (hoice\))75 2194 y Fg(IN)g(len)23 b Fo(Num)o(b)q(er)16 b(of)f(elemen)o(ts)h (\(in)o(teger\))75 2285 y Fg(arra)o(y)p 199 2285 V 18 w(of)p 259 2285 V 19 w(indices)23 b Fo(arra)o(y)12 b(of)h(b)o(yte)g(displacemen)o (ts)i(of)e(en)o(tries)g(relativ)o(e)h(to)f(start)f(\(arra)o(y)f(of)i(in)o (tegers\))75 2376 y Fg(IN)k(datat)o(yp)q(e)24 b Fo(datat)o(yp)q(e)15 b(iden)o(ti\014er)h(\(status\))166 2499 y Fg(MPI)p 275 2499 V 18 w(COMMIT)p 526 2499 V 18 w(BUFFER\()j(bu\013erin)f(\))166 2591 y Fo(Commit)h(a)g(bu\013er)g(descriptor)h(ob)s(ject.)31 b(The)19 b(ob)s(ject)g(cannot)g(b)q(e)h(c)o(hanged)g(an)o(ymore)e(after)h(it) 75 2647 y(is)h(committed.)31 b(A)19 b(bu\013er)g(descriptor)g(ob)s(ject)g (can)g(b)q(e)h(used)f(for)g(comm)o(unication)g(only)h(after)e(it)h(is)75 2704 y(committed.)h(The)15 b(parameters)g(are:)p eop %%Page: 11 12 bop 75 -100 a Fh(1.5.)34 b(COMMUNICA)l(TION)16 b(BUFFERS)1001 b Fo(11)75 45 y Fg(INOUT)18 b(bu\013er)23 b Fo(bu\013er)15 b(descriptor)h(handle)166 216 y Fm(Discussion:)166 267 y Fl(The)j(commit)d (function)j(ma)o(y)e(do)h(prepro)q(cessing)j(of)d(the)i(bu\013er)g (descriptor)g(to)f(simplify)d(its)i(use)i(and)75 316 y(commit)12 b(resources)17 b(in)d(the)i(comm)o(uni)o(cation)c(subsystem)j(for)f(its)h (use.)21 b(The)16 b(commit)11 b(function)k(ma)o(y)e(generate)75 366 y(a)h(new)g(ob)r(ject)h(and,)e(hence,)i(mo)q(dify)c(the)k(ob)r(ject)f (handle.)166 506 y Fo(Consider,)h(for)g(example,)h(the)f(follo)o(wing)h (fragmen)o(t)e(of)h(F)l(ortran)f(co)q(de)75 614 y Fi(DOUBLE)23 b(PRECISION)g(A\(10,20\))75 671 y(INTEGER)g(B,)g(C\(5,10\))75 727 y(INTEGER)g(BH)75 784 y(...)75 840 y(CALL)g(MPI_CREATE_BUFFER\(BH,)e (MPI_PERSISTENT\))75 897 y(CALL)i(MPI_APPEND_BLOCK)f(\(BH,)h(B,)h(1,)f (MPI_INT\))75 953 y(CALL)g(MPI_APPEND_VEC)f(\(BH,)i(A\(1,3\),)e(11,)i(4,)g (2,)f(MPI_DOUBLE\))75 1009 y(CALL)g(MPI_APPEND_INDEX\(BH,)e(C\(3,7\),)i (\(4,2,1\),)g(MPI_INT\))166 1118 y Fo(Then)16 b(the)f(bu\013er)g(asso)q (ciated)g(with)h(the)f(handle)i Fi(BH)e Fo(consists)g(of)g(the)g(sequence)h (of)f(v)m(ariables)166 1174 y Fi(B,)23 b(A\(1,3\),)g(A\(2,3\),)g(A\(5,3\),)g (A\(6,3\),)g(A\(9,3\),)g(A\(10,3\),)g(A\(3,4\),)g(A\(4,4\),)g(A\(7,4\),)75 1231 y(A\(8,4\),)g(A\(1,5\),)g(C\(2,8\),)g(C\(5,7\),)g(C\(4,7\))p Fo(.)166 1288 y(A)17 b(message)g(created)g(from)f(this)i(bu\013er)f(will)i (consist)e(of)g(a)g(sequence)h(of)f(one)g(in)o(teger,)h(follo)o(w)o(ed)75 1344 y(b)o(y)d(elev)o(en)h(double)h(precision)g(reals,)e(follo)o(w)o(ed)g(b)o (y)g(three)h(in)o(tegers.)166 1401 y(A)h(bu\013er)g(handle)h(can)f(b)q(e)h (used)g(for)e(comm)o(unication,)i(ev)o(en)f(if)g(it)h(is)f(not)g(asso)q (ciated)g(with)g(an)o(y)75 1457 y(v)m(ariables)i(\(i.e.,)f(ev)o(en)g(if)g(it) g(w)o(as)f(not)g(set)g(b)o(y)h(an)o(y)f Fi(MPI)p 1046 1457 15 2 v 17 w(ADD)p 1135 1457 V 17 w(xxx)g Fo(call\).)28 b(Suc)o(h)19 b(handle)g(is)f(asso)q(ciated)75 1514 y(with)e(an)f(empt)o(y)g(bu\013er,)f (and)i(a)f(message)f(created)i(from)e(it)h(con)o(tains)h(no)f(data.)166 1571 y(The)d(same)g(en)o(try)g(ma)o(y)g(app)q(ears)g(more)g(than)g(once)g(in) h(a)f(bu\013er)h(comp)q(onen)o(t,)f(and)h(distinct)g(bu\013er)75 1627 y(comp)q(onen)o(ts)j(ma)o(y)f(o)o(v)o(erlap.)23 b(If)16 b(a)g(bu\013er)g(descriptor)g(with)h(replicated)g(en)o(tries)g(is)f(used)h (to)e(receiv)o(e)i(a)75 1684 y(message,)12 b(then)h(the)g(\014nal)g(v)m(alue) h(of)e(an)g(en)o(try)g(that)g(o)q(ccurs)h(more)f(than)g(once)h(in)g(the)g (bu\013er)g(descriptor)75 1740 y(is)j(unde\014ned.)166 1873 y Fm(Discussion:)166 1923 y Fl(It)g(is)g(not)g(clear)h(to)e(me)h(that)g(o)o (v)o(erlaps)g(ha)o(v)o(e)f(an)o(y)h(useful)g(purp)q(ose,)i(and)d(they)i(ma)o (y)d(complicate)h(imple-)75 1973 y(men)o(tation.)h(But)f(w)o(e)f(had)g(an)f (o)o(v)o(erwhelming)f(v)o(ote)i(in)f(fa)o(v)o(or)g(of)h(this)g(feature.)75 2179 y Fk(1.5.1)55 b(T)n(yp)r(e)18 b(Matc)n(hing)i(and)f(Data)g(Con)n(v)n (ersion)75 2266 y Fo(One)d(can)f(think)h(of)f(message)g(transmission)g(as)g (consisting)h(of)f(three)g(phases:)131 2361 y(1.)22 b(Data)14 b(is)h(pulled)j(out)d(of)f(the)i(send)f(bu\013er)h(and)f(a)g(message)g(is)g (assem)o(bled)131 2457 y(2.)22 b(A)15 b(message)g(is)g(transfered)g(from)g (sender)g(to)g(receiv)o(er)131 2552 y(3.)22 b(Data)14 b(is)h(pulled)j(from)c (the)h(incoming)i(message)e(and)g(disassem)o(bled)i(in)o(to)e(the)g(receiv)o (e)h(bu\013er)166 2647 y(T)o(yp)q(e)k(matc)o(hing)f(has)h(to)e(b)q(e)j (observ)o(ed)e(at)g(eac)o(h)g(of)h(these)f(three)h(phases:)29 b(The)19 b(t)o(yp)q(e)h(of)f(eac)o(h)75 2704 y(v)m(ariable)d(in)f(the)f (sender)h(bu\013er)f(has)g(to)f(matc)o(h)h(the)g(t)o(yp)q(e)h(of)e(the)i (corresp)q(onding)g(en)o(try)f(in)h(the)f(bu\013er)p eop %%Page: 12 13 bop 75 -100 a Fo(12)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fo(descriptor;)d(the)g(t)o(yp)q(e)f (of)g(an)g(en)o(try)g(in)h(the)g(bu\013er)f(descriptor)h(used)f(to)g(send)h (a)f(message)g(has)g(to)g(matc)o(h)75 102 y(the)19 b(t)o(yp)q(e)f(of)g(the)h (corresp)q(onding)g(en)o(try)f(in)h(the)g(bu\013er)f(descriptor)h(used)g(to)f (receiv)o(e)h(the)g(message;)75 158 y(and)g(the)f(t)o(yp)q(e)g(of)g(eac)o(h)h (v)m(ariable)h(in)f(the)f(receiv)o(e)h(bu\013er)g(has)f(to)g(matc)o(h)g(the)g (corresp)q(onding)h(en)o(try)75 214 y(in)d(the)g(bu\013er)f(descriptor)h(for) f(this)h(bu\013er.)21 b(A)15 b(program)g(that)g(fails)h(to)f(observ)o(e)g (these)h(three)f(rules)h(is)75 271 y(erroneous.)166 328 y(T)l(o)f(de\014ne)i (more)e(precisely)i(t)o(yp)q(e)f(matc)o(hing,)f(w)o(e)g(need)h(to)f(deal)i (with)e(t)o(w)o(o)g(issues:)21 b(matc)o(hing)15 b(of)75 385 y(t)o(yp)q(es)h(of)g(the)h(host)f(language)g(with)h(bu\013er)g(descriptor)f (t)o(yp)q(es;)h(and)g(matc)o(hing)f(of)g(bu\013er)h(descriptor)75 441 y(t)o(yp)q(es)e(at)f(sender)h(and)f(receiv)o(er.)21 b(W)l(e)14 b(restrict)h(ourselv)o(es)g(to)f(the)g(case)h(where)g(sending)h(and)e (receiving)75 498 y(program)g(are)h(written)g(in)h(the)f(same)g(language.)166 555 y(The)e(t)o(yp)q(e)g(of)f(an)h(en)o(try)f(in)h(the)g(sender)g(bu\013er)g (descriptor)g(matc)o(hes)g(the)f(t)o(yp)q(e)h(of)f(an)h(en)o(try)f(in)i(the) 75 612 y(receiv)o(er)f(bu\013er)g(descriptor)g(is)h(they)e(ha)o(v)o(e)h(iden) o(tical)h(names:)19 b Fi(MPI)p 1234 612 15 2 v 17 w(INT)12 b Fo(matc)o(hes)g Fi(MPI)p 1580 612 V 17 w(INT)p Fo(,)g Fi(MPI)p 1766 612 V 17 w(REAL)75 668 y Fo(matc)o(hes)j Fi(MPI)p 326 668 V 17 w(REAL)p Fo(,)f(and)h(so)g(on.)166 725 y(The)k(t)o(yp)q(e)h(of)f(a)g (v)m(ariable)i(in)f(a)f(host)g(program)f(matc)o(hes)h(the)g(t)o(yp)q(e)h(of)e (the)i(en)o(try)f(in)h(a)f(bu\013er)75 782 y(descriptor)e(if)f(the)h(name)f (of)g(the)g(bu\013er)h(descriptor)f(en)o(try)g(t)o(yp)q(e)h(corresp)q(onds)f (to)g(the)g(basic)h(t)o(yp)q(e)f(of)75 838 y(the)i(host)f(program)f(en)o (try:)24 b(an)17 b(en)o(try)g(with)h(t)o(yp)q(e)g(name)f Fi(MPI)p 1177 838 V 17 w(INT)g Fo(matc)o(hes)g(a)g(F)l(ortran)g(v)m(ariable)h(of)75 895 y(t)o(yp)q(e)c Fi(INTEGER)p Fo(,)f(an)h(en)o(try)f(with)i(t)o(yp)q(e)f (name)g Fi(MPI)p 948 895 V 16 w(REAL)g Fo(matc)o(hes)f(a)h(F)l(ortran)f(v)m (ariable)i(of)f(t)o(yp)q(e)g Fi(REAL)p Fo(,)75 951 y(and)g(so)g(on.)20 b(There)14 b(is)h(one)f(exception)h(to)f(this)h(last)f(rule:)20 b(An)15 b(en)o(try)e(with)i(t)o(yp)q(e)f(name)g Fi(MPI)p 1683 951 V 17 w(BYTE)g Fo(can)75 1008 y(b)q(e)j(used)g(to)f(matc)o(h)g(an)o(y)g(b) o(yte)g(of)g(storage)f(\(on)i(a)f(b)o(yte-addressable)h(mac)o(hine\),)g (irresp)q(ectiv)o(e)h(of)e(the)75 1064 y(datat)o(yp)q(e)d(of)h(the)g(v)m (ariable)h(that)e(con)o(tains)h(this)g(b)o(yte.)19 b(The)c(v)m(alue)g(of)e (the)h(message)f(en)o(try)h(will)h(b)q(e)g(the)75 1121 y(binary)h(v)m(alue)g (of)f(the)g(corresp)q(onding)h(b)o(yte)f(in)h(memory)l(.)166 1178 y(These)g(t)o(yp)q(e)g(matc)o(hing)g(rules)h(imply)g(that)f(no)g(t)o(yp) q(e)g(con)o(v)o(ersion)g(o)q(ccur)g(during)h(the)f(MPI)g(com-)75 1235 y(m)o(unication.)21 b(If)15 b(a)f(v)m(alue)i(transfered)e(in)i(a)e (message)g(is)h(t)o(yp)q(ed)g(\(i.e.)20 b(of)14 b(a)h(t)o(yp)q(e)f (di\013eren)o(t)h(from)f Fi(BYTE)p Fo(\),)75 1291 y(then)g(the)g(t)o(yp)q(e)g (of)g(the)g(v)m(ariable)h(the)f(v)m(alue)h(is)f(read)g(from,)f(the)h(t)o(yp)q (e)g(of)g(the)g(v)m(alue)h(transferred)e(in)i(the)75 1347 y(message,)i(and)h (the)f(t)o(yp)q(e)g(of)g(the)h(v)m(ariable)g(where)g(the)f(v)m(alue)i(is)f (stored)f(at)f(the)i(receiv)o(er)g(all)g(matc)o(h,)75 1404 y(and)h(matc)o(h)e(the)i(t)o(yp)q(es)f(of)g(the)h(corresp)q(onding)g(en)o (tries)g(in)g(the)g(bu\013er)f(descriptors)h(at)f(sender)h(and)75 1460 y(receiv)o(er.)h(The)13 b(v)m(alue)g(stored)g(at)f(the)g(receiv)o(er)i (will)g(b)q(e)f(the)g(v)m(alue)h(loaded)f(from)f(the)h(sender.)19 b(If)13 b(sender)75 1517 y(and)h(receiv)o(er)g(run)g(in)h(an)f(iden)o(tical)h (en)o(vironmen)o(t)f(\(same)f(no)q(de)i(arc)o(hitecture,)f(same)f(compiler,)i (etc.\),)75 1573 y(then)i(the)f(binary)h(represen)o(tation)f(of)g(the)h(data) e(stored)h(at)g(the)g(receiv)o(er)h(is)g(iden)o(tical)h(to)e(the)g(binary)75 1630 y(represen)o(tation)g(of)g(the)h(data)e(loaded)i(from)f(the)g(sender.)24 b(Ho)o(w)o(ev)o(er,)15 b(if)i(sender)g(and)f(receiv)o(er)h(ha)o(v)o(e)f(a)75 1686 y(di\013eren)o(t)j(en)o(vironmen)o(t,)h(then)f(the)g(binary)g(represen)o (tation)g(of)f(the)h(v)m(alue)h(sen)o(t)e(will)j(b)q(e)e(con)o(v)o(erted)75 1743 y(to)g(the)h(binary)g(represen)o(tation)g(of)g(that)f(same)g(v)m(alue)i (in)g(the)f(receiv)o(er)g(en)o(vironmen)o(t.)34 b(Th)o(us,)21 b(if)f(a)75 1799 y(F)l(ortran)c(real)h(v)m(alue)h(is)f(transferred)g(\(from)f (a)g(v)m(ariable)j(of)d(t)o(yp)q(e)h Fi(REAL)f Fo(to)h(a)f(v)m(ariable)i(of)f (t)o(yp)q(e)g Fi(REAL)p Fo(,)75 1856 y(using)g(bu\013er)f(descriptor)h(en)o (tries)g(with)f(t)o(yp)q(e)h(state)e Fi(MPI)p 1082 1856 V 17 w(REAL)g Fo(at)h(sender)h(and)f(receiv)o(er\))h(from)f(a)g(64)75 1912 y(bit)g(mac)o(hine)g(to)e(a)h(32)f(bit)i(mac)o(hine,)g(then)f(the)g (binary)h(represen)o(tation)f(of)g(the)g(\015oating)g(p)q(oin)o(t)h(v)m(alue) 75 1968 y(sen)o(t)i(will)i(b)q(e)e(con)o(v)o(erted)g(from)f(64)h(bits)g(to)g (32)f(bits.)29 b(Suc)o(h)19 b(con)o(v)o(ersions)f(ma)o(y)f(in)o(tro)q(duce)i (rounding)75 2025 y(error)14 b(and)i(cause)f(o)o(v)o(er\015o)o(ws)f(or)h (under\015o)o(w)g(conditions.)166 2082 y(If)j(a)f(v)m(alue)h(sen)o(t)g(in)g (a)f(message)g(is)h(un)o(t)o(yp)q(ed)g(\(i.e.,)f(of)g(t)o(yp)q(e)h Fi(BYTE)p Fo(\),)e(then)i(it)g(m)o(ust)f(b)q(e)h(declared)75 2139 y(so)h(in)g(the)g(bu\013er)g(descriptor)h(of)e(b)q(oth)h(sender)h(and)f (receiv)o(er.)32 b(A)19 b(b)o(yte)g(is)g(mo)o(v)o(ed)g(from)f(sender)i(to)75 2195 y(receiv)o(er;)k(the)e(binary)f(represen)o(tation)h(of)e(the)i(b)o(yte)e (stored)h(at)g(the)g(receiv)o(er)h(is)f(iden)o(tical)i(to)e(the)75 2252 y(binary)14 b(represen)o(tation)f(of)f(the)h(b)o(yte)g(loaded)h(at)e (the)h(sender.)20 b(This)13 b(holds)h(true,)f(whether)g(sender)h(and)75 2308 y(receiv)o(er)i(run)f(in)h(the)g(same)f(or)f(in)i(distinct)h(en)o (vironmen)o(ts.)166 2366 y(Consider)f(the)f(follo)o(wing)h(examples.)75 2478 y Fi(!)48 b(FIRST)23 b(EXAMPLE)75 2534 y(INTEGER)g(A\(100\))75 2591 y(...)75 2647 y(CALL)g(MPI_CREATE_BUFFER\(BH,)e(MPI_PERSISTENT\))75 2704 y(CALL)i(MPI_ADD_CONTIGUOUS)f(\(BH,)h(A\(1\),)g(1,)h(MPI_CHAR\))p eop %%Page: 13 14 bop 75 -100 a Fh(1.5.)34 b(COMMUNICA)l(TION)16 b(BUFFERS)1001 b Fo(13)166 45 y(This)16 b(program)f(is)i(erroneous,)e(since)j(the)e(t)o(yp)q (e)g(of)f(the)h(bu\013er)g(descriptor)h(do)q(es)f(not)f(matc)o(h)h(the)75 102 y(t)o(yp)q(e)f(of)g(the)g(v)m(ariable)h(in)g(the)f(bu\013er.)20 b(The)15 b(b)q(eha)o(vior)h(or)e(this)i(program)e(is)h(unde\014ned.)22 b(A)15 b(desireable)75 158 y(b)q(eha)o(vior)e(is)h(for)e(that)g(error)g(to)h (b)q(e)g(detected)h(\(at)e(compile)i(time)f(or)g(run)g(time\))g(and)g(result) g(in)h(an)f(error)75 214 y(condition.)75 320 y Fi(!)48 b(SECOND)23 b(EXAMPLE)75 376 y(INTEGER)g(A\(100\))75 433 y(...)75 489 y(CALL)g (MPI_CREATE_BUFFER\(BH,)e(MPI_PERSISTENT\))75 546 y(CALL)i (MPI_ADD_CONTIGUOUS)f(\(BH,)h(A\(1\),)g(1,)h(MPI_BYTE\))166 651 y Fo(This)16 b(program)f(is)h(correct.)k(The)c(b)o(yte)g(that)e(will)k(b) q(e)e(transmitted)f(in)i(a)e(message)g(created)h(with)75 708 y(this)i(bu\013er)g(descriptor)g(is)g(the)g(\014rst)g(b)o(yte)f(in)i(the)e (storage)g(o)q(ccupied)j(b)o(y)d Fi(A\(1\))p Fo(.)27 b(The)18 b(v)m(alue)h(of)f(this)75 764 y(b)o(yte)d(is)h(implemen)o(tation)h(dep)q (enden)o(t,)f(and)g(will)h(v)m(ary)e(according)h(to)f(the)g(n)o(um)o(b)q(er)h (of)f(b)o(ytes)g(used)h(to)75 821 y(store)e(F)l(ortran)h(v)m(ariables)h(of)f (t)o(yp)q(e)g Fi(INTEGER)f Fo(and)i(the)f(b)o(yte)g(ordering)g(on)h(the)f (host)g(mac)o(hine.)75 926 y Fi(!)48 b(THIRD)23 b(EXAMPLE)75 983 y(INTEGER)g(A\(100\))75 1039 y(...)75 1096 y(CALL)g (MPI_CREATE_BUFFER\(BH1,)e(MPI_PERSISTENT\))75 1152 y(CALL)i (MPI_ADD_CONTIGUOUS)f(\(BH1,)h(A\(1\),)g(1,)h(MPI_INTEGER\))75 1208 y(...)75 1265 y(CALL)f(MPI_CREATE_BUFFER\(BH2,)e(MPI_PERSISTENT\))75 1321 y(CALL)i(MPI_ADD_CONTIGUOUS)f(\(BH2,)h(A\(1\),)g(4,)h(MPI_BYTE\))166 1427 y Fo(Consider)16 b(the)f(follo)o(wing)h(three)f(cases:)131 1520 y(1.)22 b(A)d(message)g(is)h(sen)o(t)f(using)h(bu\013er)f(descriptor)h (handle)h Fi(BH1)e Fo(and)g(receiv)o(ed)i(using)f(bu\013er)f(de-)189 1577 y(scriptor)e(handle)h Fi(BH2)p Fo(.)25 b(The)18 b(program)e(is)h (erroneous,)h(since)g(the)f(sender)h(and)f(receiv)o(er)h(bu\013er)189 1633 y(descriptors)g(don't)g(matc)o(h.)28 b(The)19 b(b)q(eha)o(vior)f(of)g (the)h(program)e(is)h(unde\014ned,)j(and)d(a)g(p)q(ossible)189 1690 y(\(desireable)e(but)g(unlik)o(ely\))h(b)q(eha)o(vior)e(is)h(for)f(an)g (error)f(condition)j(to)d(o)q(ccur.)131 1783 y(2.)22 b(Both)17 b(sender)h(and)g(receiv)o(er)g(use)g(bu\013er)g(descriptor)g(handle)h Fi(BH1)p Fo(.)26 b(The)18 b(program)f(is)h(correct,)189 1840 y(and)h(an)f(in)o(teger)h(v)m(alue)h(is)f(transmitted)g(from)f(sender)h(to)f (receiv)o(er.)31 b(The)19 b(v)m(alue)h(assigned)g(to)189 1896 y Fi(A\(1\))e Fo(at)g(the)g(receiv)o(er)i(is)f(iden)o(tical)h(to)e(the)h(v)m (alue)h(of)e Fi(A\(1\))g Fo(at)g(the)g(sender,)i(but)f(the)f(binary)189 1953 y(represen)o(tation)g(of)g(this)g(v)m(alue)i(ma)o(y)d(b)q(e)i (di\013eren)o(t,)g(if)f(sender)h(and)f(receiv)o(er)h(run)g(in)g(di\013eren)o (t)189 2009 y(en)o(vironmen)o(ts.)131 2103 y(3.)j(Both)17 b(sender)h(and)g (receiv)o(er)g(use)g(bu\013er)g(descriptor)g(handle)h Fi(BH2)p Fo(.)26 b(The)18 b(program)f(is)h(correct.)189 2159 y(Assume)13 b(that)f(b)q(oth)h(sender)h(and)f(receiv)o(er)h(use)f(four)g(b)o(ytes)f(to)h (store)f(in)o(tegers.)19 b(Then)14 b(the)f(v)m(alue)189 2215 y(stored)h(in)i Fi(A\(1\))e Fo(at)h(the)g(receiv)o(er)h(has)e(the)i(same)e (binary)i(represen)o(tation)f(as)g(the)g(v)m(alue)h(of)e Fi(A\(1)189 2272 y Fo(at)i(the)i(sender.)26 b(If)18 b(the)f(same)g(b)o(yte)g(ordering)h (and)g(the)f(same)g(enco)q(ding)i(of)e(in)o(teger)g(v)m(alues)h(is)189 2328 y(used)d(at)g(b)q(oth)g(ends)h(then)f Fi(A\(1\))f Fo(at)h(the)g(receiv)o (er)h(is)f(set)g(to)g(the)g(v)m(alue)h(of)f Fi(A\(1\))f Fo(at)h(the)g (sender;)189 2385 y(otherwise)g(the)g(same)g(32)g(bits)g(ma)o(y)g(enco)q(de)h (di\013eren)o(t)g(v)m(alues.)166 2554 y Fm(Implemen)o(tati)o(on)c(note:)166 2604 y Fl(The)i(curren)o(t)h(de\014nition)e(do)q(es)h(not)g(require)g (messages)g(to)f(carry)i(t)o(yp)q(e)f(information.)h(A)e(message)h(can)f(b)q (e)75 2654 y(comp)q(osed)h(and)h(sen)o(t)h(using)e(only)g(the)h(information)d (in)i(the)i(sender)g(bu\013er)g(descriptor,)g(and)e(can)h(b)q(e)h(receiv)o (ed)75 2704 y(and)c(stored)h(using)f(only)g(the)h(information)c(in)j(the)h (receiv)o(er)h(bu\013er)f(descriptor.)19 b(If)11 b(messages)i(are)f(sen)o(t)i (b)q(et)o(w)o(een)p eop %%Page: 14 15 bop 75 -100 a Fo(14)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fl(di\013eren)o(t)e(mac)o(hines)f (then)g(one)h(can)f(either)h(use)g(a)f(\\univ)o(ersal")g(data)g(enco)q(ding)g (for)g(messages,)g(use)h(kno)o(wledge)75 95 y(of)g(the)i(receiv)o(er)h(en)o (vironmen)o(t)d(in)g(order)i(to)f(con)o(v)o(ert)h(data)e(at)h(the)h(sender,)h (or)e(use)g(kno)o(wledge)g(of)g(the)g(sender)75 145 y(en)o(vironmen)o(t)f(in) g(order)i(to)f(con)o(v)o(ert)g(data)g(at)g(the)g(receiv)o(er.)23 b(In)15 b(either)h(case)g(the)f(lo)q(cal)g(bu\013er)g(descriptor)i(can)75 195 y(b)q(e)e(used)g(to)f(deriv)o(e)h(the)g(t)o(yp)q(es)g(of)f(the)g(v)n (alues)g(transferred.)21 b(Ho)o(w)o(ev)o(er,)15 b(additional)d(t)o(yp)q(e)j (information)c(carried)75 244 y(b)o(y)j(messages)g(can)g(pro)o(vide)f(b)q (etter)j(error)f(detection.)166 295 y(There)i(is)e(no)h(agreemen)o(t)f(y)o (et)h(on)g(the)g(need)h(for)e(a)g(de\014nition)h(and,)f(if)g(needed,)i(on)f (the)g(righ)o(t)f(de\014nition)75 344 y(of)j(t)o(yp)q(e)i(matc)o(hing)d(and)i (con)o(v)o(ersion)g(b)q(et)o(w)o(een)i(programs)d(written)h(in)g(di\013eren)o (t)h(languages.)33 b(The)19 b(natural)75 394 y(in)o(terpretation)f(of)g (\\send)h(from)d(C)i(and)g(receiv)o(e)h(in)e(F)m(ortran")h(is)g(either)h (\\send)g(from)d(C,)h(receiv)o(e)j(in)d(C)h(and)75 444 y(con)o(v)o(ert)11 b(to)g(F)m(ortran",)f(or)h(\\con)o(v)o(ert)g(to)f(F)m(ortran,)h(send)h(from)c (F)m(ortran)j(and)f(receiv)o(e)i(in)f(F)m(ortran".)16 b(Ho)o(w)o(ev)o(er,)11 b(the)75 494 y(corresp)q(ondence)j(b)q(et)o(w)o(een)e(F)m(ortran)e(t)o(yp)q (es)i(and)e(C)h(t)o(yp)q(es)g(is)g(implemen)o(tatio)o(n)d(dep)q(enden)o(t.)19 b(Th)o(us,)11 b(the)g(diagram)75 544 y(sho)o(wn)k(b)q(elo)o(w)f(is)h(not)f (comm)o(utativ)o(e,)e(and)j(the)g(t)o(w)o(o)f(in)o(terpretations)i(of)e(C)g (to)h(F)m(ortran)g(comm)o(unicati)o(on)d(ma)o(y)75 593 y(lead)i(to)f (di\013eren)o(t)i(results.)251 633 y(mac)o(hine)d(1)300 b(mac)o(hine)13 b(2)544 683 y(send)244 732 y(C)h(program)121 b Fc(!)i Fl(C)14 b(program)332 832 y Fc(#)463 b(#)544 882 y Fl(send)191 932 y(F)m(ortran)14 b(program)67 b Fc(!)j Fl(F)m(ortran)13 b(program)166 980 y(A)19 b(go)q(o)q(d)f(solution)g(w)o(ould)f(mak)o(e)g(this)i(diagram)d (comm)o(utativ)o(e.)29 b(Otherwise,)21 b(one)e(needs)h(to)e(pic)o(k)h(one)75 1030 y(of)c(the)i(t)o(w)o(o)e(paths)h(as)g(the)h(canonical)e(de\014nition)g (of)h(in)o(terlanguage)f(comm)o(unicatio)o(n.)21 b(If)16 b(the)g(upp)q(er)h (path)f(is)75 1080 y(c)o(hosen,)d(the)f(receiv)o(er)h(of)e(a)h(message)f (needs)i(to)f(kno)o(w)f(the)h(language)f(the)h(message)g(w)o(as)f(sen)o(t)i (from;)d(if)h(the)h(lo)o(w)o(er)75 1129 y(path)k(is)h(c)o(hosen,)g(the)g (sender)h(needs)g(to)e(kno)o(w)g(the)g(language)g(the)h(message)f(is)g (receiv)o(ed)i(in)o(to.)24 b(Rather)17 b(than)75 1179 y(struggling)c(with)h (these)i(issues)g(it)e(migh)o(t)e(b)q(e)j(simpler)e(to)h(let)g(the)h(b)q(eha) o(vior)f(of)g(C)g(to)g(F)m(ortran)g(comm)o(unication)75 1229 y(b)q(e)19 b(unde\014ned.)31 b(The)19 b(user)g(can)f(alw)o(a)o(ys)f(matc)o(h) g(language)g(at)h(sending)h(and)e(receiving)i(end)f(if)g(he)g(or)g(she)h(is) 75 1279 y(willing)12 b(to)h(struggle)i(with)e(the)i(issue)f(of)g(calling)e (MPI)i(C)g(functions)g(from)e(F)m(ortran)i(or)g(vice-v)o(ersa.)75 1507 y Fn(1.6)70 b(Receiv)n(e)20 b(Criteria)75 1609 y Fo(The)14 b(selection)h(of)e(a)g(message)g(b)o(y)g(a)h(receiv)o(e)g(op)q(eration)g(is)g (done)g(uniquely)h(according)f(to)f(the)h(v)m(alue)g(of)75 1665 y(the)e(message)g(en)o(v)o(elop)q(e.)20 b(The)13 b(receiv)o(e)g(op)q (eration)f(sp)q(eci\014es)j(an)d Fg(en)o(v)o(elop)q(e)i(pattern)p Fo(;)f(a)f(message)g(can)75 1722 y(b)q(e)17 b(receiv)o(ed)g(b)o(y)e(that)h (receiv)o(e)g(op)q(eration)g(only)h(if)f(its)g(en)o(v)o(elop)q(e)h(matc)o (hes)f(that)f(pattern.)21 b(A)16 b(pattern)75 1778 y(sp)q(eci\014es)e(v)m (alues)f(for)e(the)h Fi(source)p Fo(,)f Fi(tag)g Fo(and)h Fi(context)f Fo(\014elds)i(of)f(the)g(message)f(en)o(v)o(elop)q(e.)20 b(In)12 b(addition,)75 1835 y(the)17 b(v)m(alue)h(for)f(the)g Fi(dest)g Fo(\014eld)h(is)f(set,)g(implicitl)q(y)l(,)j(to)c(b)q(e)i(equal)g(to)e(the)h (receiving)i(pro)q(cess)e(id.)27 b(The)75 1891 y(receiv)o(er)13 b(ma)o(y)e(sp)q(ecify)i(a)f Fi(DONTCARE)e Fo(v)m(alue)j(for)f Fi(source)p Fo(,)f(or)g Fi(tag)p Fo(,)h(indicating)i(that)d(an)o(y)h(source)g (and/or)75 1948 y(tag)k(are)h(acceptable.)26 b(It)17 b(cannot)g(sp)q(ecify)h (a)f(DONTCARE)g(v)m(alue)h(for)e Fi(context)g Fo(or)h Fi(dest)p Fo(.)24 b(Th)o(us,)17 b(a)75 2004 y(message)d(can)h(b)q(e)h(receiv)o(ed)f(b)o (y)g(a)f(receiv)o(e)i(op)q(eration)f(only)g(if)g(it)g(is)h(addressed)f(to)f (the)h(receiving)h(task,)75 2061 y(has)h(a)g(matc)o(hing)g(con)o(text,)g(has) g(matc)o(hing)g(source)g(unless)i(source=DONTCARE)f(in)g(the)f(pattern,)75 2117 y(and)g(has)f(a)g(matc)o(hing)g(tag)g(unless)h(tag=DONTCARE)f(in)h(the)g (pattern.)22 b(DONTCARE)17 b(v)m(alues)h(for)75 2174 y(source)d(or)f(tag)f (are)h(sp)q(eci\014ed)j(via)e(a)f(named)h(constan)o(t)f(for)f(source)i(don)o (tcare)f(and)h(a)f(named)h(constan)o(t)75 2230 y(for)g(tag)f(don)o(tcare.)166 2287 y(The)21 b(length)g(of)f(the)h(receiv)o(ed)h(message)e(m)o(ust)h(b)q(e)g (less)g(or)f(equal)i(the)f(length)g(of)f(the)h(receiv)o(e)75 2343 y(bu\013er.)31 b(I.e.,)19 b(all)g(incoming)h(data)e(m)o(ust)h(\014t,)g (without)f(truncation,)i(in)o(to)f(the)f(receiv)o(e)i(bu\013er.)31 b(It)18 b(is)75 2400 y(erroneous)e(to)f(receiv)o(e)h(a)g(message)f(whic)o(h)i (length)f(exceed)h(the)f(receiv)o(e)h(bu\013er,)e(and)h(the)g(outcome)f(of)75 2456 y(program)f(where)i(this)f(o)q(ccurs)h(is)f(undetermined.)75 2601 y Fn(1.7)70 b(Comm)n(unication)20 b(Mo)r(de)75 2704 y Fo(A)15 b(send)h(op)q(eration)f(can)h(o)q(ccur)f(in)h(one)g(of)f(three)g(mo)q (des:)p eop %%Page: 15 16 bop 75 -100 a Fh(1.8.)34 b(COMMUNICA)l(TION)16 b(OBJECTS)1004 b Fo(15)75 45 y Fg(ST)l(AND)o(ARD)22 b Fo(The)15 b(send)h(ma)o(y)e(start)g (whether)i(or)e(not)h(a)g(matc)o(hing)g(receiv)o(e)h(has)f(b)q(een)i(p)q (osted.)75 140 y Fg(READ)o(Y)22 b Fo(The)15 b(send)h(ma)o(y)f(start)f(only)h (if)h(a)f(matc)o(hing)g(receiv)o(e)h(has)f(b)q(een)i(p)q(osted.)75 236 y Fg(SECURE)24 b Fo(The)19 b(send)h(op)q(eration)f(will)i(not)e(complete) h(successfully)h(un)o(til)g(it)e(is)h(guaran)o(teed)f(that)189 292 y(the)c(message)g(sen)o(t)g(will)h(b)q(e)g(receiv)o(ed.)166 387 y(A)i Fg(ready)j(send)d Fo(can)h(start)e(only)i(if)g(a)f(matc)o(hing)h (receiv)o(e)g(is)g(already)g(p)q(osted;)h(otherwise)f(the)75 444 y(op)q(eration)g(is)g(erroneous)g(and)g(its)g(outcome)f(is)h (unde\014ned.)33 b(In)19 b(some)f(systems,)h(this)g(will)i(allo)o(w)e(to)75 500 y(optimize)d(comm)o(unication)g(and)g(a)o(v)o(oid)f(a)f(hand-shaking)j (op)q(eration)e(that)g(is)g(otherwise)h(required.)166 557 y(A)d Fg(secure)h(send)f Fo(will)i(complete)e(if)h(a)f(matc)o(hing)g(receiv)o(e)h (is)f(p)q(osted,)g(and)h(the)f(receiv)o(e)h(op)q(eration)75 613 y(is)k(committed)g(to)g(receiv)o(e)g(the)g(message)g(sen)o(t)f(b)o(y)h (the)g(secure)g(send.)29 b(\(E.g.,)17 b(the)h(send)g(and)g(receiv)o(e)75 670 y(op)q(erations)12 b(ha)o(v)o(e)g(already)g(b)q(een)h(matc)o(hed,)f(the)g (receiv)o(e)h(op)q(eration)f(has)g(already)g(started)f(and)h(cannot)75 726 y(b)q(e)k(cancelled)h(an)o(ymore.\))166 783 y(A)d(receiv)o(e)i(op)q (eration)e(can)h(o)q(ccur)g(in)g(one)f(of)g(t)o(w)o(o)f(mo)q(des:)20 b Fg(ST)l(AND)o(ARD)13 b Fo(and)i Fg(SECURE)p Fo(.)g(A)75 840 y(secure)g(send)g(can)g(b)q(e)g(matc)o(hed)f(only)h(b)o(y)f(a)g Fg(secure)i(receiv)o(e)p Fo(;)e(a)g(program)f(where)i(a)f(secure)h(send)g (can)75 896 y(b)q(e)j(matc)o(hed)g(b)o(y)g(a)f(standard)h(receiv)o(e)g(is)h (erroneous)e(and)h(its)g(b)q(eha)o(vior)h(is)f(unde\014ned.)29 b(The)18 b(secure)75 953 y(receiv)o(e)e(can)f(b)q(e)h(p)q(osted)g(either)g(b) q(efore)f(or)g(after)f(the)i(matc)o(hing)f(send.)166 1009 y(A)f(standard)g (send)h(and)g(a)f(ready)g(send)h(can)f(b)q(oth)h(b)q(e)g(matc)o(hed)f(only)h (b)o(y)f(a)g Fg(standard)j(receiv)o(e)p Fo(;)75 1066 y(a)c(program)g(where)h (a)g(secure)g(receiv)o(e)h(can)f(matc)o(h)f(a)g(nonsecure)i(send)f(is)h (erroneous)e(and)h(its)g(b)q(eha)o(vior)75 1122 y(is)j(unde\014ned.)25 b(The)17 b(receiv)o(e)g(has)f(to)g(b)q(e)h(p)q(osted)f(ahead)h(of)f(the)g (send)h(for)f(a)g(ready)g(send)h(and)g(can)f(b)q(e)75 1179 y(p)q(osted)f(either)h(b)q(efore)g(or)f(after)f(the)h(send)h(is)g(p)q(osted)f (for)g(a)g(standard)f(send.)166 1312 y Fm(Implemen)o(tati)o(on)e(note:)166 1362 y Fl(The)18 b(curren)o(t)g(de\014nition)f(assumes)g(it)g(si)g(the)h (user)h(resp)q(onsibilit)o(y)e(to)g(prev)o(en)o(t)h(matc)o(hing)d(of)i(a)g (regular)75 1412 y(send)12 b(with)f(a)f(secure)j(receiv)o(e)f(and)f(vice-v)o (ersa.)18 b(Otherwise,)13 b(messages)e(need)h(b)q(e)f(tagged)g(as)g(standard) g(or)g(secure.)166 1462 y(Assume)18 b(an)f(implemen)o(tation)e(of)i(MPI)h (comm)o(unication)d(that)j(a)o(v)o(oids)f(retries.)31 b(What)18 b(proto)q(cols)g(can)75 1512 y(b)q(e)f(used?)27 b(Since)17 b(a)g(receiv)o(e)h(ma)o(y)c(ha)o(v)o(e)j(a)f(wildcard)g(source)i(\014eld,)f (the)g(\014rst)g(step)h(in)e(the)h(proto)q(col)g(m)o(ust)e(b)q(e)75 1561 y(executed)h(b)o(y)d(the)i(sender.)166 1612 y(F)m(or)e(a)h(ready)g (send,)h(this)f(\014rst)g(step)h(ma)o(y)d(in)o(v)o(olv)o(e)h(sending)h(the)h (data,)e(since)i(the)f(receiv)o(er)i(is)d(guaran)o(teed)75 1661 y(to)h(ha)o(v)o(e)f(space)i(to)f(store)h(it.)j(The)c(proto)q(col)g(is)f (send,)i(p)q(ossibly)e(follo)o(w)o(ed)g(b)o(y)g(an)h(ac)o(k-send.)166 1712 y(The)h(proto)q(col)g(for)g(secure)i(send)f(is)f(lik)o(ely)f(to)h(b)q(e) h(the)f(follo)o(wing:)j(The)e(sender)h(sends)f(a)f(request-to-send)75 1761 y(message.)j(The)13 b(receiv)o(er)i(stores)g(this)e(request)i(\(whic)o (h)f(con)o(tains)f(a)g(cop)o(y)g(of)g(the)h(message)f(en)o(v)o(elop)q(e\).)19 b(When)13 b(a)75 1811 y(matc)o(hing)g(receiv)o(e)k(is)e(p)q(osted,)h(the)g (receiv)o(er)g(sends)h(bac)o(k)e(an)g(ac)o(kno)o(wledge,)g(and)g(the)g (sender)i(no)o(w)e(sends)h(the)75 1861 y(message.)i(The)c(proto)q(col)g(is)g (\(req,)g(ac)o(k-req,)f(send\),)i(p)q(ossibly)e(follo)o(w)o(ed)g(b)o(y)g(an)h (ac)o(k-send.)166 1911 y(What)f(proto)q(col)f(is)h(used)h(for)f(standard)h (sends?)19 b(Either)14 b(the)f(second)h(proto)q(col)f(\(req,)h(ac)o(k-req,)f (send\))h(or)f(a)75 1961 y(mix)f(of)h(the)i(\014rst)f(and)g(the)h(second)g (\(e.g.)i(\014rst)e(used)g(for)e(short)i(messages,)e(second)i(for)f(long)f (messages\).)166 2011 y(When)j(the)g(receiv)o(er)h(do)q(es)g(not)e(wildcard)h (the)g(source)h(\014eld)f(then)g(the)g(proto)q(col)g(can)g(b)q(e)g(rev)o (ersed:)24 b(The)75 2061 y(receiv)o(er)15 b(sends)h(a)d(clear-to-send)i (message,)e(and)h(the)g(sender)i(can)e(pro)q(ceed)h(with)f(the)g(short)h (proto)q(col)e(if)g(it)h(has)75 2111 y(a)g(clear-to-send)g(when)h(the)f(send) h(is)f(executed.)166 2161 y(What)f(do)f(w)o(e)i(gain)e(from)f(the)j(\\secure) h(receiv)o(e"?)j(Some)12 b(optimizations)f(for)i(regular)g(receiv)o(es)i (\(e.g.,)d(in)g(a)75 2211 y(regular)i(receiv)o(e)i(of)e(a)g(short)g(message)g (the)h(receiv)o(er)h(kno)o(ws)e(the)h(short)g(proto)q(col)f(will)f(b)q(e)i (used,)g(and)f(need)h(not)75 2261 y(c)o(hec)o(k)g(for)e(a)h (request-to-send\).)20 b(Q:)14 b(is)g(that)g(enough)g(justi\014cation)f(for)h (the)g(secure)i(receiv)o(e?)75 2489 y Fn(1.8)70 b(Comm)n(unication)20 b(Ob)t(jects)75 2591 y Fo(An)g(opaque)g(comm)o(unication)h(ob)s(ject)e(iden)o (ti\014es)i(v)m(arious)g(prop)q(erties)f(of)g(a)f(comm)o(unication)i(op)q (er-)75 2647 y(ation,)i(suc)o(h)f(as)f(the)h(bu\013er)g(descriptor)g(that)f (is)h(asso)q(ciated)g(with)g(it,)h(its)f(con)o(text,)g(the)g(tag)f(and)75 2704 y(destination)16 b(parameters)f(to)g(b)q(e)h(used)g(for)e(a)h(send,)h (or)f(the)g(tag)g(and)h(source)f(parameters)g(to)f(b)q(e)i(used)p eop %%Page: 16 17 bop 75 -100 a Fo(16)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fo(for)g(a)g(receiv)o(e.)24 b(In)17 b(addition,)g(this)g(ob)s(ject)f(stores)g(information)g(ab)q(out)g (the)h(status)e(of)h(the)h(last)f(com-)75 102 y(m)o(unication)g(op)q(eration) g(that)e(w)o(as)h(p)q(erformed)g(with)h(this)g(ob)s(ject.)j(This)d(ob)s(ject) f(is)g(accessed)h(using)g(a)75 158 y(comm)o(unication)g(handle.)166 214 y(One)g(can)f(consider)h(comm)o(unication)g(op)q(erations)f(to)f(consist) i(of)f(the)g(follo)o(wing)g(sub)q(op)q(erations:)75 315 y Fg(INIT\(op)q (eration,)k(params,)d(handle\))24 b Fo(Pro)q(cess)15 b(pro)o(vides)f(all)i (relev)m(an)o(t)e(parameters)g(for)g(its)g(par-)189 371 y(ticipation)k(in)g (the)g(comm)o(unication)g(op)q(eration)f(\(t)o(yp)q(e)g(of)g(op)q(eration,)h (data)e(bu\013er,)i(tag,)e(par-)189 428 y(ticipan)o(ts,)f(etc.\).)k(An)d(ob)s (ject)e(is)i(created)f(that)g(iden)o(ti\014es)i(the)e(op)q(eration.)75 520 y Fg(ST)l(AR)l(T\(handle\))24 b Fo(The)16 b(comm)o(unication)g(op)q (eration)f(is)h(started)75 612 y Fg(COMPLETE\(handle\))25 b Fo(The)16 b(comm)o(unication)g(op)q(eration)f(is)h(completed.)75 704 y Fg(FREE\(handle\))25 b Fo(The)16 b(comm)o(unication)g(ob)s(ject,)e(and) h(asso)q(ciated)h(resources)f(are)g(freed.)75 804 y(Correct)f(in)o(v)o(o)q (cation)i(of)f(these)g(sub)q(op)q(erations)h(is)g(a)f(sequence)h(of)f(the)g (form)551 901 y Fg(INIT)i Fo(\()p Fg(ST)l(AR)l(T)g(COMPLETE)p Fo(\))1208 882 y Fb(\003)1246 901 y Fg(FREE)p Fd(:)75 998 y Fo(I.e.,)12 b(an)g(ob)s(ject)g(needs)h(b)q(e)f(created)h(b)q(efore)f(comm)o (unication)h(o)q(ccurs;)g(it)f(can)g(b)q(e)h(reused)g(only)g(after)e(the)75 1054 y(previous)16 b(use)f(has)f(completed;)i(and)f(it)g(needs)h(to)e(b)q(e)h (freed)h(ev)o(en)o(tually)f(\(of)g(course,)f(one)h(can)g(assume)75 1111 y(that)f(all)j(ob)s(jects)d(are)h(freed)h(at)e(program)g(termination,)h (b)o(y)h(default\).)166 1167 y(The)h(ab)q(o)o(v)o(e)f(scenario)i(p)q(ertains) f(to)f Fj(p)n(ersistent)g Fo(ob)s(jects.)24 b(One)18 b(can)f(also)f(create)h Fj(ephemer)n(al)g Fo(ob-)75 1223 y(jects.)23 b(Suc)o(h)17 b(ob)s(ject)e(p)q (ersists)i(only)g(un)o(til)g(the)f(comm)o(unication)h(op)q(eration)g(is)f (completed,)i(at)d(whic)o(h)75 1280 y(p)q(oin)o(t)i(it)g(is)g(destro)o(y)o (ed.)24 b(Th)o(us,)16 b(correct)h(in)o(v)o(o)q(cation)g(of)f(sub)q(op)q (erations)h(with)g(an)g(ephemeral)h(ob)s(ject)75 1336 y(is)e Fg(INIT)h(ST)l(AR)l(T)g(COMPLETE)p Fo(.)166 1393 y(A)i(user)h(ma)o(y)e (directly)j(in)o(v)o(ok)o(es)e(these)g(sub)q(op)q(erations.)33 b(This)20 b(w)o(ould)g(allo)o(w)f(to)g(amortize)g(the)75 1449 y(o)o(v)o(erhead)h(of)h(setting)f(up)i(a)e(comm)o(unication)h(o)o(v)o(er)f (man)o(y)h(successiv)o(e)h(uses)f(of)f(the)h(same)f(handle,)75 1506 y(and)h(allo)o(ws)h(to)e(o)o(v)o(erlap)h(comm)o(unication)h(and)f (computation.)38 b(Simpler)23 b(comm)o(unication)f(op)q(era-)75 1562 y(tions)c(com)o(bine)h(sev)o(eral)g(of)e(these)i(sub)q(op)q(erations)g (in)o(to)f(one)g(op)q(eration,)h(th)o(us)f(simplifying)i(the)f(use)75 1619 y(of)f(comm)o(unication)i(primitiv)o(es.)32 b(Th)o(us,)20 b(one)f(only)g(needs)h(to)e(sp)q(ecify)j(precisely)f(the)f(seman)o(tics)g(of) 75 1675 y(these)14 b(sub)q(op)q(erations)g(in)h(order)e(to)g(sp)q(ecify)i (the)f(seman)o(tics)f(of)h(MPI)f(p)q(oin)o(t)h(to)f(p)q(oin)o(t)h(comm)o (unication)75 1732 y(op)q(erations.)166 1788 y(W)l(e)22 b(sa)o(y)g(that)f(a)h (comm)o(unication)h(op)q(eration)f(\(send)h(or)e(receiv)o(e\))i(is)g Fg(p)q(osted)f Fo(once)h(a)f Fg(start)75 1844 y Fo(sub)q(op)q(eration)d(w)o (as)d(in)o(v)o(ok)o(ed;)j(the)f(op)q(eration)g(is)g Fg(completed)g Fo(once)g(the)g Fg(complete)g Fo(sub)q(op)q(eration)75 1901 y(completes.)29 b(A)18 b(send)h(and)f(a)f(receiv)o(e)i(op)q(eration)g Fg(matc)o(h)f Fo(if)g(the)g(receiv)o(e)h(pattern)f(sp)q(eci\014ed)i(b)o(y)e (the)75 1957 y(receiv)o(e)e(matc)o(hes)f(the)g(message)g(en)o(v)o(elop)q(e)h (created)f(b)o(y)h(the)f(send.)75 2078 y Fk(1.8.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)75 2247 y Fm(Discussion:)34 b Fl(This)14 b(section)g(has)g(not)g(y)o(et)g(b)q(een)h(appro)o(v)o(ed)166 2386 y Fo(An)j(ob)s(ject)e(for)h(a)g(send)h(op)q(eration)g(is)g(created)f(b)o (y)g(a)g(call)i(to)e Fg(MPI)p 1373 2386 16 2 v 18 w(INIT)p 1508 2386 V 18 w(SEND)p Fo(.)h(A)f(call)i(to)75 2442 y Fg(MPI)p 184 2442 V 18 w(INIT)p 319 2442 V 19 w(RECV)e Fo(is)i(similarly)h(used)e(for) f(creating)h(an)g(ob)s(ject)g(for)f(a)h(receiv)o(e)g(op)q(eration.)29 b(The)75 2499 y(creation)11 b(of)g(a)g(comm)o(unication)h(ob)s(ject)e(is)i(a) f(lo)q(cal)h(op)q(eration)f(that)g(need)h(not)f(in)o(v)o(olv)o(e)g(comm)o (unication)75 2555 y(with)16 b(a)e(remote)h(pro)q(cess.)166 2647 y Fg(MPI)p 275 2647 V 18 w(INIT)p 410 2647 V 19 w(SEND)k(\(handle,)h (bu\013er)p 931 2647 V 19 w(handle,)g(dest,)e(tag,)i(con)o(text,)f(mo)q(de,)g (p)q(ersis-)75 2704 y(tence\))p eop %%Page: 17 18 bop 75 -100 a Fh(1.8.)34 b(COMMUNICA)l(TION)16 b(OBJECTS)1004 b Fo(17)166 45 y(Creates)14 b(a)h(send)h(comm)o(unication)g(ob)s(ject.)j(P)o (arameters)14 b(are)75 142 y Fg(OUT)k(handle)24 b Fo(message)11 b(handle.)20 b(The)12 b(handle)h(should)f(not)f(b)q(e)i(asso)q(ciated)f(with) f(an)o(y)h(ob)s(ject)f(b)q(efore)189 198 y(the)k(call.)75 295 y Fg(IN)i(bu\013er)p 290 295 16 2 v 19 w(handle)24 b Fo(handle)16 b(to)f(send)h(bu\013er)f(descriptor)75 392 y Fg(IN)i(dest)23 b Fo(rank)15 b(in)h(con)o(text)e(of)h(destination)h(\(in)o(teger\))75 489 y Fg(IN)h(tag)24 b Fo(user)15 b(tag)f(for)h(messages)g(sen)o(t)g(with)g (this)h(handle)g(\(in)o(teger\))75 586 y Fg(IN)h(con)o(text)23 b Fo(con)o(text)15 b(of)f(messages)h(sen)o(t)g(with)h(this)f(handle)75 683 y Fg(IN)i(mo)q(de)23 b Fo(send)11 b(mo)q(de)f(\(state)f(t)o(yp)q(e,)i (with)g(three)f(v)m(alues:)18 b Fi(MPI)p 1186 683 15 2 v 17 w(STANDARD)p Fo(,)9 b Fi(MPI)p 1489 683 V 16 w(READY)h Fo(and)g Fi(MPI)p 1790 683 V 17 w(SECURE)p Fo(\))75 780 y Fg(IN)17 b(p)q(ersistence)23 b Fo(handle)h(p)q(ersistence)g(\(state)d(t)o(yp)q(e,)j(with)f(t)o(w)o(o)e(v)m (alues:)35 b Fi(MPI)p 1527 780 V 17 w(PERSISTENT)21 b Fo(and)189 836 y Fi(MPI)p 264 836 V 16 w(EPHEMERAL)p Fo(\))166 969 y Fg(MPI)p 275 969 16 2 v 18 w(INIT)p 410 969 V 19 w(RECV)d(\(handle,)i(bu\013er)p 937 969 V 19 w(handle,)g(source,)e(tag,)h(con)o(text,)g(mo)q(de,)g(p)q(er-)75 1025 y(sistence\))166 1118 y Fo(Create)c(a)f(receiv)o(e)i(handle.)22 b(P)o(arameters)13 b(are)75 1215 y Fg(OUT)18 b(handle)24 b Fo(message)11 b(handle.)20 b(The)12 b(handle)h(should)f(not)f(b)q(e)i(asso)q (ciated)f(with)f(an)o(y)h(ob)s(ject)f(b)q(efore)189 1271 y(the)k(call.)75 1368 y Fg(IN)i(bu\013er)p 290 1368 V 19 w(handle)24 b Fo(handle)16 b(to)f(receiv)o(e)h(bu\013er)f(descriptor.)75 1465 y Fg(IN)i(source)22 b Fo(rank)15 b(in)h(con)o(text)f(of)g(source,)g(or)f Fi(MPI)p 974 1465 15 2 v 17 w(ANY)p 1063 1465 V 17 w(SOURCE)g Fo(\(in)o(teger\).)75 1562 y Fg(IN)j(tag)24 b Fo(user)15 b(tag)f(for)h(messages)g(receiv)o(ed)h (with)g(this)f(handle,)h(or)f Fi(MPI)p 1341 1562 V 17 w(ANY)p 1430 1562 V 16 w(TAG)g Fo(\(in)o(teger\).)75 1659 y Fg(IN)i(con)o(text)23 b Fo(con)o(text)15 b(of)f(messages)h(receiv)o(ed)h(with)g(this)g(handle.)75 1756 y Fg(IN)h(mo)q(de)23 b Fo(receiv)o(e)16 b(mo)q(de)g(\(state)e(t)o(yp)q (e,)h(with)g(t)o(w)o(o)f(v)m(alues:)21 b Fi(MPI)p 1233 1756 V 17 w(STANDARD)p Fo(,)13 b(and)j Fi(MPI)p 1629 1756 V 17 w(SECURE)p Fo(\))75 1853 y Fg(IN)h(p)q(ersistence)23 b Fo(handle)h(p)q(ersistence)g (\(state)d(t)o(yp)q(e,)j(with)f(t)o(w)o(o)e(v)m(alues:)35 b Fi(MPI)p 1527 1853 V 17 w(PERSISTENT)21 b Fo(and)189 1909 y Fi(MPI)p 264 1909 V 16 w(EPHEMERAL)p Fo(\))166 2006 y(See)16 b(Section)g(1.4.2)d(for)i(a)g(discussion)i(of)d(source,)h(tag)g(and)g(con)o (text.)166 2139 y Fm(Discussion:)33 b Fl(I)13 b(ha)o(v)o(e)h(not)f(included)g (prop)q(osals)h(for)f(partially)e(sp)q(eci\014ed)k(message)e(handles,)g(that) h(some)75 2189 y(p)q(eoples)h(seem)e(to)h(desire.)166 2240 y(I)g(ha)o(v)o(e)f(merged)h(all)f(handle)g(setup)i(in)o(to)e(one)h(call.)75 2449 y Fk(1.8.2)55 b(Comm)n(unication)21 b(Start)75 2571 y Fg(MPI)p 184 2571 16 2 v 18 w(ST)l(AR)l(T\(handle\))75 2704 y(IN)c(handle)24 b Fo(comm)o(unication)16 b(handle)p eop %%Page: 18 19 bop 75 -100 a Fo(18)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 45 y Fo(The)i Fi(MPI)p 337 45 15 2 v 17 w(START)g Fo(function)h(starts)e(the)h(execution)i(of)d(a)h (comm)o(unication)h(op)q(eration)g(\(send)f(or)75 102 y(receiv)o(e\).)i(A)13 b(sender)h(should)g(not)f(up)q(date)h(the)g(send)g(bu\013er)f(after)g(a)g (send)g(op)q(eration)h(has)f(started)g(and)75 158 y(un)o(til)g(it)f(is)g (completed.)20 b(A)12 b(receiv)o(er)g(should)h(not)e(access)h(the)g(receiv)o (e)h(bu\013er)f(after)f(a)g(receiv)o(e)i(op)q(eration)75 214 y(w)o(as)18 b(started)g(and)h(un)o(til)h(it)f(is)h(completed.)32 b(A)19 b(program)e(that)i(do)q(es)g(not)f(satisfy)h(this)g(condition)h(is)75 271 y(erroneous)15 b(and)g(its)h(outcome)f(is)g(undetermined.)75 395 y Fk(1.8.3)55 b(Comm)n(unication)21 b(Completion)75 517 y Fg(MPI)p 184 517 16 2 v 18 w(W)-6 b(AIT)17 b(\()h(handle,)g(return)p 730 517 V 18 w(status)p 884 517 V 19 w(handle\))75 647 y(IN)f(handle)24 b Fo(comm)o(unication)16 b(handle)75 743 y Fg(OUT)i(return)p 357 743 V 19 w(handle)23 b Fo(handle)17 b(that)d(is)i(asso)q(ciated)f(with)h (return)f(status)f(ob)s(ject.)166 838 y(A)i(call)i(to)e Fg(MPI)p 467 838 V 18 w(W)-6 b(AIT)16 b Fo(returns)g(when)h(the)f(send)h(op)q(eration) g(iden)o(ti\014ed)h(b)o(y)f Fi(handle)e Fo(is)i(com-)75 894 y(plete.)24 b(The)17 b(completion)g(of)f(a)g(send)h(op)q(eration)g(indicates) h(that)d(the)i(sender)g(is)f(no)o(w)g(free)h(to)e(up)q(date)75 951 y(the)i(lo)q(cations)h(in)g(the)f(send)g(bu\013er,)h(or)e(an)o(y)h(other) f(lo)q(cation)i(that)e(can)i(b)q(e)f(referenced)h(b)o(y)f(the)g(send)75 1007 y(op)q(eration.)j(Ho)o(w)o(ev)o(er,)13 b(it)i(do)q(es)g(not)f(indicate)i (that)d(the)i(message)f(has)g(b)q(een)i(receiv)o(ed;)f(rather)f(it)h(ma)o(y) 75 1064 y(ha)o(v)o(e)g(b)q(een)h(bu\013ered)g(b)o(y)f(the)g(comm)o(unication) h(subsystem.)166 1120 y(The)g(completion)g(of)g(a)f(receiv)o(e)h(op)q (eration)g(indicates)h(that)e(the)h(receiv)o(er)g(is)g(no)o(w)f(free)h(to)f (access)75 1177 y(the)d(lo)q(cations)h(in)g(the)f(receiv)o(e)i(bu\013er,)e (whic)o(h)h(con)o(tain)f(the)h(receiv)o(ed)g(message,)f(or)g(an)o(y)f(other)h (lo)q(cation)75 1233 y(that)g(can)g(b)q(e)h(referenced)h(b)o(y)e(the)h (receiv)o(e)g(op)q(eration.)19 b(It)13 b(do)q(es)g(not)f(indicate)i(that)d (the)i(matc)o(hing)f(send)75 1290 y(op)q(eration)j(has)g(completed.)166 1347 y(The)i(call)g(returns)g(a)f(handle)i(to)e(an)g(opaque)h(ob)s(ject)f (that)g(con)o(tains)g(information)h(on)f(the)h(com-)75 1403 y(pleted)f(op)q(eration)g({)f(the)g Fg(return)i(status)e Fo(ob)s(ject.)166 1495 y Fg(MPI)p 275 1495 V 18 w(ST)l(A)l(TUS)i(\(handle,)h(\015ag,)g(return)p 974 1495 V 18 w(handle\))75 1626 y(IN)f(handle)24 b Fo(comm)o(unication)16 b(handle)75 1721 y Fg(OUT)i(\015ag)24 b Fo(logical)75 1817 y Fg(OUT)18 b(return)p 357 1817 V 19 w(handle)23 b Fo(handle)17 b(that)d(is)i(asso)q(ciated)f(with)h(return)f(status)f(ob)s(ject.)166 1912 y(A)21 b(call)h(to)f Fg(MPI)p 481 1912 V 18 w(ST)l(A)l(TUS)g Fo(returns)g Fi(flag=true)f Fo(if)h(the)g(op)q(eration)h(iden)o(ti\014ed)h(b) o(y)e Fi(handle)75 1968 y Fo(is)i(complete,)i(In)e(suc)o(h)g(case,)h(the)f (return)g(handle)h(p)q(oin)o(ts)f(to)f(an)g(opaque)h(ob)s(ject)f(that)g(con)o (tains)75 2025 y(information)14 b(on)f(the)h(completed)h(information.)k(It)14 b(returns)f Fi(flag=false)p Fo(,)g(otherwise.)19 b(In)14 b(suc)o(h)g(case,)75 2081 y(the)h(v)m(alue)i(of)d(the)i(return)f(handle)h(is)g(unde\014ned.)166 2138 y(A)e(call)i(to)e Fi(MPI)p 427 2138 15 2 v 16 w(WAIT)g Fo(blo)q(c)o(ks)h(only)g(the)g(executing)g(thread.)20 b(If)14 b(the)h(executing)g(pro)q(cess)g(is)g(m)o(ulti-)75 2194 y(threaded,)g(then)h (other)e(threads)h(within)i(the)e(pro)q(cess)h(can)f(b)q(e)h(sc)o(heduled)h (for)d(execution.)166 2251 y(The)21 b(use)h(of)f(a)f(blo)q(c)o(king)j(receiv) o(e)f(op)q(eration)f(\()p Fi(MPI)p 1095 2251 V 17 w(WAIT)p Fo(\))f(allo)o(ws)h(the)h(op)q(erating)f(system)g(to)75 2308 y(desc)o(hedule)e(the)e(blo)q(c)o(k)o(ed)i(thread)e(and)g(sc)o(hedule)i (another)d(thread)h(for)g(execution,)h(if)g(suc)o(h)f(is)h(a)o(v)m(ail-)75 2364 y(able.)k(The)16 b(use)g(of)f(a)g(non)o(blo)q(c)o(king)i(receiv)o(e)g (op)q(eration)e(\()p Fi(MPI)p 1152 2364 V 17 w(STATUS)p Fo(\))f(allo)o(ws)i (the)g(user)f(to)g(sc)o(hedule)75 2421 y(alternativ)o(e)g(activities)i (within)f(a)f(single)i(thread)e(of)f(execution.)166 2477 y(The)h(in)o(tended) h(implemen)o(tation)g(of)e Fi(MPI)p 894 2477 V 17 w(STATUS)g Fo(is)h(for)f(that)g(op)q(eration)h(to)f(return)h(as)f(so)q(on)g(as)75 2534 y(p)q(ossible.)22 b(Ho)o(w)o(ev)o(er,)14 b(if)i(rep)q(eatedly)h(called)g (for)d(an)i(op)q(eration)f(that)g(is)h(enabled,)h(it)e(m)o(ust)g(ev)o(en)o (tually)75 2590 y(succeed.)166 2647 y(The)23 b(return)f(status)f(ob)s(ject)h (for)g(a)g(send)h(op)q(eration)g(carries)f(no)h(information.)41 b(The)23 b(return)75 2704 y(status)17 b(ob)s(ject)h(for)f(a)h(receiv)o(e)h (op)q(eration)g(carries)f(information)g(on)g(the)g(source,)h(tag)e(and)i (length)f(of)p eop %%Page: 19 20 bop 75 -100 a Fh(1.8.)34 b(COMMUNICA)l(TION)16 b(OBJECTS)1004 b Fo(19)75 45 y(the)20 b(receiv)o(ed)h(message.)34 b(These)20 b(\014elds)h(are)e(required)i(b)q(ecause)g(the)f(receiv)o(e)h(op)q(eration)f (ma)o(y)f(ha)o(v)o(e)75 102 y(sp)q(eci\014ed)i Fi(DONTCARE)d Fo(in)i(either)f(source)g(or)g(tag)f(\014eld,)i(and)f(the)g(message)g(ma)o(y) f(ha)o(v)o(e)h(b)q(een)h(shorter)75 158 y(than)15 b(the)g(receiv)o(e)h (bu\013er.)166 250 y Fg(MPI)p 275 250 16 2 v 18 w(QUER)l(Y\()i(handle,)g (len,)f(source,)g(tag\))75 362 y(IN)g(handle)24 b Fo(handle)16 b(to)f(return)g(status)f(ob)s(ject)75 448 y Fg(OUT)k(len)24 b Fo(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(receiv)o(ed.)75 535 y Fg(OUT)i(source)23 b Fo(rank)14 b(of)h(message)g(sender)h(in)g(message) f(con)o(text)f(\(in)o(teger\).)75 622 y Fg(OUT)k(tag)24 b Fo(tag)14 b(of)h(receiv)o(ed)h(message)f(\(in)o(teger\).)166 734 y Fg(MPI)p 275 734 V 18 w(CREA)l(TE)p 510 734 V 19 w(ST)l(A)l(T\()i(handle\))166 825 y Fo(Creates)d(a)h(new)h(return)f(status)f(ob)s(ject.)75 902 y Fg(OUT)k(handle)24 b Fo(handle)16 b(to)f(return)g(status)g(ob)s(ject.) 166 1054 y Fm(Discussion:)166 1104 y Fl(The)e(use)g(of)f(a)h(return)g(status) h(ob)r(ject,)f(rather)g(than)g(a)f(list)g(of)g(parameters)h(ma)o(y)d (simplify)g(the)j(use)h(of)e(MPI)75 1154 y(routines,)17 b(if)f(the)g(v)n (alues)g(stored)i(in)e(the)h(ob)r(ject)g(are)f(seldom)f(c)o(hec)o(k)o(ed.)27 b(A)16 b(prede\014ned)j(return)e(status)g(ob)r(ject)75 1203 y(should)d(b)q(e)h(pro)o(vided,)f(to)g(ease)h(programming.)h(The)f(main)d (reason)j(for)f(the)h(use)g(of)f(a)g(return-status)i(ob)r(ject)f(is)75 1253 y(that)f(one)h(w)o(an)o(ts)f(to)g(b)q(e)g(able)g(to)g(use)h(the)g(same)e Ff(MPI)p 923 1253 14 2 v 15 w(WAIT)g Fl(and)h Ff(MPI)p 1186 1253 V 15 w(STATUS)f Fl(calls)h(for)g(c)o(hec)o(king)g(on)g(di\013eren)o(t)75 1303 y(t)o(yp)q(es)h(of)f(op)q(erations,)f(e.g.)19 b(b)q(oth)14 b(sends)i(and)d(receiv)o(es,)j(and)e(p)q(erhaps)h(new)g(op)q(erations)f(in)g (the)g(future.)20 b(These)75 1353 y(di\013eren)o(t)12 b(op)q(erations)g (return)g(di\013eren)o(t)g(information)c(\(alb)q(eit)j(w)o(e)h(could)f (decree)i(that)e(a)g(send)h(returns)h(the)f(same)75 1403 y(information)f(as)j (a)g(receiv)o(e\).)166 1453 y(The)i(issue)h(of)e(memory)f(allo)q(cation)g (for)i(return)p 939 1453 13 2 v 16 w(status)h(handles)f(is)g(not)g(y)o(et)g (solv)o(ed.)24 b(There)17 b(is)f(a)g(desire)75 1502 y(that)e(the)g(user)h (will)e(b)q(e)h(able)g(to)g(allo)q(cate)f(space)i(for)f(them)f(\(e.g.,)f(on)i (the)h(stac)o(k\).)166 1552 y(W)m(e)i(return)i(n)o(um)o(b)q(er)d(of)h(elemen) o(ts)g(receiv)o(ed,)j(rather)e(than)f(b)o(ytes)i(receiv)o(ed)f(b)q(ecause)h (\\n)o(um)o(b)q(er)e(of)g(ele-)75 1602 y(men)o(ts")g(is)h(closer)h(to)e(the)i (application)d(seman)o(tic)i(lev)o(el)f(\(user)i(do)q(es)g(not)f(need)h(to)f (b)q(e)g(a)o(w)o(are)g(of)f(the)i(size)g(of)75 1652 y(elemen)o(ts\),)g(and)f (is)f(more)h(in)o(v)n(arian)o(t.)28 b(The)19 b(return)g(status)g(ob)r(ject)g (returns)h(the)e(n)o(um)o(b)q(er)g(of)f Fm(elemen)o(ts)e Fl(re-)75 1702 y(ceiv)o(ed.)k(If)14 b(there)h(w)o(as)f(no)g(truncation,)g(then)h(this)f (is)g(equal)g(to)f(the)i(n)o(um)o(b)q(er)e(of)h(elemen)o(ts)g(sen)o(t.)19 b(On)c(the)f(other)75 1751 y(hand,)k(the)f(n)o(um)o(b)q(er)g(of)g(b)o(ytes)h (collected)g(form)d(the)j(sender)h(memory)m(,)c(the)j(n)o(um)o(b)q(er)e(of)h (b)o(ytes)h(sen)o(t)g(o)o(v)o(er)f(the)75 1801 y(wire,)12 b(and)g(the)g(n)o (um)o(b)q(er)f(of)g(b)o(ytes)i(stored)f(in)g(the)g(receiv)o(er)h(memory)d(ma) o(y)g(all)g(b)q(e)i(di\013eren)o(t,)h(in)e(a)h(heterogeneous)75 1851 y(en)o(vironmen)o(t.)23 b(The)17 b(n)o(um)o(b)q(er)e(of)g(elemen)o(ts)h (sen)o(t)h(can)f(b)q(e)g(computed)g(form)e(the)j(send)g(bu\013er)g (descriptor;)g(the)75 1901 y(n)o(um)o(b)q(er)c(of)g(elemen)o(ts)h(receiv)o (ed)h(can)f(b)q(e)h(computed)e(form)g(the)h(receiv)o(e)h(bu\013er)g (descriptor)g(and)f(the)g(length)g(\(in)75 1951 y(b)o(ytes\))h(of)e(the)i (receiv)o(ed)g(message.)75 2152 y Fk(1.8.4)55 b(Multiple)20 b(Completions)75 2238 y Fo(It)c(is)h(con)o(v)o(enien)o(t)f(to)g(b)q(e)g(able) h(to)f(w)o(ait)f(for)h(the)g(completion)h(of)e(an)o(y)h(or)g(all)h(the)f(op)q (erations)g(in)h(a)f(set,)75 2294 y(rather)g(than)g(ha)o(ving)h(to)e(w)o(ait) h(for)g(sp)q(eci\014c)i(message.)23 b(A)16 b(call)i(to)e Fi(MPI)p 1318 2294 15 2 v 16 w(WAITANY)g Fo(or)f Fi(MPI)p 1646 2294 V 17 w(STATUSANY)75 2350 y Fo(can)c(b)q(e)h(used)g(to)e(w)o(ait)h(for)f(the)i (completion)g(of)e(one)i(out)e(of)h(sev)o(eral)g(op)q(erations;)i(a)d(call)j (to)d Fi(MPI)p 1694 2350 V 17 w(WAITALL)75 2407 y Fo(can)15 b(b)q(e)h(used)g(to)f(w)o(ait)f(for)h(all)h(p)q(ending)h(op)q(erations)e(in)h (a)f(list.)166 2499 y Fg(MPI)p 275 2499 16 2 v 18 w(W)-6 b(AIT)l(ANY)16 b(\()i(list)p 677 2499 V 19 w(of)p 738 2499 V 19 w(handles,)g(len,)f(index,)g (return)p 1362 2499 V 19 w(handle\))166 2591 y Fo(Blo)q(c)o(ks)h(un)o(til)g (one)g(of)f(the)g(op)q(erations)h(asso)q(ciated)g(with)f(the)h(comm)o (unication)g(handles)h(in)f(the)75 2647 y(arra)o(y)13 b(has)i(completed.)21 b(Returns)15 b(the)g(index)h(of)e(that)g(handle)i(in)f(the)g(arra)o(y)l(,)f (and)g(returns)h(the)g(status)75 2704 y(of)g(that)f(op)q(eration)i(in)g(the)f (ob)s(ject)g(asso)q(ciated)g(with)h(the)f(return)p 1223 2704 14 2 v 16 w(handle.)21 b(The)16 b(parameters)e(are:)p eop %%Page: 20 21 bop 75 -100 a Fo(20)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fg(IN)h(list)p 225 45 16 2 v 20 w(of)p 287 45 V 19 w(handles)23 b Fo(list)16 b(of)f(handles)h (to)f(comm)o(unication)h(ob)s(jects.)75 144 y Fg(IN)h(len)23 b Fo(list)16 b(length)g(\(in)o(teger\))75 243 y Fg(OUT)i(index)23 b Fo(index)17 b(of)d(handle)j(for)d(op)q(eration)i(that)e(completed)i(\(in)o (teger\).)75 341 y Fg(OUT)i(return)p 357 341 V 19 w(handle)23 b Fo(handle)c(that)e(is)i(asso)q(ciated)f(with)g(return)g(status)f(ob)s (ject.)27 b(Set)18 b(to)f(return)189 398 y(status)d(of)h(op)q(eration)g(that) g(completed.)166 510 y(The)d(successful)i(execution)f(of)f Fi(MPI)p 784 510 15 2 v 16 w(WAITANY\(list)p 1088 510 V 16 w(of)p 1152 510 V 17 w(handles,)23 b(index,)g(return)p 1695 510 V 16 w(handle\))75 567 y Fo(has)15 b(the)g(same)g(e\013ect)g(as)g(the)h (successful)g(execution)h(of)d Fi(MPI)p 1139 567 V 17 w(WAIT\(handle[i],)22 b(return)p 1682 567 V 16 w(handle\))p Fo(,)75 623 y(where)17 b Fi(i)g Fo(is)g(the)g(v)m(alue)h(returned)f(b)o(y)g Fi(index)f Fo(and)g Fi(handle[i])g Fo(is)h(the)g Fi(i)p Fo(-th)g(handle)h(in)f(the)g (list,)h(and)75 679 y(the)d(cancelation)i(of)d(all)j(remaining)f(w)o(ait)f (op)q(erations.)166 737 y(If)21 b(more)g(then)g(one)g(op)q(eration)h(is)f (enabled)i(and)e(can)g(terminate,)h(one)f(is)h(arbitrarily)g(c)o(hosen)75 794 y(\(sub)s(ject)15 b(to)f(the)i(restrictions)f(on)g(op)q(eration)h (termination)f(order,)g(and)g(fairness,)g(see)h(Section)g(1.13\).)166 851 y Fi(MPI)p 241 851 V 17 w(WAITANY)23 b(\()g(list)p 592 851 V 17 w(of)p 657 851 V 17 w(handles,)g(len,)g(index,)g(return)p 1319 851 V 16 w(status)p 1479 851 V 17 w(handle\))14 b Fo(is)147 951 y Fi({MPI_WAIT)22 b(\(handle[0],)h(return_handle\);)e(index)j(=)f(0})h (||)f(...)75 1008 y(||)147 1064 y({MPI_WAIT)f(\(handle[len-1],)g (return_handle\);)g(index)h(=)h(len-1})166 1163 y Fo(\(\\)p Fe(jj)p Fo(")13 b(indicates)k(c)o(hoice;)f(one)f(of)g(the)g(alternativ)o(es)h (is)f(c)o(hosen,)g(nondeterministically)m(.\))166 1256 y Fg(MPI)p 275 1256 16 2 v 18 w(ST)l(A)l(TUSANY)h(\()i(list)p 739 1256 V 20 w(of)p 801 1256 V 19 w(handles,)f(len,)h(index,)f(return)p 1425 1256 V 18 w(handle\))166 1349 y Fo(Causes)12 b(either)h(one)f(or)g(none) h(of)f(the)g(op)q(erations)g(asso)q(ciated)h(with)f(the)h(comm)o(unication)g (handles)75 1406 y(to)i(return.)23 b(In)17 b(the)f(former)f(case,)h(it)g(has) g(the)g(same)g(return)g(seman)o(tics)g(as)g(a)g(call)h(to)e Fi(MPI)p 1664 1406 15 2 v 17 w(WAIT)p 1777 1406 V 17 w(ANY)p Fo(.)75 1462 y(In)j(the)g(later)g(case,)f(it)h(returns)g(a)f(v)m(alue)i(of)e (-1)g(in)i Fi(index)e Fo(and)h Fi(return)p 1343 1462 V 16 w(handle)f Fo(is)h(unde\014ned.)29 b(The)75 1518 y(parameters)14 b(are:)75 1631 y Fg(IN)j(list)p 225 1631 16 2 v 20 w(of)p 287 1631 V 19 w(handles)23 b Fo(list)16 b(of)f(handles)h(to)f(comm)o(unication)h(ob)s (jects.)75 1730 y Fg(IN)h(len)23 b Fo(list)16 b(length)g(\(in)o(teger\))75 1828 y Fg(OUT)i(index)23 b Fo(index)18 b(of)e(handle)h(for)f(op)q(eration)h (that)e(completed,)j(or)e(-1)g(if)h(none)f(completed)i(\(in)o(te-)189 1885 y(ger\).)75 1983 y Fg(OUT)g(return)p 357 1983 V 19 w(handle)23 b Fo(handle)c(that)e(is)i(asso)q(ciated)f(with)g(return)g(status)f(ob)s (ject.)27 b(Set)18 b(to)f(return)189 2040 y(status)d(of)h(op)q(eration)g (that)g(completed,)g(if)h(an)o(y;)f(unde\014ned)i(when)e Fi(index)24 b(=)f(-1)p Fo(.)166 2188 y Fg(MPI)p 275 2188 V 18 w(W)-6 b(AIT)l(ALL\(list)p 624 2188 V 20 w(of)p 686 2188 V 19 w(handles,)18 b(len,)f(list)p 1077 2188 V 20 w(of)p 1139 2188 V 19 w(return)p 1304 2188 V 18 w(handles\))166 2281 y Fo(Blo)q(c)o(ks)c(un)o(til)h(all)g(comm)o (unication)g(op)q(erations)f(asso)q(ciated)g(with)g(handles)h(in)g(the)f (list)h(complete,)75 2337 y(and)h(return)h(the)f(status)f(of)h(all)h(these)f (op)q(erations.)20 b(The)c(parameters)e(are:)75 2450 y Fg(IN)j(list)p 225 2450 V 20 w(of)p 287 2450 V 19 w(handles)23 b Fo(list)16 b(of)f(handles)h(to)f(comm)o(unication)h(ob)s(jects.)75 2548 y Fg(IN)h(len)23 b Fo(lists)16 b(length)g(\(in)o(teger\))75 2647 y Fg(OUT)i(list)p 280 2647 V 20 w(of)p 342 2647 V 19 w(return)p 507 2647 V 18 w(handles)24 b Fo(Must)18 b(ha)o(v)o(e)h(the)g(same)g(length)g (as)g(the)g(\014rst)g(list.)32 b(Eac)o(h)19 b(return)189 2704 y(status)12 b(ob)s(ject)g(is)i(set)e(to)h(the)g(return)f(status)h(of)f(the)h (corresp)q(onding)h(op)q(eration)f(in)h(the)f(\014rst)g(list.)p eop %%Page: 21 22 bop 75 -100 a Fh(1.9.)34 b(BLOCKING)17 b(COMMUNICA)l(TION)966 b Fo(21)166 45 y Fm(Discussion:)166 96 y Fl(The)14 b(fairness)f(requiremen)o (t)g(giv)o(en)g(in)g(Section)g(1.13)f(implies)g(that)h(the)h(use)g(of)e Ff(WAIT)p 1525 96 14 2 v 15 w(ANY)h Fl(cannot)g(lead)g(to)75 145 y(starv)n(ation:)21 b(If)15 b(the)h(sending)g(pro)q(cess)h(has)f(issued)g (a)f(send)i(complete)d(op)q(eration,)i(and)f(the)h(receiving)g(pro)q(cess)75 195 y(rep)q(eatedly)21 b(p)q(ost)f(a)g(receiv)o(e)h(for)e(the)h(message)g (sen)o(t,)h(then)g(that)e(message)h(m)o(ust)f(b)q(e)h(ev)o(en)o(tually)f (receiv)o(ed.)75 245 y(Section)c(1.13)e(has)i(not)f(y)o(et)h(b)q(een)h (discussed.)21 b(The)15 b(fairness)g(requiremen)o(t)g(can)f(b)q(e)i(attac)o (k)o(ed)e(either)i(for)e(b)q(eing)75 295 y(to)q(o)j(w)o(eak)h(\(ev)o(en)o (tually)f(is)g(not)g(go)q(o)q(d)g(enough\),)h(or)g(to)q(o)f(strong)h(\(hard)f (to)g(implemen)o(t\).)26 b(In)18 b(the)g(later)f(case,)75 345 y(if)d(the)h(requiremen)o(t)g(of)f(fairness)h(is)g(drop)q(ed)h(from)d(MPI)i (implem)o(en)o(tations,)d(then)j(some)f(mec)o(hanism)f(need)i(b)q(e)75 394 y(pro)o(vided)k(to)g(the)g(user)h(to)f(ac)o(hiev)o(e)g(fairness)h(b)o(y)e (his)h(or)g(her)h(o)o(wn)e(devices.)35 b(One)19 b(suc)o(h)h(prop)q(osal)f(is) g(to)f(b)q(e)75 444 y(able)c(to)g(sp)q(ecify)g(a)g(rotating)g(priorit)o(y)f (order)i(on)f(the)g(op)q(erations)h(p)q(osted)g(b)o(y)f(a)f Ff(MPI)p 1431 444 V 15 w(WAITANY)p Fl(:)g(searc)o(h)i(the)f(list)75 494 y(sequen)o(tially)m(,)e(starting)i(form)e(a)i(user)h(sp)q(eci\014ed)g(p)q (osition.)75 724 y Fn(1.9)70 b(Blo)r(c)n(king)21 b(Comm)n(unication)75 827 y Fo(Blo)q(c)o(king)g(send)e(and)h(receiv)o(e)g(op)q(erations)f(com)o (bine)h(all)h(comm)o(unication)f(sub)q(op)q(erations)g(in)o(to)f(one)75 883 y(call.)h(The)12 b(op)q(eration)h(returns)f(only)g(when)h(the)f(comm)o (unication)h(completes)g(and)f(no)g(comm)o(unication)75 940 y(ob)s(ject)19 b(p)q(ersists)i(after)e(the)i(call)g(completed.)35 b(Ho)o(w)o(ev)o(er,)20 b(the)g(bu\013er)g(descriptor)h(ob)s(ject)e(needs)i(b) q(e)75 996 y(created)15 b(ahead)h(of)e(the)i(call.)166 1053 y(W)l(e)f(use)h(the)f(follo)o(wing)h(naming)g(con)o(v)o(en)o(tion)f(for)f (suc)o(h)i(op)q(erations:)782 1116 y Fa(2)782 1189 y(6)782 1216 y(4)833 1156 y Fe(\000)831 1213 y Fg(R)836 1269 y(S)891 1116 y Fa(3)891 1189 y(7)891 1216 y(5)926 1141 y(")975 1185 y Fg(SEND)971 1241 y(RECV)1143 1141 y Fa(#)166 1375 y Fo(The)i(\014rst)f (letter)h(\(v)o(oid,)g Fg(R)g Fo(or)g Fg(S)p Fo(\))f(indicates)j(the)d(start) g(mo)q(de)h(\(standard,)g(ready)l(,)g(or)f(secure\).)75 1432 y(Only)f(t)o(w)o(o)e(of)h(the)g(com)o(binations)h(\(standard)f(and)g (secure\))g(are)g(meaningful)i(for)d(receiv)o(es.)166 1524 y Fg(MPI)p 275 1524 16 2 v 18 w(SEND)k(\(bu\013er)p 609 1524 V 19 w(handle,)g(dest,)f(tag,)h(con)o(text\))d Fo(is)75 1657 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h(dest,)h(tag,)g(context,)g (MPI_REGULAR,)f(MPI_EPHEMERAL\))75 1713 y(MPI_START\(handle\))75 1770 y(MPI_WAIT\(handle,)g(null\))166 1902 y Fg(MPI)p 275 1902 V 18 w(RSEND)c(\(bu\013er)p 648 1902 V 19 w(handle,)g(dest,)f(tag,)h(con)o (text\))d Fo(is)75 2034 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h (dest,)h(tag,)g(context,)g(MPI_READY,)g(MPI_EPHEMERAL\))75 2091 y(MPI_START\(handle\))75 2147 y(MPI_WAIT\(handle,)f(null\))166 2279 y Fg(MPI)p 275 2279 V 18 w(SSEND)c(\(bu\013er)p 638 2279 V 19 w(handle,)g(dest,)f(tag,)h(con)o(text\))d Fo(is)75 2412 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h(dest,)h(tag,)g(context,)g (MPI_SECURE,)g(MPI_EPHEMERAL\))75 2468 y(MPI_START\(handle\))75 2525 y(MPI_WAIT\(handle,)f(null\))166 2657 y Fg(MPI)p 275 2657 V 18 w(RECV\(bu\013er)p 598 2657 V 19 w(handle,)c(source,)e(tag,)i(con)o (text,)f(return)p 1427 2657 V 19 w(handle\))f Fo(is)p eop %%Page: 22 23 bop 75 -100 a Fo(22)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fi(MPI_INIT_RECV\(handle,)21 b(buffer_handle,)h(source,)h(tag,)g(context,)g(MPI_STANDARD,)f (MPI_EPHEMERAL\))75 102 y(MPI_START\(handle\))75 158 y (MPI_WAIT\(handle,return_han)o(dle\))166 288 y Fg(MPI)p 275 288 16 2 v 18 w(SRECV\(bu\013er)p 627 288 V 19 w(handle,)c(source,)e(tag,)i (con)o(text,)f(return)p 1456 288 V 19 w(handle\))f Fo(is)75 417 y Fi(MPI_INIT_RECV\(handle,)21 b(buffer_handle,)h(source,)h(tag,)g (context,)g(MPI_SECURE,)f(MPI_EPHEMERAL\))75 474 y(MPI_START\(handle\))75 530 y(MPI_WAIT\(handle,return_han)o(dle\))166 700 y Fm(Implemen)o(tati)o(on) 12 b(note:)166 750 y Fl(While)f(these)i(functions)f(can)h(b)q(e)f(implemen)o (ted)e(via)h(calls)h(to)f(functions)h(that)g(implemen)o(t)e(sub)q(op)q (erations,)75 800 y(as)k(describ)q(ed)h(in)e(this)g(subsection,)i(an)e (e\016cien)o(t)h(implemen)o(tatio)o(n)d(ma)o(y)g(optimize)h(a)o(w)o(a)o(y)h (these)i(m)o(ultiple)c(calls,)75 850 y(pro)o(vided)j(it)f(do)q(es)i(not)f(c)o (hange)g(the)g(b)q(eha)o(vior)g(of)f(correct)j(programs.)75 1076 y Fn(1.10)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)75 1178 y Fo(Non)o(blo)q(c)o(king)f(send)f(and)f(receiv)o(e)i(op)q(erations)f (com)o(bine)g(the)g(\014rst)f(t)o(w)o(o)f(sub)q(op)q(erations)i(\()p Fi(INIT)f Fo(and)75 1234 y Fi(START)p Fo(\))c(in)o(to)g(one)h(call.)22 b(They)16 b(use)g(ephemeral)h(comm)o(unication)f(ob)s(jects,)f(so)g(that)g (the)h(op)q(eration)f(is)75 1291 y(completed,)g(and)g(the)g(asso)q(ciated)f (resources)h(are)f(freed,)h(b)o(y)f(using)h(one)g(of)f(the)h(functions)g Fi(MPI)p 1742 1291 15 2 v 17 w(WAIT,)75 1347 y(MPI)p 150 1347 V 17 w(STATUS,)23 b(MPI)p 430 1347 V 16 w(WAITANY,)g(MPI)p 733 1347 V 17 w(STATUSANY)p Fo(,)c(or)i Fi(MPI)p 1132 1347 V 17 w(WAITALL)p Fo(.)e(Here,)j(to)q(o,)g(a)f(bu\013er)g(ob)s(ject)75 1403 y(has)15 b(to)g(b)q(e)g(created)h(ahead)f(of)g(the)g(comm)o(unication)h (initiation)h(op)q(eration.)166 1460 y(W)l(e)d(use)h(the)f(same)g(naming)g (con)o(v)o(en)o(tion)g(as)g(for)f(blo)q(c)o(king)j(op)q(erations:)j(a)14 b(pre\014x)h(of)e Fg(R)i Fo(\()p Fg(S)p Fo(\))e(indi-)75 1516 y(cates)h(the)g Fi(READY)g Fo(\()p Fg(SECURE)p Fo(\))g(mo)q(de.)20 b(In)15 b(addition,)g(a)f(pre\014x)h(of)f Fg(I)g Fo(is)g(used)h(to)f (indicate)i Fj(imme)n(diate)75 1573 y Fo(\(i.e.,)f(non)o(blo)q(c)o(king\))h (execution.)166 1665 y Fg(MPI)p 275 1665 16 2 v 18 w(ISEND)i(\(handle,)g (bu\013er)p 812 1665 V 19 w(handle,)g(dest,)f(tag,)g(con)o(text\))f Fo(is)75 1794 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h(dest,)h(tag,) g(context,)g(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1851 y(MPI_START\(handle\)) 166 1980 y Fg(MPI)p 275 1980 V 18 w(IRSEND)c(\(handle,)g(bu\013er)p 851 1980 V 19 w(handle,)g(dest,)f(tag,)h(con)o(text\))d Fo(is)75 2110 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h(dest,)h(tag,)g (context,)g(MPI_READY,)g(MPI_EPHEMERAL\))75 2167 y(MPI_START\(handle\))166 2296 y Fg(MPI)p 275 2296 V 18 w(ISSEND)18 b(\(handle,)g(bu\013er)p 841 2296 V 19 w(handle,)g(dest,)f(tag,)g(con)o(text\))f Fo(is)75 2426 y Fi(MPI_INIT_SEND\(handle,)21 b(buffer_handle,)h(dest,)h(tag,)g (context,)g(MPI_SECURE,)g(MPI_EPHEMERAL\))75 2482 y(MPI_START\(handle\))166 2612 y Fg(MPI)p 275 2612 V 18 w(IRECV\(handle,)13 b(bu\013er)p 796 2612 V 19 w(handle,)g(source,)f(tag,)h(con)o(text,)g(return)p 1607 2612 V 18 w(status)p 1761 2612 V 19 w(handle\))75 2668 y Fo(is)p eop %%Page: 23 24 bop 75 -100 a Fh(1.11.)34 b(CONTIGUOUS)16 b(BLOCK)g(SENDING)g(OPERA)l(TIONS) 586 b Fo(23)75 45 y Fi(MPI_INIT_RECV\(handle,)21 b(buffer_handle,)h(source,)h (tag,)g(context,)g(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 102 y (MPI_START\(handle\))166 238 y Fg(MPI)p 275 238 16 2 v 18 w(ISRECV\(handle,) 13 b(bu\013er)p 825 238 V 19 w(handle,)g(source,)f(tag,)h(con)o(text,)g (return)p 1636 238 V 18 w(status)p 1790 238 V 19 w(handle\))75 294 y Fo(is)75 432 y Fi(MPI_INIT_RECV\(handle,)21 b(buffer_handle,)h(source,) h(tag,)g(context,)g(MPI_SECURE,)f(MPI_EPHEMERAL\))75 488 y (MPI_START\(handle\))75 641 y Fn(1.11)70 b(Con)n(tiguous)23 b(Blo)r(c)n(k)f(Sending)h(Op)r(erations)75 746 y Fo(The)e(most)f(frequen)o(t) h(t)o(yp)q(e)f(of)h(bu\013er)g(used)g(is)g(a)g(bu\013er)f(with)h(one)g(con)o (tiguous)g(comp)q(onen)o(t.)37 b(W)l(e)75 802 y(sp)q(ecialize)17 b(the)d(functions)h(in)g(the)f(t)o(w)o(o)f(previous)i(subsections)g(to)f (this)g(case,)g(th)o(us)g(a)o(v)o(oiding)h(the)f(need)75 859 y(for)h(the)g(creation)h(of)f(a)g(bu\013er)g(descriptor)h(ob)s(ject.)k(W)l(e) c(use)f(the)h(same)f(naming)h(sc)o(heme)g(used)g(in)g(the)75 915 y(previous)g(subsections,)g(and)f(app)q(end)h(a)f Fg(C)g Fo(in)h(the)g(function)g(name,)e(for)h Fi(CONTIGUOUS)p Fo(.)166 1009 y Fg(MPI)p 275 1009 V 18 w(SENDC)i(\(start,)h(len,)f(t)o(yp)q(e,)g (dest,)g(tag,)h(con)o(text\))e Fo(is)75 1146 y Fi (MPI_CREATE_BUFFER\(buffer_h)o(andle,)k(MPI_EPHEMERAL\))75 1203 y(MPI_ADD\(buffer_handle,)h(start,)i(len,)g(type\))75 1259 y(MPI_SEND)g(\(buffer_handle,)f(len,)h(dest,)g(tag,)g(context\))166 1395 y Fg(MPI)p 275 1395 V 18 w(RSENDC)18 b(\(handle,)g(start,)f(len,)h(t)o (yp)q(e,)e(dest,)h(tag,)h(con)o(text\))e Fo(is)75 1533 y Fi (MPI_CREATE\(buffer_handle,)k(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 1590 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 1646 y(MPI_RSEND)g(\(buffer_handle,)e(len,)j(dest,)f(tag,)g(context\))166 1782 y Fg(MPI)p 275 1782 V 18 w(SSENDC)18 b(\(handle,)g(start,)f(len,)g(t)o (yp)q(e,)g(dest,)g(tag,)h(con)o(text\))e Fo(is)75 1920 y Fi (MPI_CREATE\(buffer_handle,)k(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 1976 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 2033 y(MPI_SSEND)g(\(buffer_handle,)e(len,)j(dest,)f(tag,)g(context\))166 2169 y Fg(MPI)p 275 2169 V 18 w(RECV)o(C)18 b(\(start,)i(len,)g(t)o(yp)q(e,)f (source,)g(tag,)h(con)o(text,)f(return)p 1533 2169 V 19 w(status)p 1688 2169 V 19 w(handle\))75 2225 y Fo(is)75 2363 y Fi (MPI_CREATE\(buffer_handle,)h(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 2419 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 2476 y(MPI_RECV\(buffer_handle,)e(source,)i(tag,)g(context,)g (return_status_handle\))166 2612 y Fg(MPI)p 275 2612 V 18 w(SRECV)o(C)14 b(\(start,)i(len,)f(t)o(yp)q(e,)g(source,)g(tag,)h(con)o(text,)f(return)p 1533 2612 V 19 w(status)p 1688 2612 V 19 w(handle\))75 2668 y Fo(is)p eop %%Page: 24 25 bop 75 -100 a Fo(24)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fi(MPI_CREATE\(buffer_handle,)k (MPI_BUFFER,)j(MPI_EPHEMERAL\))75 102 y(MPI_ADD\(buffer_handle,)e(start,)i (len,)g(type\))75 158 y(MPI_SRECV\(buffer_handle,)e(source,)h(tag,)i (context,)f(return_status_handle\))166 281 y Fg(MPI)p 275 281 16 2 v 18 w(ISENDC)17 b(\(handle,)i(start,)e(len,)g(t)o(yp)q(e,)g(dest,)g (tag,)h(con)o(text\))d Fo(is)75 404 y Fi(MPI_CREATE\(buffer_handle,)20 b(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 460 y(MPI_ADD\(buffer_handle,)e(start,)i (len,)g(type\))75 517 y(MPI_ISEND\(handle,)f(buffer_handle,)f(dest,)j(tag,)f (context\))166 640 y Fg(MPI)p 275 640 V 18 w(IRSENDC)17 b(\(handle,)i(start,) e(len,)g(t)o(yp)q(e,)g(dest,)g(tag,)h(con)o(text\))e Fo(is)75 763 y Fi(MPI_CREATE\(buffer_handle,)k(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 819 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 876 y(MPI_IRSEND\(handle,)e(buffer_handle,)h(dest,)i(tag,)f(context\))166 999 y Fg(MPI)p 275 999 V 18 w(ISSENDC)17 b(\(handle,)i(start,)e(len,)g(t)o (yp)q(e,)g(dest,)g(tag,)h(con)o(text\))d Fo(is)75 1121 y Fi (MPI_CREATE\(buffer_handle,)20 b(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 1178 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 1234 y(MPI_ISSEND\(handle,)e(buffer_handle,)h(dest,)i(tag,)f(context\))166 1357 y Fg(MPI)p 275 1357 V 18 w(IRECV)o(C\(handle,)12 b(start,)h(len,)g(t)o (yp)q(e,)f(source,)g(tag,)h(con)o(text,)f(return)p 1670 1357 V 19 w(status)p 1825 1357 V 19 w(handle\))75 1414 y Fo(is)75 1537 y Fi(MPI_CREATE\(buffer_handle,)20 b(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 1593 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 1650 y(MPI_IRECV\(handle,)f(buffer_handle,)f(source,)i(tag,)h(context\))166 1773 y Fg(MPI)p 275 1773 V 18 w(ISRECV)o(C\(handle,)13 b(start,)f(len,)h(t)o (yp)q(e,)f(source,)g(tag,)h(con)o(text,)g(return)p 1700 1773 V 18 w(status)p 1854 1773 V 19 w(handle\))75 1829 y Fo(is)75 1952 y Fi(MPI_CREATE\(buffer_handle,)20 b(MPI_BUFFER,)j(MPI_EPHEMERAL\))75 2008 y(MPI_ADD\(buffer_handle,)e(start,)i(len,)g(type\))75 2065 y(MPI_ISRECV\(handle,)e(buffer_handle,)h(source,)h(tag,)h(context\))75 2207 y Fn(1.12)70 b(Prob)r(e)23 b(and)h(Cancel)75 2308 y Fo(The)14 b Fi(MPI)p 242 2308 15 2 v 16 w(PROBE)f Fo(op)q(eration)g(allo)o(ws)h(to)e(c) o(hec)o(k)i(for)f(incoming)h(messages,)f(without)g(actually)h(receiving)75 2365 y(them.)20 b(The)c(user)f(can)h(then)f(decide)i(where)f(to)f(receiv)o(e) h(them,)f(based)g(on)h(the)f(information)h(returned)75 2421 y(b)o(y)f(the)f(prob)q(e)h(\(basically)l(,)h(the)f(information)g(on)f(the)h (message)f(en)o(v)o(elop)q(e\).)21 b(An)14 b(additional)i(function,)75 2478 y Fi(MPI)p 150 2478 V 17 w(GET)p 239 2478 V 17 w(LEN)d Fo(allo)o(ws)h(to)f(compute)h(the)g(amoun)o(t)f(of)h(storage)e(needed)j(to)e (receiv)o(e)i(the)f(message,)f(when)75 2534 y(this)j(length)g(is)f(not)g (readily)h(computed)g(from)e(the)i(information)f(returned)h(b)o(y)f Fi(MPI)p 1544 2534 V 17 w(PROBE)p Fo(.)166 2591 y(The)g Fi(MPI)p 334 2591 V 17 w(CANCEL)f Fo(op)q(eration)i(allo)o(ws)f(to)g(cancel)h(p)q (ending)h(comm)o(unications.)j(This)c(is)g(required)75 2647 y(for)f(clean)o(up.)22 b(P)o(osting)16 b(a)f(send)h(or)f(a)h(receiv)o(e)g (ties)g(user)g(resources)g(\(send)g(or)f(receiv)o(e)h(bu\013ers\),)f(and)h(a) 75 2704 y(cancel)g(ma)o(y)f(b)q(e)h(needed)g(to)f(free)g(these)g(resources)h (gracefully)l(.)p eop %%Page: 25 26 bop 75 -100 a Fh(1.12.)34 b(PR)o(OBE)16 b(AND)f(CANCEL)1138 b Fo(25)166 45 y Fg(MPI)p 275 45 16 2 v 18 w(PR)o(OBE\()18 b(source,)e(tag,)i(con)o(text,)f(\015ag,)h(t)o(yp)q(e,)f(return)p 1388 45 V 18 w(status\))75 190 y(IN)g(source)22 b Fo(rank)15 b(in)h(con)o(text)f(of)g(source,)g(or)f Fi(MPI)p 974 190 15 2 v 17 w(ANY)p 1063 190 V 17 w(SOURCE)g Fo(\(in)o(teger\).)75 286 y Fg(IN)j(tag)24 b Fo(user)15 b(tag)f(for)h(messages)g(receiv)o(ed)h (with)g(this)f(handle,)h(or)f Fi(MPI)p 1341 286 V 17 w(ANY)p 1430 286 V 16 w(TAG)g Fo(\(in)o(teger\).)75 382 y Fg(IN)i(con)o(text)23 b Fo(con)o(text)15 b(of)f(messages)h(receiv)o(ed)h(with)g(this)g(handle.)75 479 y Fg(OUT)i(\015ag)24 b Fo(\(logical\))75 575 y Fg(IN)17 b(t)o(yp)q(e)23 b Fo(assumed)15 b(t)o(yp)q(e)g(of)g(data)g(in)h(message)e (\(status)g(v)m(ariable\).)75 671 y Fg(OUT)k(return)p 357 671 16 2 v 19 w(handle)23 b Fo(handle)17 b(that)d(is)i(asso)q(ciated)f(with)h (return)f(status)f(ob)s(ject.)166 781 y Fi(MPI)p 241 781 15 2 v 17 w(PROBE)h Fo(returns)g Fi(flag)24 b(=)f(true)15 b Fo(if)i(a)e(there)h (is)g(a)f(message)h(that)f(can)h(b)q(e)g(receiv)o(ed)h(and)f(that)75 837 y(matc)o(hes)k(the)g(pattern)g(sp)q(eci\014ed)j(b)o(y)d(the)g(parameters) g Fi(source)p Fo(,)g Fi(tag)p Fo(,)h(and)f Fi(context)p Fo(.)35 b(It)20 b(returns)75 894 y Fi(flag)j(=)h(false)p Fo(,)15 b(otherwise.)21 b(If)15 b Fi(MPI)p 731 894 V 17 w(PROBE)g Fo(returns)g Fi(flag)24 b(=)f(true)p Fo(,)15 b(then)h(the)f(length,)h(source)g(and)75 950 y(tag)g(of)g(the)h(message)g(matc)o(hed)g(are)f(returned)h(in)h(the)f (return)g(status)f(ob)s(ject.)24 b(These)18 b(are)e(the)h(same)75 1007 y(v)m(alues)i(that)d(w)o(ould)i(ha)o(v)o(e)f(b)q(een)i(returned)f(b)o(y) f(a)h(call)g(to)f Fi(MPI)p 1176 1007 V 17 w(RECV)g Fo(or)g(to)g Fi(MPI)p 1494 1007 V 16 w(SRECV)g Fo(executed)i(at)75 1063 y(the)d(same)f(p)q(oin)o(t)h(in)h(the)f(program)e(\(with)i(a)f(ca)o(v)o(eat)g (concerning)i(length;)f(see)g(b)q(elo)o(w\).)22 b(These)16 b(v)m(alues)75 1119 y(can)j(b)q(e)h(deco)q(ded)g(from)e(the)h(return)g (status)f(ob)s(ject)g(using)i(the)f Fi(MPI)p 1302 1119 V 16 w(RETURN)p 1462 1119 V 17 w(STAT)f Fo(function.)32 b(The)75 1176 y(v)m(alue)16 b(returned)g(in)g(the)f(return)g(status)g(ob)s(ject)f(is)i (unde\014ned)h(if)f Fi(flag=false)p Fo(.)166 1233 y(The)21 b(length)g(v)m(alue)h(returned)g(b)o(y)e(the)h(return)g(status)f(ob)s(ject)g (is)h(\(correctly\))f(the)h(n)o(um)o(b)q(er)g(of)75 1289 y(elemen)o(ts)14 b(in)h(the)f(message,)f(pro)o(vided)i(that)e(all)h(elemen)o(ts)h(in)f(the)g (message)f(are)h(of)f(the)h(t)o(yp)q(e)g(sp)q(eci\014ed)75 1346 y(b)o(y)h(the)g Fi(type)g Fo(parameter;)f(otherwise)h(the)h(length)g(v)m (alue)g(returned)g(is)f(unde\014ned.)166 1403 y(A)e(subsequen)o(t)i(receiv)o (e)f(executed)g(with)g(the)g(same)f(con)o(text,)g(and)g(the)h(source)f(and)h (tag)f(returned)75 1459 y(b)o(y)f(the)g(call)i(to)d Fi(MPI)p 418 1459 V 17 w(PROBE)g Fo(will)j(receiv)o(e)f(the)g(message)e(that)h(w)o(as) f(matc)o(hed)h(b)o(y)g(the)h(prob)q(e,)g(if)f(no)g(other)75 1516 y(in)o(terv)o(ening)k(receiv)o(e)g(o)q(ccurred)f(after)f(the)h(prob)q (e.)21 b(If)15 b(the)g(receiving)h(pro)q(cess)f(is)h(m)o(ultithreaded,)f(it)g (is)75 1572 y(the)g(user)h(resp)q(onsibilit)o(y)h(to)e(ensure)h(that)e(the)h (last)h(condition)g(holds.)166 1705 y Fm(Discussion:)166 1756 y Fl(Do)g(w)o(e)g(w)o(an)o(t)g Ff(MPI)p 471 1756 14 2 v 15 w(PROBE)f Fl(to)i(return)g(mo)q(de)e(\(standard)i(or)g(secure\)?)27 b(If)16 b(y)o(es,)h(w)o(e)g(need)g(to)f(carry)h(mo)q(de)75 1806 y(information)c(with)i(messages)g(\(it's)g(deja)h(vu)f(all)f(o)o(v)o(er) i(again\).)21 b(If)15 b(not,)h(then)g(it)f(is)g(up)g(to)h(the)g(user)g(to)g (enco)q(de)75 1856 y(this)e(information)d(\(in)i(the)i(tag\))f(so)g(that)g (receiv)o(er)h(can)f(decide)h(whether)g(to)f(use)h(a)e(secure)j(or)e (standard)g(send.)166 1906 y(MPI)h(guaran)o(tees)g(that)g(successiv)o(e)i (messages)e(sen)o(t)h(from)d(a)h(source)i(to)f(a)f(destination)h(within)e (the)j(same)75 1956 y(con)o(text)21 b(are)g(receiv)o(ed)h(in)e(the)h(order)g (they)g(are)g(sen)o(t.)39 b(Th)o(us,)22 b(MPI)e(m)o(ust)g(supp)q(ort,)i (either)g(explicitly)d(or)75 2006 y(implicitly)l(,)13 b(a)i(FIF)o(O)h (structure)i(to)d(manage)f(messages)i(b)q(et)o(w)o(een)h(eac)o(h)f(pair)f(of) g(messages,)g(for)h(eac)o(h)g(con)o(text.)75 2055 y Ff(MPI)p 144 2055 V 15 w(PROBE)h Fl(returns)i(information)c(on)j(the)h(\014rst)f(matc) o(hing)e(message)i(in)f(this)h(FIF)o(O;)g(this)g(will)f(also)g(b)q(e)h(the)75 2105 y(message)i(receiv)o(ed)i(b)o(y)e(the)h(\014rst)h(subsequen)o(t)g (receiv)o(e)g(with)e(the)h(same)e(source,)k(tag)d(and)h(con)o(text)g(as)f (the)75 2155 y(message)14 b(matc)o(hed)f(b)o(y)g Ff(MPI)p 528 2155 V 16 w(PROBE)p Fl(.)166 2205 y(Message)g(passing)f(in)g(MPI)g(can)h(b)q (e)g(implem)o(en)o(ted)d(without)i(app)q(ending)g(t)o(yp)q(e)h(information)c (to)j(messages.)75 2255 y(A)h(message)h(is)f(merely)f(a)h(string)h(of)f(b)o (ytes)h(and)f(the)h(in)o(terpretation)g(of)e(these)j(b)o(ytes)f(in)o(to)f(a)g (sequence)j(of)c(t)o(yp)q(ed)75 2305 y(elemen)o(ts)18 b(is)g(done)g(using)g (the)g(information)d(in)j(the)g(bu\013er)h(descriptors)h(at)e(eac)o(h)g(end.) 31 b(The)18 b(abilit)o(y)e(to)i(use)75 2355 y(suc)o(h)g(implem)o(en)o(tation) c(strategy)j(is)g(deemed)g(to)g(b)q(e)g(an)g(imp)q(ortan)o(t)e(goal.)25 b(In)17 b(suc)o(h)h(implemen)o(tatio)o(n,)c(when)75 2405 y(a)j(message)g (arriv)o(es,)h(it)f(is)g(not)g(b)q(e)g(kno)o(wn)g(ho)o(w)g(man)o(y)e(elemen)o (ts)i(it)g(con)o(tains,)h(or)f(ev)o(en)h(ho)o(w)f(m)o(uc)o(h)f(storage)75 2455 y(is)e(needed)h(to)f(receiv)o(e)h(that)f(message)f(\(b)q(ecause)j(of)d (p)q(ossible)h(represen)o(tation)i(con)o(v)o(ersion)e(in)f(a)h(heterogeneous) 75 2504 y(en)o(vironmen)o(t\).)i(The)c(prob)q(e)f(function)g(cannot)g(use)h (a)f(bu\013er)h(descriptor;)h(this)e(defeats)h(the)g(purp)q(ose)g(of)f (probing)75 2554 y(in)k(order)h(to)g(decide)g(where)h(to)e(receiv)o(e)i(a)e (message.)23 b(Therefore,)17 b(prob)q(e)f(cannot,)g(in)f(general,)h(return)g (correct)75 2604 y(length)i(information.)29 b(Still,)18 b(it)g(is)g(often)h (the)g(case)g(that)g(prob)q(e)g(is)f(used)h(to)g(decide)g(ho)o(w)f(m)o(uc)o (h)f(storage)i(to)75 2654 y(allo)q(cate)14 b(in)g(order)h(to)f(receiv)o(e)i (a)e(message.)20 b(Enco)q(ding)14 b(suc)o(h)h(information)d(in)i(the)h (message)f(tag)g(is)h(deemed)f(to)75 2704 y(b)q(e)d(to)q(o)f(akw)o(ard,)g (and)g(it)g(is)g(deemed)g(imp)q(ortan)o(t)e(for)i Ff(MPI)p 971 2704 V 15 w(PROBE)g Fl(to)g(return)h(some)e(useful)i(size)g(information.) j(The)p eop %%Page: 26 27 bop 75 -100 a Fo(26)608 b Fh(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 45 y Fl(curren)o(t)d(de\014nition)e(of)f Ff(MPI)p 512 45 14 2 v 15 w(PROBE)g Fl(is)i(a)e(compromise)f(b)q(et)o(w)o (een)k(these)g(t)o(w)o(o)e(goals.)16 b(F)m(or)11 b(the)h(most)e(common)e (case)75 95 y(of)13 b(messages)g(where)i(all)d(en)o(tries)j(ha)o(v)o(e)e(the) h(same)f(t)o(yp)q(e)h Ff(MPI)p 1039 95 V 15 w(PROBE)f Fl(returns)i(the)f (correct)h(length)e(information;)75 145 y(the)h(more)f(esoteric)i(case)f(is)g (handled)f(b)o(y)g(the)i Ff(MPI)p 883 145 V 15 w(GET)p 964 145 V 15 w(LEN)e Fl(that)g(is)h(describ)q(ed)h(b)q(elo)o(w.)j(The)c(curren)o (t)h(solution)75 195 y(sa)o(v)o(es)h(us)f(the)h(need)g(for)f(one)g (additional)f(w)o(ord)h(p)q(er)h(message)f(that)g(w)o(ould)f(otherwise)j(b)q (e)e(needed)i(to)e(transfer)75 244 y(the)f(message)g(length)g(\(in)f(elemen)o (ts\))h(with)g(the)g(message.)166 426 y Fg(MPI)p 275 426 16 2 v 18 w(GET)p 404 426 V 20 w(LEN\()k(len,)f(return)p 813 426 V 19 w(status,)g(bu\013er)p 1134 426 V 19 w(descriptor\))166 524 y Fo(Computes)g(the)h(n)o(um)o(b)q(er)h(of)e(elemen)o(ts)i(that)e(w)o (ould)h(b)q(e)h(receiv)o(ed,)g(if)f(the)g(message)g(that)f(is)h(as-)75 580 y(so)q(ciated)e(with)h(the)f(return)f(status)g(handle)i(w)o(ould)g(b)q(e) f(receiv)o(ed)h(in)g(the)f(bu\013er)g(asso)q(ciated)g(with)g(the)75 637 y(bu\013er)f(descriptor)h(handle.)75 757 y Fg(OUT)i(len)24 b Fo(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(that)e(w)o(ould)i(b)q(e)g (receiv)o(ed)g(\(in)o(teger\))75 877 y Fg(IN)h(return)p 302 877 V 18 w(status)23 b Fo(handle)17 b(to)d(return)i(status)e(descriptor)75 998 y Fg(IN)j(bu\013er)p 290 998 V 19 w(descriptor)23 b Fo(handle)16 b(to)f(bu\013er)g(descriptor)166 1194 y Fm(Discussion:)166 1250 y Fl(In)f(order)g(to)g(supp)q(ort)g(this)g(function,)f(and)h(additional) e(\014eld)i(is)f(needed)j(to)d(in)h(the)g(return)h(status)g(ob)r(ject,)75 1300 y(e.g.,)10 b(n)o(um)o(b)q(er)f(of)g(b)o(ytes)i(in)f(the)h(incoming)c (message)j(matc)o(hed)g(b)o(y)g(prob)q(e,)h(or)f(v)n(alue)f(of)h(t)o(yp)q(e)g (parameter)g(pro)o(vided)75 1350 y(b)o(y)k(prob)q(e)g(\(unless)h(w)o(e)f(add) g(this)g(parameter)f(to)h(the)h Ff(MPI)p 1001 1350 14 2 v 15 w(GET)p 1082 1350 V 15 w(LEN)e Fl(function,)g(whic)o(h)h(is)g(to)q(o)f(ugly)g (to)h(b)q(ear\).)166 1531 y Fg(MPI)p 275 1531 16 2 v 18 w(CANCEL\()j(handle,) h(\015ag\))75 1706 y(IN)f(handle)24 b Fo(handle)16 b(to)f(comm)o(unication)h (ob)s(ject)75 1826 y Fg(OUT)i(\015ag)24 b Fo(\(logical\))166 1966 y(A)19 b(call)i(to)e Fi(MPI)p 442 1966 15 2 v 17 w(CANCEL)f Fo(cancels)j(a)e(p)q(ending)i(comm)o(unication)f(op)q(eration)g(\(send)g(or)f (receiv)o(e\).)75 2022 y(The)14 b(call)h(returns)e Fi(flag)23 b(=)h(true)13 b Fo(if)h(the)g(cancel)h(op)q(eration)f(succeeeded,)h Fi(flag)23 b(=)h(false)13 b Fo(otherwise.)75 2079 y(It)j(m)o(ust)f(b)q(e)i (the)f(case)g(that)f(either)i(the)f(cancel)h(op)q(eration)f(succeeds)h(or)e (that)g(the)h(p)q(ending)i(comm)o(u-)75 2135 y(nication)g(op)q(eration)e (completes)i(\(but)e(not)g(b)q(oth\).)24 b(If)17 b(a)f(comm)o(unication)h(op) q(eration)g(w)o(as)f(cancelled)75 2191 y(successfully)l(,)k(the)d(e\013ect)h (is)f(as)h(if)f(this)h(op)q(eration)g(w)o(as)f(nev)o(er)g(executed.)28 b(If)17 b(a)g(send)i(w)o(as)d(canceleed)75 2248 y(successfully)22 b(then)e(the)g(message)f(sen)o(t)g(will)j(not)d(b)q(e)h(receiv)o(ed,)i(the)e (receiv)o(e)g(bu\013er)g(of)f(an)o(y)h(p)q(osted)75 2304 y(receiv)o(e)g(for)f (that)g(message)g(will)i(not)e(b)q(e)h(altered,)g(and)g(an)o(y)f(suc)o(h)h (receiv)o(e)g(has)f(to)g(b)q(e)h(satis\014ed)g(b)o(y)75 2361 y(another)15 b(send;)g(if)g(a)g(receiv)o(e)h(w)o(as)e(cancelled)j (successfully)l(,)g(then)e(the)g(reciev)o(e)h(bu\013er)f(p)q(osted)g(will)h (not)75 2417 y(b)q(e)g(altered,)f(and)h(an)o(y)e(send)i(matc)o(hing)f(this)h (receiv)o(e)g(has)f(to)g(b)q(e)h(satis\014ed)f(b)o(y)h(another)e(receiv)o(e.) 166 2556 y Fm(Implemen)o(tati)o(on)e(note:)166 2613 y Fl(There)j(is)f(not)f (exp)q(ectation)i(that)f(a)g(cancel)g(op)q(eration)g(will)e(b)q(e)j(fast.)p eop %%Page: 27 28 bop 75 -100 a Fh(1.13.)34 b(CORRECTNESS)1280 b Fo(27)75 45 y Fn(1.13)70 b(Correctness)75 234 y Fm(Discussion:)17 b Fl(The)d(material)f (in)h(this)g(section)h(has)g(not)f(y)o(et)h(b)q(een)h(discussed)g(b)o(y)e (MPIF.)g(Some)g(or)g(all)f(of)h(it)g(is)75 291 y(lik)o(ely)f(to)g(mo)o(v)o(e) g(to)g(Section)i Fm(??)p Fl(.)i(It)d(is)f(incorp)q(orated)i(here)g(for)e (completeness.)75 510 y Fk(1.13.1)55 b(Order)75 601 y Fo(MPI)17 b(preserv)o(es)f(the)h(order)g(of)f(messages)g(b)q(et)o(w)o(een)h(an)o(y)g (\014xed)g(pair)g(of)g(pro)q(cesses.)24 b(In)18 b(other)e(w)o(ords,)75 657 y(if)22 b(pro)q(cess)f(A)h(executes)f(t)o(w)o(o)f(successiv)o(e)j(send)f Fi(start)e Fo(sub)q(op)q(erations,)j(pro)q(cess)f(B)f(executes)h(t)o(w)o(o)75 713 y(successiv)o(e)e(receiv)o(e)g Fi(start)e Fo(op)q(erations,)i(and)f(b)q (oth)h(receiv)o(es)f(matc)o(h)g(either)h(sends,)g(then)f(the)g(\014rst)75 770 y(receiv)o(e)f(will)h(receiv)o(e)g(the)e(message)g(sen)o(t)g(b)o(y)h(the) f(\014rst)g(send,)i(and)e(the)h(second)g(receiv)o(e)g(will)h(receiv)o(e)75 826 y(the)f(message)f(sen)o(t)h(b)o(y)g(the)g(second)g(send.)29 b(Th)o(us,)18 b(if)h(a)e(t)o(w)o(o)g(messages)g(from)g(the)h(same)g(source)g (can)75 883 y(satisfy)g(a)g(p)q(ending)i(receiv)o(e,)g(the)e(\014rst)g (message)g(sen)o(t)g(is)g(accepted;)i(if)f(a)f(message)g(can)g(satisfy)g(t)o (w)o(o)75 939 y(p)q(ending)f(receiv)o(es,)f(the)f(\014rst)g(receiv)o(e)h(p)q (osted)f(is)h(satis\014ed.)166 998 y(The)c(last)f(paragraph)g(assumes)g(that) g(the)h(send)g Fi(start)e Fo(op)q(erations)i(are)f(ordered)h(b)o(y)f(the)h (program)75 1055 y(order)20 b(at)f(pro)q(cess)h(A,)g(and)g(the)g(receiv)o(e)h Fi(start)e Fo(op)q(erations)h(are)f(ordered)i(b)o(y)e(the)h(program)f(order) 75 1111 y(at)h(pro)q(cess)h(B.)g(If)g(a)g(pro)q(cess)g(is)g(m)o(ultithreaded) h(and)f(the)g(op)q(erations)h(are)e(executed)i(b)o(y)f(distinct)75 1168 y(threads,)d(then)g(the)g(seman)o(tics)f(of)h(the)f(threaded)h(system)g (ma)o(y)f(not)g(de\014ne)i(an)e(order)h(b)q(et)o(w)o(een)g(the)75 1224 y(t)o(w)o(o)c(op)q(erations,)h(in)h(whic)o(h)g(case)f(the)g(condition)i (is)f(v)o(oid.)75 1361 y Fk(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)75 1451 y Fo(W)l(e)20 b(can)g(mo)q(del)g(the)g(execution)h(of)e (MPI)h(programs)e(as)h(an)h(in)o(teraction)g(b)q(et)o(w)o(een)g(executing)h (pro-)75 1508 y(cesses)d(that)f(execute)h(eac)o(h)g(their)g(o)o(wn)f (program,)g(and)g(the)h Fg(comm)o(unication)j(subsystem)p Fo(.)26 b(The)75 1564 y(comm)o(unication)16 b(subsystem)g(ma)o(y)f(ha)o(v)o(e)g(v)m (arious)h(constrain)o(ts)g(on)f(the)h(amoun)o(t)f(of)g(resources)h(it)g(can) 75 1621 y(use.)k(E.g.:)166 1680 y(Bounds)e(on)f(the)g(n)o(um)o(b)q(er)h(and)f (total)g(sizes)h(of)f(activ)o(e)h(comm)o(unication)g(ob)s(jects.)25 b(Suc)o(h)18 b(b)q(ound)75 1736 y(can)d(b)q(e)h(global,)g(p)q(er)f(no)q(de,)h (or)e(p)q(er)i(pair)g(of)e(comm)o(unicating)i(no)q(des.)166 1795 y(Bounds)e(on)g(the)g(n)o(um)o(b)q(er)g(and)h(total)e(sizes)i(of)e (messages)g(bu\013ered)i(in)f(the)g(system.)19 b(Suc)o(h)c(b)q(ound)75 1852 y(can,)k(again,)g(b)q(e)g(global,)g(p)q(er)g(no)q(de,)g(or)f(p)q(er)h (pair)f(of)g(comm)o(unicating)h(no)q(de.)30 b(In)19 b(addition,)h(a)e(mes-)75 1908 y(sage)f(ma)o(y)g(b)q(e)h(bu\013ered)g(at)f(the)g(sender,)i(at)d(the)i (receiv)o(er,)g(at)f(b)q(oth,)h(or)f(p)q(erhaps)h(at)f(another)g(place)75 1965 y(altogether.)166 2024 y(Th)o(us,)i(it)g(will)i(b)q(e)e(di\016cult)h(to) e(set)h(rules)g(on)g(resource)g(managemen)o(t)f(of)g(the)h(comm)o(unication) 75 2080 y(subsystem.)29 b(Ho)o(w)o(ev)o(er,)17 b(it)i(is)g(generally)g(exp)q (ected)g(that)f(implemen)o(ters)h(will)h(pro)o(vide)f(information)75 2136 y(on)c(the)g(mec)o(hanism)g(used)h(for)e(resource)h(allo)q(cation,)h (and)f(that)f(query)i(and)f(set)f(functions)i(will)h(allo)o(w)75 2193 y(to)e(query)g(and)g(p)q(ossibly)i(con)o(trol)e(the)g(amoun)o(t)g(of)f (a)o(v)m(ailable)j(resources.)166 2252 y(W)l(e)g(pro)o(vide)h(in)g(this)f (section)h(a)f(set)g(of)g(minimal)h(requiremen)o(ts)g(on)f(the)h(comm)o (unication)f(sub-)75 2308 y(system.)26 b(Programs)15 b(that)i(execute)h(on)f (an)o(y)g(subsystem)g(that)f(ful\014ls)j(these)f(minimal)g(requiremen)o(ts)75 2365 y(are)12 b Fg(safe)h Fo(and)g(will)h(p)q(ort)f(to)f(an)o(y)g(MPI)h (implemen)o(tation.)21 b Fg(Unsafe)13 b Fo(programs)e(ma)o(y)h(execute)i(on)e (some)75 2421 y(MPI)k(implemen)o(tations,)h(dep)q(ending)h(on)e(the)g(amoun)o (t)f(of)g(a)o(v)m(ailable)j(resources)e(and)g(the)g(implemen-)75 2478 y(tation)k(used)g(for)g(the)g(MPI)g(comm)o(unication)h(subsystem.)34 b(Finally)22 b Fg(erroneous)d Fo(programs)g(nev)o(er)75 2534 y(execute)e(correctly)l(.)22 b(\(While)17 b(it)f(is)g(desirable)i(to)d (detect)h(erroneous)g(programs,)e(it)i(is)h(not)e(p)q(ossible)j(to)75 2591 y(do)c(so)h(at)e(compile)j(time,)f(and)g(often)f(prohibitiv)o(e)i(to)e (do)h(so)f(a)g(run)h(time.)20 b(Th)o(us,)14 b(the)h(do)q(cumen)o(t)g(do)q(es) 75 2647 y(not)f(sp)q(ecify)h(a)f(b)q(eha)o(vior)h(for)f(erroneous)g (programs,)f(although)h(the)g(desired)i(b)q(eha)o(vior)f(is)f(to)g(return)g (a)75 2704 y(useful)i(error)f(message.\))p eop %%Page: 28 29 bop 75 -100 a Fo(28)613 b Fh(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)131 45 y Fo(1.)22 b(If)15 b(a)g(pro)q(cess)g (executes)h(an)f Fi(INIT)f Fo(op)q(eration,)h(then)g(the)h(op)q(eration)f(ev) o(en)o(tually)h(succeeds,)g(or)e(a)189 102 y Fj(r)n(esour)n(c)n(e)f(exc)n (eption)f Fo(o)q(ccurs.)19 b(The)13 b(standard)f(do)q(es)g(not)h(sp)q(ecify)g (when)g(a)f(resource)h(exception)g(is)189 158 y(allo)o(w)o(ed)h(to)g(o)q (ccur.)20 b(It)14 b(is)h(exp)q(ected)h(that)d(an)h(op)q(erational)h (de\014nition)h(will)g(b)q(e)f(made)g(a)o(v)m(ailable,)189 214 y(in)20 b(the)g(form)e(of)h(test)g(programs)g(that)f(ha)o(v)o(e)h(to)g (execute)h(with)g(no)g(resource)f(exceptions.)34 b(It)189 271 y(is)21 b(highly)i(desirable)f(to)f(ha)o(v)o(e)f(generous)h(b)q(ounds)h(on)f (the)g(n)o(um)o(b)q(er)h(of)e(concurren)o(tly)i(activ)o(e)189 327 y(comm)o(unication)17 b(ob)s(jects)f(eac)o(h)g(pro)q(cess)h(ma)o(y)f(ha)o (v)o(e,)g(so)g(that,)g(in)i(practice,)f Fi(INIT)f Fo(op)q(erations)189 384 y(will)g(alw)o(a)o(ys)f(b)q(e)h(guaran)o(teed)f(to)f(succeed.)131 480 y(2.)22 b(Eac)o(h)16 b(pro)q(cess)g(can)h(initiate)h(a)e(comm)o (unication)h(op)q(eration)g(for)e(eac)o(h)i(activ)o(e)f(comm)o(unication)189 536 y(ob)s(ject.)j(I.e.)h(correct)15 b Fi(START)f Fo(op)q(erations)i(alw)o(a) o(ys)e(succeed)i(\(ev)o(en)o(tually\).)131 632 y(3.)22 b(A)e(send)h(op)q (eration)f(is)h Fg(enabled)h Fo(if)e(the)h(sending)g(pro)q(cess)g(has)f (issued)i(a)e Fi(COMPLETE)f Fo(op)q(era-)189 689 y(tion)f(and)g(the)f (receiving)j(pro)q(cess)e(has)g(issued)g(a)g Fi(START)f Fo(op)q(eration)h (for)f(a)h(matc)o(hing)f(receiv)o(e.)189 745 y(Symmetrically)l(,)k(a)d (receiv)o(e)i(op)q(eration)f(is)g Fg(enabled)g Fo(if)h(the)e(receiving)j(pro) q(cess)e(has)f(issued)i(a)189 802 y Fi(COMPLETE)10 b Fo(op)q(eration)i(and)f (the)h(sending)g(pro)q(cess)g(has)f(issued)i(a)e Fi(START)g Fo(op)q(eration)g(for)g(a)g(matc)o(h-)189 858 y(ing)19 b(send.)31 b(An)19 b(enabled)h(op)q(eration)e(ma)o(y)g(b)q(ecome)i Fg(disabled)f Fo(either)h(b)q(ecause)f(it)g(completes)189 914 y(successfully)e(or,)e(in)h (the)f(case)h(of)f(a)g(receiv)o(e,)h(b)q(ecause)g(the)g(matc)o(hing)f (message)g(is)h(successfully)189 971 y(receiv)o(ed)g(b)o(y)f(another)g (receiv)o(e)g(op)q(eration)h(or,)e(in)i(the)f(case)g(of)g(a)f(send,)i(b)q (ecause)g(the)f(matc)o(hing)189 1027 y(receiv)o(e)h(successfully)h(receiv)o (es)f(another)f(message.)189 1104 y Fg(An)24 b(enabled)j(op)q(eration)f (either)g(completes)f(successfully)h(or)e(b)q(ecomes)i(p)q(erma-)189 1160 y(nen)o(tly)17 b(disabled.)189 1236 y Fo(I.e.,)d(an)h(enabled)i(op)q (eration)e(either)h(ev)o(en)o(tually)g(succeeds,)g(or)e(b)q(ecomes)i (disabled)h(\(progress\);)189 1293 y(and)i(an)g(op)q(eration)g(cannot)g(b)q (e)h(enabled)h(in\014nitely)g(man)o(y)e(times)g(without)g(ev)o(er)g (succeeding)189 1349 y(\(fairness\).)131 1445 y(4.)j(A)15 b Fi(FREE)f Fo(op)q(eration)i(alw)o(a)o(ys)e(succeeds)j(\(ev)o(en)o(tually\).) 166 1554 y(The)i(four)f(conditions)i(guaran)o(tee)d(progress)h(in)i(the)e (comm)o(unication)i(subsystem.)30 b(The)18 b(third)75 1610 y(condition)e(guaran)o(tee)f(\(w)o(eak\))f(fairness)h(among)g(comp)q(eting)h (comm)o(unication)g(requests.)166 1667 y(Examples)g(\(in)o(v)o(olving)g(t)o (w)o(o)e(pro)q(cessors)h(with)g(ranks)g(0)g(and)g(1\))166 1724 y(The)g(follo)o(wing)h(program)e(is)i(safe,)f(and)g(should)h(alw)o(a)o(ys)f (succeed.)75 1877 y Fi(CALL)23 b(MPI_RANK\(rank,)f(context\))75 1934 y(IF)i(\(rank.EQ.0\))e(THEN)147 1990 y(CALL)h(MPI_SENDB\(sendbuf,)e (len,)j(1,)f(tag,)h(context\))147 2047 y(CALL)f(MPI_RECVB\(recvbuf,)e(len,)j (1,)f(tag,)h(context\))75 2103 y(ELSE)95 b(!)24 b(rank.EQ.1)147 2160 y(CALL)f(MPI_RECVB\(recvbuf,)e(len,)j(0,)f(tag,)h(context\))147 2216 y(CALL)f(MPI_SENDB\(sendbuf,)e(len,)j(0,)f(tag,)h(context\))75 2272 y(END)f(IF)166 2368 y Fo(The)15 b(follo)o(wing)h(program)e(is)i (erroneous,)f(and)g(should)h(alw)o(a)o(ys)f(fail.)75 2534 y Fi(CALL)23 b(MPI_RANK\(rank,)f(context\))75 2591 y(IF)i(\(rank.EQ.0\))e(THEN) 147 2647 y(CALL)h(MPI_RECVB\(recvbuf,)e(len,)j(1,)f(tag,)h(context\))147 2704 y(CALL)f(MPI_SENDB\(sendbuf,)e(len,)j(1,)f(tag,)h(context\))p eop %%Page: 29 30 bop 75 -100 a Fh(1.14.)34 b(MISSING)1432 b Fo(29)75 45 y Fi(ELSE)95 b(!)24 b(rank.EQ.1)147 102 y(CALL)f(MPI_RECVB\(recvbuf,)e(len,)j(0,)f(tag,)h (context\))147 158 y(CALL)f(MPI_SENDB\(sendbuf,)e(len,)j(0,)f(tag,)h (context\))75 214 y(END)f(IF)166 321 y Fo(The)15 b(receiv)o(e)g(op)q(eration) g(of)f(the)h(\014rst)f(pro)q(cess)h(m)o(ust)f(complete)i(b)q(efore)f(its)f (send,)h(and)g(can)g(com-)75 377 y(plete)j(only)g(if)g(the)f(matc)o(hing)g (send)h(of)f(the)g(second)h(pro)q(cessor)f(is)h(executed;)g(the)g(receiv)o(e) g(op)q(eration)75 434 y(of)c(the)h(second)g(pro)q(cess)g(m)o(ust)f(complete)h (b)q(efore)g(its)f(send)i(and)e(can)h(complete)g(only)g(if)g(the)g(matc)o (hing)75 490 y(send)h(of)f(the)g(\014rst)g(pro)q(cess)g(is)h(executed.)21 b(This)15 b(program)f(will)j(deadlo)q(c)o(k.)166 547 y(The)10 b(follo)o(wing)h(program)f(is)g(unsafe,)h(and)g(ma)o(y)e(succeed)j(or)e (fail,)h(dep)q(ending)i(on)d(implemen)o(tation.)75 709 y Fi(CALL)23 b(MPI_RANK\(rank,)f(context\))75 766 y(IF)i(\(rank.EQ.0\))e(THEN)147 822 y(CALL)h(MPI_SENDB\(sendbuf,)e(len,)j(1,)f(tag,)h(context\))147 879 y(CALL)f(MPI_RECVB\(recvbuf,)e(len,)j(1,)f(tag,)h(context\))75 935 y(ELSE)95 b(!)24 b(rank.EQ.1)147 992 y(CALL)f(MPI_SENDB\(sendbuf,)e(len,) j(0,)f(tag,)h(context\))147 1048 y(CALL)f(MPI_RECVB\(recvbuf,)e(len,)j(0,)f (tag,)h(context\))75 1104 y(END)f(IF)166 1267 y Fo(The)11 b(message)g(sen)o (t)g(b)o(y)g(eac)o(h)g(pro)q(cess)g(has)g(to)f(b)q(e)i(copied)g(out)f(b)q (efore)g(the)g(send)h(op)q(eration)f(returns)75 1324 y(and)k(the)g(receiv)o (e)h(op)q(eration)f(starts.)k(F)l(or)14 b(the)h(program)f(to)g(complete,)h (it)h(is)f(necessary)g(that)g(at)f(least)75 1380 y(one)i(of)g(the)h(t)o(w)o (o)e(messages)g(sen)o(t)h(is)h(bu\013ered)g(out)f(of)f(either)i(pro)q (cesses')g(address)f(space.)23 b(Th)o(us,)17 b(this)75 1437 y(program)c(can)i(succeed)g(only)g(if)g(the)f(comm)o(unication)h(system)f (has)g(su\016cien)o(t)h(bu\013er)g(space)f(to)g(bu\013er)75 1493 y Fi(len)h Fo(w)o(ords)f(of)h(data.)166 1549 y(If)g(additional)h (requiremen)o(ts)f(will)i(b)q(ecome)e(part)f(of)h(the)g(standard)f(\(e.g.,)f (b)q(ounds)j(on)f(the)f(mini-)75 1606 y(mal)i(n)o(um)o(b)q(er)f(of)g (concurren)o(tly)h(activ)o(e)f(handles)i(that)e(need)h(b)q(e)g(supp)q(orted,) f(then)h(further)f(programs)75 1662 y(b)q(ecome)h(safe.)75 1806 y Fn(1.14)70 b(Missing)143 1907 y Fe(\017)23 b Fo(send-receiv)o(e)143 2001 y Fe(\017)g Fo(Send)16 b(and)f(receiv)o(e)h(from)f(same)g(bu\013er)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 14 18:27:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23295; Wed, 14 Apr 93 18:27:46 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00782; Wed, 14 Apr 93 18:26:45 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 14 Apr 1993 18:26:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00774; Wed, 14 Apr 93 18:26:43 -0400 Message-Id: <9304142226.AA00774@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1773; Wed, 14 Apr 93 18:26:41 EDT Date: Wed, 14 Apr 93 18:19:16 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: new draft of point to point section Reply-To: SNIR@watson.ibm.com Just went out. Comments, corrections are welcome. I lost some of my notes from the last meeting, so that I may have omitted some changes that were agreed there. Please draw my attention to such omissions, if any. Hopefully, by next meeting, you will have this draft with full C/F syntactic sugar, courtesy of Berryman and Geist. A few points to notice: 1. There are two new object creation functions (for comm objects and buffer descriptor objects), rather than the generic function in the previous draft. 2. Block/contiguous messages are typed. 3. We have a secure send/receive now (I am not happy about the later). 4. We have probe and cancel. 5. We need to think carefully about object freeing (the decision last meeting was to have "lazy" object freeing, but this is a problem with objects that can have multiple references, like buffer descriptors). 6. I clarified the issues of type/representation matching and conversion, and added some pointers for a future discussion on interlanguage communication. From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 14 18:35:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23379; Wed, 14 Apr 93 18:35:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01272; Wed, 14 Apr 93 18:34:34 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 14 Apr 1993 18:34:33 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01264; Wed, 14 Apr 93 18:34:31 -0400 Received: from Electryon.Think.COM by mail.think.com; Wed, 14 Apr 93 18:34:29 -0400 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA08324; Wed, 14 Apr 93 18:34:29 EDT Date: Wed, 14 Apr 93 18:34:29 EDT Message-Id: <9304142234.AA08324@electryon.think.com> To: gropp@mcs.anl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: <9304061606.AA05217@godzilla.mcs.anl.gov> "gropp@mcs.anl.gov" Subject: ready receive From: William Gropp Date: Tue, 6 Apr 93 11:06:55 CDT ... I'll start. I do not anticipate that ready receive will be of benefit to the CRAY T3D (MPP system). Why? I've heard arguments that other systems operate in what I'll call receiver-probably-ready which may (or may not) reduce the need for receiver-ready, but I have not heard anyone offer EVIDENCE that receiver-ready will not help. The only evidence that we have is that the one vendor that has implemented it does see a significant performance improvement. Because of the strong assertion that receiver-ready makes about the communication, there is a some reason to believe that any system could take advantage of the additional semantic information, though it need not do so. Does anyone care to provide some evidence that ready-receive can not provide additional performance on other systems? Within the confines of the higher levels of the CMMD message passing system (ie *not* the active message layer), a ready-receive, used correctly, could improve the performance for sending small messages. It would not appreciably effect the performance sending suitably large messages. The important factor for us is that a misused ready-receive would create unrecoverable havoc within the messaging protocol. Should this beast be accepted by the forum, we will implement it as an ordinary blocking send. moose From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 15 07:01:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01077; Thu, 15 Apr 93 07:01:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20512; Thu, 15 Apr 93 07:00:21 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 15 Apr 1993 07:00:19 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20479; Thu, 15 Apr 93 07:00:11 -0400 Date: Thu, 15 Apr 93 11:41:58 BST Message-Id: <7794.9304151041@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: new draft of point to point section To: SNIR@watson.ibm.com, mpi-pt2pt@cs.utk.edu In-Reply-To: Marc Snir's message of Wed, 14 Apr 93 18:19:16 EDT Reply-To: lyndon@epcc.ed.ac.uk Dear mpi-pt2pt colleagues First thanks to Marc for getting the new draft out so soon. I had a quick read just now, and wanted to make a small number of comments/suggestions/questions. STANDARD and SECURE modes ------------------------- 1) We were using the term REGULAR, but now see STANDARD. Why? It seems like a bad change, because it can be read to imply that the other modes are somehow non-standard. 2) The definitions seem to be incomplete or inaccurate. Sould they not say something more like: READY The send may start only if a matching receive has started. The send may complete when the user buffer is available for reuse. REGULAR The send may start whether or not a matching receive has started. The send may complete when the user buffer is available for reuse. SECURE The send may start whether or not a matching receive has started. The send may complete when the user buffer is available for reuse and a matching receive has started. SECURE Receive -------------- Marc includes some discussion of communication protocols. I promised to think more on this and return some better discussion. We (myself and the implementors of our system) talked long on the subject of mixing SECURE and REGULAR communication modes over the last couple of days, and optimisations of the modes. I will write up this discusssion and send it out if there is seen to be enough interest. The conclusions were: a) The optimisation for SECURE communication mode which I was thinking about when I asked for SECURE receive is also an optimisation for REGULAR communication mode (whether the optimisations which I was thinking of appear is a quality of implementation issue); b) There is no difficulty mixing SECURE and REGULAR communication modes, and little to be gained from the effort of not mixing them; c) The SECURE Receive procedure is redundant! I suggest that the SECURE receive is deleted. EPHEMERAL --------- What does inclusion of EPHEMERAL communication handles really buy? They can be used in non-blocking communications to avoid having to do a FREE, which might mean one less transition from user space to system space. This is an optimisation for non-blocking operations. Is this all? If so, is it worth having them for that purpose? WAIT_ANY -------- There was some discussion at the meeting about fairness and WAIT_ANY. Marc reflects this with a discussion on Page 21 of the draft. We (at EPCC) would prefer that WAIT_ANY were fair, or at least that it can be used fairly. There seems to be some concern about implementing WAIT_ANY to be fair. The rotating priority mechanism which Jim suggested is sufficient to allow WAIT_ANY to be used fairly, and was common practice for occam programmers of transputers. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Apr 19 09:47:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01593; Mon, 19 Apr 93 09:47:32 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09328; Mon, 19 Apr 93 09:45:54 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 19 Apr 1993 09:45:53 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09302; Mon, 19 Apr 93 09:45:46 -0400 Date: Mon, 19 Apr 93 14:45:01 BST Message-Id: <3994.9304191345@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: operation modes To: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear mpi-pt2pt and mpi-collcomm colleagues, I'm sending this to both subcommittees. There is a section for pt2pt and a section for collcomm, however these sections deal with an subject which probably should be consistent across both subcommittees, hence I send to both. o----------o pt2pt ----- Here is (yet another) suggestion which if adopted would help to reduce the multiplicity of send calls. In particular the multiplicity derived from the three extant communication modes REGULAR (STANDARD?), READY and SECURE (SYNCHRONOUS?). Observe that the send call in the case of each mode has the same syntax class, unlike the multiplicity derived from data buffer nature. The suggestion is to have one send procedure which accepts a MODE argument describing the communication mode, i.e. is one of: REGULAR (STANDARD?); READY; SECURE (SYNCHRONOUS?). This lets the MPI user make either local code decisions about which mode is appropriate, by using the above names, or global code decisions by use of #define in C and use of PARAMETER in Fortran (for example). I also suggest that we say SYNCHRONOUS rather than SECURE, so as not to give the impression that REGULAR (rather than STANDARD), is always not secure, since it may be secure some of the time. I propose to the pt2pt subcommittee the suggestions made here. o----------o collcomm -------- There is a class of collcomm procedures which we see may or may not barrier synchronise the calling group. The suggestion at the last meeting was that users have to write code which allows such procedures to barrier whereas they may not. The suggestion here is that those procedures which are not implicitly barrier synchronising accept a MODE argument which determines whether they certainly barrier synchronise, or whether they may or may not barrier synchronize depending on the implementation. This mode argument is one of: REGULAR; SYNCHRONOUS. Obviously I suggest that SYNCHRONOUS is the mode which forces barrier synchronisation of the group. This is consistent with the pt2pt suggestion above, except that READY is not a collcomm mode, and again lets the MPI user make either local code decisions about which mode is appropriate, by using the above names, or global code decisions by use of #define in C and use of PARAMETER in Fortran (for example). I propose to the collcomm subcommittee the suggestions made here. o----------o Comments, questions, (flames :-) please. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 20 14:09:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04494; Tue, 20 Apr 93 14:09:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22314; Tue, 20 Apr 93 14:08:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 20 Apr 1993 14:08:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22052; Tue, 20 Apr 93 14:06:10 -0400 Date: Tue, 20 Apr 93 19:06:06 BST Message-Id: <5045.9304201806@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: proposal -- context and tag limits To: rj_littlefield@pnlg.pnl.gov, mpi-context@cs.utk.edu In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Fri, 9 Apr 93 19:43:26 PDT Reply-To: lyndon@epcc.ed.ac.uk Cc: d39135@carbon.pnl.gov, gropp@mcs.anl.gov, mpi-collcomm@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Rik writes: > ============ PROPOSAL TO ENVIRONMENT COMMITTEE ============== Yes, I support the spirit and detail of the proposal. > Everyone I know would MUCH prefer suboptimal performance > over HAVING to rewrite applications to conform to varying and > inconsistent hard limits. Yes, this claim is true of everyone I know except for one very small community of academic scientists who will write their relatively simple programs from scratch for every machine on which they will do major scientific production runs. I know a whole lot more academics and commercials who just will not write programs from scratch in this way. > Yes, I recall the many arguments against mandating specific > limits. But, I claim that those arguments are misdirected. Indeed I believe that your claim is valid. > I believe that we will not meet our goal of portability > if we do not specify usable limits on tag and context values. I have the same belief. I also believe that if we fail on portability then we fail period. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 21 15:28:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07484; Wed, 21 Apr 93 15:28:49 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15627; Wed, 21 Apr 93 15:27:44 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 21 Apr 1993 15:27:42 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15586; Wed, 21 Apr 93 15:27:40 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16125; Wed, 21 Apr 1993 15:27:38 -0400 Date: Wed, 21 Apr 1993 15:27:38 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304211927.AA16125@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Precise meaning of MPI_COMMIT Suppose I have an array of particles: struct { int type; double position[3]; double velocity[3]; } particle[1000]; and I want to take the positions of all the particles of type=1 and type=2 and send them somewhere else. In addition I want to also send the number of particles of each type that I'm sending. And I only want to send one message. Can MPI do this? Here's one approach that I think doesn't work according to the current version of MPI. int count1=0; int count2=0; mpi_create_buffer (handle, MPI_PERSISTENT); mpi_add_contiguous(handle, count1, 1, MPI_INT); mpi_add_contiguous(handle, count2, 1, MPI_INT); for (i=0;i<1000;i++) { if ( particle[i].itype == 1 ) { count1++; mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); } else if ( particle[i].itype == 2 ) { count2++; mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); } } mpi_commit_buffer (handle); This doesn't work because MPI only guarantees that the message associated with the handle is no longer volatile once mpi_commit_buffer returns. So the above example will fail if an implementation actually copies count1 and count2 to the message buffer when mpi_add_contiguous is called. The example will work in an implementation that copies the data into the message buffer only when mpi_commit_buffer is called. You might ask why not put mpi_add_contiguous(handle, count1, 1, MPI_INT); mpi_add_contiguous(handle, count2, 1, MPI_INT); immediately before the call to mpi_commit_buffer. Because in this case its impossible to construct the buffer for the receiving process unless it already knows what count1 and count2 are. So my suggestion is that maybe we should stipulate that no data gets moved around until the call to mpi_commit_buffer. BTW this is not a contrived example. You need to do something similar in molecular dynamics simulations in which you cells containing differing numbers of particles between processors. Regards, David -------------------------------------------------------------------------- | David W. Walker | Office : (615) 574-7401 | | Oak Ridge National Laboratory | Fax : (615) 574-0680 | | Building 6012/MS-6367 | Messages : (615) 574-1936 | | P. O. Box 2008 | Email : walker@msr.epm.ornl.gov | | Oak Ridge, TN 37831-6367 | | -------------------------------------------------------------------------- From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 21 15:55:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07872; Wed, 21 Apr 93 15:55:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17538; Wed, 21 Apr 93 15:54:03 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 21 Apr 1993 15:54:02 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17530; Wed, 21 Apr 93 15:54:00 -0400 Received: from Electryon.Think.COM by mail.think.com; Wed, 21 Apr 93 15:53:57 -0400 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA23825; Wed, 21 Apr 93 15:53:56 EDT Date: Wed, 21 Apr 93 15:53:56 EDT Message-Id: <9304211953.AA23825@electryon.think.com> To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: <9304211927.AA16125@rios2.epm.ornl.gov> "walker@rios2.epm.ornl.gov" Subject: Precise meaning of MPI_COMMIT Date: Wed, 21 Apr 1993 15:27:38 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Suppose I have an array of particles: struct { int type; double position[3]; double velocity[3]; } particle[1000]; and I want to take the positions of all the particles of type=1 and type=2 and send them somewhere else. In addition I want to also send the number of particles of each type that I'm sending. And I only want to send one message. Can MPI do this? Here's one approach that I think doesn't work according to the current version of MPI. int count1=0; int count2=0; mpi_create_buffer (handle, MPI_PERSISTENT); mpi_add_contiguous(handle, count1, 1, MPI_INT); mpi_add_contiguous(handle, count2, 1, MPI_INT); for (i=0;i<1000;i++) { if ( particle[i].itype == 1 ) { count1++; mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); } else if ( particle[i].itype == 2 ) { count2++; mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); } } mpi_commit_buffer (handle); This doesn't work because MPI only guarantees that the message associated with the handle is no longer volatile once mpi_commit_buffer returns. So the above example will fail if an implementation actually copies count1 and count2 to the message buffer when mpi_add_contiguous is called. The example will work in an implementation that copies the data into the message buffer only when mpi_commit_buffer is called. You might ask why not put mpi_add_contiguous(handle, count1, 1, MPI_INT); mpi_add_contiguous(handle, count2, 1, MPI_INT); immediately before the call to mpi_commit_buffer. Because in this case its impossible to construct the buffer for the receiving process unless it already knows what count1 and count2 are. So my suggestion is that maybe we should stipulate that no data gets moved around until the call to mpi_commit_buffer. BTW this is not a contrived example. You need to do something similar in molecular dynamics simulations in which you cells containing differing numbers of particles between processors. I'm a bit perturbed by this. Two issues: 1. I agree with your suggestion. I don't beleive we shoudl be in the business of producing a specification that leaves holes so large that the hapless user need worry about issues like this. 2. I admit to being puzzled about this usage. I would think that you could code your solution differently so that this contorted logic would not be required. moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 21 16:27:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08725; Wed, 21 Apr 93 16:27:41 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20069; Wed, 21 Apr 93 16:26:35 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 21 Apr 1993 16:26:34 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20060; Wed, 21 Apr 93 16:26:32 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16737; Wed, 21 Apr 1993 16:26:31 -0400 Date: Wed, 21 Apr 1993 16:26:31 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304212026.AA16737@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: array of handles In section 1.2.2 of the latest p2p draft there is a notion of array of handles. These are required in the waitall(), waitany() functions. There is also discussion about len of these arrays. My question: Is there a good reason against (or for) having the notion of a handle to an array of handles in MPI ? It has certain elegance in its generality and future extensibility but I may be missing some fundamental flaw. We have a similar concept with a group id, which can be thought of as a handle to a list of process handles. (-: Hardly thinking Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 21 18:47:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11652; Wed, 21 Apr 93 18:47:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28735; Wed, 21 Apr 93 18:45:59 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 21 Apr 1993 18:45:58 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28727; Wed, 21 Apr 93 18:45:52 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 21 Apr 93 15:27 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06228; Wed, 21 Apr 93 15:25:18 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA08186; Wed, 21 Apr 93 15:25:14 PDT Date: Wed, 21 Apr 93 15:25:14 PDT From: d39135@sodium.pnl.gov Subject: Re: Precise meaning of MPI_COMMIT To: mpi-pt2pt@cs.utk.edu, walker@rios2.epm.ornl.gov Cc: d39135@sodium.pnl.gov Message-Id: <9304212225.AA08186@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu David Walker writes: > So my suggestion is that maybe we should stipulate that no data gets moved > around until the call to mpi_commit_buffer. My understanding is that no data gets moved until the send call is made. Otherwise I see no use for persistent buffer descriptors. The document should state this explicitly, however. This solves David's problem on the send side. However, it does NOT solve the corresponding problem on the receive side of David's application. The receive buffer descriptor must be generated before the receive call, thus before any of the data is transferred to user variables, thus the descriptor cannot depend on contents of the message. (Aside.. This problem could be addressed by allowing descriptors to contain pointers to values (as opposed to containing the values themselves), or by allowing messages to be received in pieces. The latter is a more general approach, e.g. it allows a later portion of a message to be received using an expression derived from an earlier part. However, all of this is MUCH different from commonly accepted practice in message-passing systems. I for one would vote early and often against such extensions.) I'm with Moose about being puzzled by the usage of David's example. I'd guess that the code David shows is designed to be efficient by avoiding an extra copy to rearrange the data after receiving it. But that's only worthwhile if one assumes that MPI can rearrange the data faster by using the buffer descriptors (one per particle!) than his application could by using application-specific data. Maybe I'm jaded, but this seems highly optimistic to me. Certainly it would be an extraordinary Fortran runtime system that would handle READ (IUNIT) (J(I),I=1,N),(A(J(I)),I=1,N) anywhere near as fast as INTEGER J(N) REAL DATA(N), A(*) READ (IUNIT) J, DATA DO I = 1,N A(J(I)) = DATA(I) ENDDO My guess is that the application would run faster, be easier to understand, and use less total memory if it didn't ask MPI to do so much. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 21 19:18:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11892; Wed, 21 Apr 93 19:18:27 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00597; Wed, 21 Apr 93 19:17:38 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 21 Apr 1993 19:17:37 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00589; Wed, 21 Apr 93 19:17:35 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA16370; Wed, 21 Apr 93 23:17:32 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA26582; Wed, 21 Apr 93 17:16:08 MDT Date: Wed, 21 Apr 93 17:16:08 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304212316.AA26582@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Precise meaning of MPI_COMMIT Rik writes, > The receive buffer descriptor must be generated before the receive > call, thus before any of the data is transferred to user variables, > thus the descriptor cannot depend on contents of the message. I seem to remember that we actually voted against providing support in MPI for reception of a message when the receiver is not aware of message contents (data types, numbers of items, etc.). I think this was the meeting before last? Tom Henderson From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 03:48:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20011; Thu, 22 Apr 93 03:48:34 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05717; Thu, 22 Apr 93 03:47:42 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 03:47:41 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05709; Thu, 22 Apr 93 03:47:39 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA17829; Thu, 22 Apr 93 03:47:22 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA14439; Thu, 22 Apr 93 03:47:12 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Thu, 22 Apr 93 03:47:10 -0400 Message-Id: <9304220747.AA21694@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA21694; Thu, 22 Apr 93 03:47:10 -0400 To: walker@rios2.epm.ornl.gov Subject: Re: Precise meaning of MPI_COMMIT Cc: mpi-pt2pt@cs.utk.edu >Suppose I have an array of particles: > >struct { int type; > double position[3]; > double velocity[3]; >} particle[1000]; > >and I want to take the positions of all the particles of type=1 and type=2 and >send them somewhere else. In addition I want to also send the number of >particles of each type that I'm sending. And I only want to send one message. >Can MPI do this? > >Here's one approach that I think doesn't work according to the current >version of MPI. > > int count1=0; > int count2=0; > mpi_create_buffer (handle, MPI_PERSISTENT); > mpi_add_contiguous(handle, count1, 1, MPI_INT); > mpi_add_contiguous(handle, count2, 1, MPI_INT); > for (i=0;i<1000;i++) { > if ( particle[i].itype == 1 ) { > count1++; > mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); > } else if ( particle[i].itype == 2 ) { > count2++; > mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE); > } > } > mpi_commit_buffer (handle); ---- Am I interpreting this code right in assuming the syntax would read as follows (assuming the March 25 draft) or have I been "sleeping" and missed something major? int count1=0; int count2=0; mpi_create (handle, MPI_BUFFER, MPI_PERSISTENT); mpi_add_block(handle, count1, 1, MPI_INT); mpi_add_block(handle, count2, 1, MPI_INT); for (i=0;i<1000;i++) { if ( particle[i].itype == 1 ) { count1++; mpi_add_block(handle, particle[i].position, 3, MPI_DOUBLE); } else if ( particle[i].itype == 2 ) { count2++; mpi_add_block(handle, particle[i].position, 3, MPI_DOUBLE); } } start(handle); ---- >This doesn't work because MPI only guarantees that the message associated >with the handle is no longer volatile once mpi_commit_buffer returns. So >the above example will fail if an implementation actually copies count1 and >count2 to the message buffer when mpi_add_contiguous is called. The example >will work in an implementation that copies the data into the message buffer >only when mpi_commit_buffer is called. > Why are you calculating "count1" and "count2" within the mpi_add_xxx loop? It seems, to me , that it is, in general, a bad idea to be changing data as the buffers are being created. It seems to me that your program could have figured out these values before the loop. >So my suggestion is that maybe we should stipulate that no data gets moved >around until the call to mpi_commit_buffer. Sounds good to me, although I agree with "moose" and still worry about the practice. > >BTW this is not a contrived example. You need to do something similar in >molecular dynamics simulations in which you cells containing differing >numbers of particles between processors. Why do you NEED this? I am "deadly" curious since I also do particle codes ... Seems to me that the situation you are describing above is sending some particles (say ions=type1, electrons = type2) that have wandered out of their local cell(s) in a PIC simulation. Surely their numbers can be kept track of earlier? BTW, the above code segment seems to loose the individual particle's type info. unless you are assuming (reasonably, I may add) that all electrons are numbered before ions, or vice versa. Pardon me. if I am missing something -- it is 3:45am in the morning... Anne C. Elster elster@cs.cornell.edu > > >Regards, >David From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 08:33:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24669; Thu, 22 Apr 93 08:33:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25641; Thu, 22 Apr 93 08:32:22 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 08:32:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25630; Thu, 22 Apr 93 08:32:19 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17667; Thu, 22 Apr 1993 08:32:10 -0400 Date: Thu, 22 Apr 1993 08:32:10 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304221232.AA17667@rios2.epm.ornl.gov> To: d39135@sodium.pnl.gov, mpi-pt2pt@cs.utk.edu Subject: Re: Precise meaning of MPI_COMMIT Rik writes: >> David Walker writes: >> So my suggestion is that maybe we should stipulate that no data gets moved >> around until the call to mpi_commit_buffer. >My understanding is that no data gets moved until the send call is >made. Otherwise I see no use for persistent buffer descriptors. >The document should state this explicitly, however. Yes the reuse of persistent buffer descriptors is a more fundamental reason why data should not be moved before the commit. Once the description of the message (using MPI_ADD_XXXX) and the composition of the message (using MPI_COMMIT_BUFFER, or better MPI_COMPOSE) are clearly separated then it is no longer clear why you need a commit/compose routine, since the assembly/disassembly of the message would be done by the send/receive routines. So I propose that the draft makes clear the difference between describing a message and composing it, and that MPI_COMMIT_BUFFER be deleted. >This solves David's problem on the send side. >However, it does NOT solve the corresponding problem on the receive >side of David's application. >The receive buffer descriptor must be generated before the receive >call, thus before any of the data is transferred to user variables, >thus the descriptor cannot depend on contents of the message. Isn't this just the usual problem that you have when you don't know how many things you're going to receive. You just have to provide a buffer that's big enough to handle the largest number of things that you expect to receive. >(Aside.. This problem could be addressed by allowing descriptors to >contain pointers to values (as opposed to containing the values >themselves), or by allowing messages to be received in pieces. The >latter is a more general approach, e.g. it allows a later portion >of a message to be received using an expression derived from >an earlier part. However, all of this is MUCH different from >commonly accepted practice in message-passing systems. I for one >would vote early and often against such extensions.) Me too. >I'm with Moose about being puzzled by the usage of David's example. >I'd guess that the code David shows is designed to be efficient by >avoiding an extra copy to rearrange the data after receiving it. My example was trying to use buffer descriptors to send a message containing mixed data types. I want to send everything in one message in the (possibly mistaken) belief that it will be faster to send everything in one message rather than sending one message for the integer counts and another one for the particle positions. If the time to send a message is dominated by the time to handle the buffer descriptors then nobody will use them. Do you think my code would run faster if I cast the counts as doubles, pack them and the particle positions into a contiguous buffer myself and then communciate them using MPI_SENDC/MPI_RECVC (the simple version for contiguous buffer)? >But that's only worthwhile if one assumes that MPI can rearrange the >data faster by using the buffer descriptors (one per particle!) than >his application could by using application-specific data. > >Maybe I'm jaded, but this seems highly optimistic to me. >Certainly it would be an extraordinary Fortran runtime system >that would handle > > READ (IUNIT) (J(I),I=1,N),(A(J(I)),I=1,N) > >anywhere near as fast as > > INTEGER J(N) > REAL DATA(N), A(*) > READ (IUNIT) J, DATA > DO I = 1,N > A(J(I)) = DATA(I) > ENDDO >My guess is that the application would run faster, be easier to >understand, and use less total memory if it didn't ask MPI to do >so much. Well can we have some examples from real applications in which the generality and flexibility offered by MPI buffer descriptors can be used, and still have the code run reasonably efficiently. David -------------------------------------------------------------------------- | David W. Walker | Office : (615) 574-7401 | | Oak Ridge National Laboratory | Fax : (615) 574-0680 | | Building 6012/MS-6367 | Messages : (615) 574-1936 | | P. O. Box 2008 | Email : walker@msr.epm.ornl.gov | | Oak Ridge, TN 37831-6367 | | -------------------------------------------------------------------------- From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 16:32:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06621; Thu, 22 Apr 93 16:32:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29236; Thu, 22 Apr 93 16:31:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 16:31:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29228; Thu, 22 Apr 93 16:31:08 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14405; Thu, 22 Apr 1993 16:31:06 -0400 Date: Thu, 22 Apr 1993 16:31:06 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304222031.AA14405@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: MPI_QUERY What does mpi_query return for the len argument when the buffer descriptor object for the operation identified by the input handle contains a buffer component of type hvector or hindexed? David From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 17:04:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07716; Thu, 22 Apr 93 17:04:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01423; Thu, 22 Apr 93 17:03:02 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 17:03:01 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01415; Thu, 22 Apr 93 17:03:00 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA21091; Thu, 22 Apr 93 14:02:55 PDT Message-Id: <9304222102.AA21091@ocfmail.ocf.llnl.gov> Date: Thu, 22 Apr 1993 14:12:52 -0800 To: mpi-pt2pt@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: MPI_QUERY Cc: walker@rios2.epm.ornl.gov >What does mpi_query return for the len argument when the buffer descriptor >object for the operation identified by the input handle >contains a buffer component of type hvector or hindexed? > >David This should be the number of elements. Eventhough the stride or index is specified in bytes for (resp.) mpi_append_hvec() and mpi_append_hindexed(), the buffer still holds a certain number of elements. The stide/index is simply a byte-based means to indicate where in the buffer those elements are. Dan From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 19:26:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11298; Thu, 22 Apr 93 19:26:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10424; Thu, 22 Apr 93 19:24:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 19:24:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10415; Thu, 22 Apr 93 19:24:30 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA22660; Thu, 22 Apr 93 16:24:25 PDT Message-Id: <9304222324.AA22660@ocfmail.ocf.llnl.gov> Date: Thu, 22 Apr 1993 16:34:23 -0800 To: mpi-pt2pt@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: Precise meaning of MPI_COMMIT Here is another question about MPI_COMMIT. Suppose I want to build a buffer and reuse it several times, changing the data in it between uses. For example, I have a data structure similar to the one David Walker suggests : struct { int type; double position[3]; double velocity[3]; } particle; struct particle particles[1000]; Instead of sending the data based on the value of type, I simply want to send the whole array for each iteration of some algorithm. I build the buffer descriptor as follows : mpi_create_buffer (handle, MPI_PERSISTENT); mpi_append_hvec(handle, &(particles[0].type), 1000, sizeof(particles), sizeof(particle), 1, MPI_INT); mpi_append_hvec(handle, &(particles[0].position), 1000, sizeof(particles), sizeof(particle), 1, MPI_DOUBLE); mpi_append_hvec(handle, &(particles[0].velocity), 1000, sizeof(particles), sizeof(particle), 1, MPI_DOUBLE); mpi_commit_buffer (handle); After the MPI_COMMIT, I am not allowed to change the data refered to by the buffer object. Is this true forever, implying I must execute the buffer_creation code each time I want to send the buffer? Or can I change the information in the particles array after it has been sent and provide the same buffer handle in a new send operation? Similarly, after receiving information into the particles array and processing it, am I allowed to resubmit the same buffer handle to a new receive? Dan From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 20:38:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12645; Thu, 22 Apr 93 20:38:51 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14624; Thu, 22 Apr 93 20:37:35 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 20:37:34 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14616; Thu, 22 Apr 93 20:37:27 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 22 Apr 93 17:34 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA10416; Thu, 22 Apr 93 17:32:11 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA10812; Thu, 22 Apr 93 17:32:09 PDT Date: Thu, 22 Apr 93 17:32:09 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: Precise meaning of MPI_COMMIT To: d39135@sodium.pnl.gov, mpi-pt2pt@cs.utk.edu, walker@rios2.epm.ornl.gov Cc: d39135@carbon.pnl.gov Message-Id: <9304230032.AA10812@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu David Walker writes: > ... Once the > description of the message (using MPI_ADD_XXXX) and the composition of > the message (using MPI_COMMIT_BUFFER, or better MPI_COMPOSE) are > clearly separated then it is no longer clear why you need a > commit/compose routine, since the assembly/disassembly of the message > would be done by the send/receive routines. > > So I propose that the draft makes clear the difference between > describing a message and composing it, and that MPI_COMMIT_BUFFER be > deleted. Commit is included to permit optimization. In the absence of an explicit commit, the buffer descriptor must always be maintained in some format that allows further MPI_ADD_XXXX calls to be made. With an explicit commit, MPI is permitted to translate the buffer descriptor to a format that can be used quickly by send/receive, but may not be modifiable. I am not aware of current message passing systems where this is done, but there is enough precedent in other areas to convince me that commit is a good thing to include. I oppose deleting the commit. ------ David continues: > > (Rik wrote..) > >This solves David's problem on the send side. > >However, it does NOT solve the corresponding problem on the receive > >side of David's application. > >The receive buffer descriptor must be generated before the receive > >call, thus before any of the data is transferred to user variables, > >thus the descriptor cannot depend on contents of the message. > > Isn't this just the usual problem that you have when you don't know > how many things you're going to receive. You just have to provide > a buffer that's big enough to handle the largest number of things that > you expect to receive. No code was shown for the receive side. I assumed that it would do a scatter similar to the gather on the send side. Perhaps that was not the case? Still reading between the lines, I now get the feeling that the receive side wants to get two integers followed by an unknown number of contiguous doubles, in which case I agree that just providing a big buffer is adequate. > My example was trying to use buffer descriptors to send a message > containing mixed data types. I want to send everything in one message > in the (possibly mistaken) belief that it will be faster to send > everything in one message rather than sending one message for the > integer counts and another one for the particle positions. If the time > to send a message is dominated by the time to handle the buffer > descriptors then nobody will use them. > > Do you think my code would run > faster if I cast the counts as doubles, pack them and the particle > positions into a contiguous buffer myself and then communciate them > using MPI_SENDC/MPI_RECVC (the simple version for contiguous buffer)? Yes. In gross generality, I would expect buffer descriptors to be fast if and only if they contain substantially fewer components than the number of data elements to be transferred. Thus it would be better to describe a couple of integers and a couple of indexed vectors (a total of 4 descriptor components) than to describe each data element separately. Is it faster to pack/unpack the data yourself? Depends. If the descriptor gets reused lots of times, then it is conceivable that say describing an indexed vector (and thus allowing send to gather the data) would be faster than gathering the data yourself and sending it as a contiguous buffer. However, if the descriptor gets used for only a single send, then for sure it's gonna be slower to generate the descriptor and make send interpret it, than to just copy the data yourself. I am not clear on whether David's application intended to reuse the descriptor. > Well can we have some examples from real applications in which the > generality and flexibility offered by MPI buffer descriptors can be > used, and still have the code run reasonably efficiently. For indexed vectors, any application with a repeating communication pattern. Extreme example, Dmitri Mavriplis's FLO72 code (NASA/ICASE), hundreds of identical iterations. Not so extreme, molecular dynamics with neighbor recalculation say every 20 time steps (and thus repeating the comm pattern 20 times for each determination). Say, DISCOVER, AMBER, CHARMM. For strided vectors, any application wanting to send an array slice containing at least a few tens of data elements. Wild guesses, of course... --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 22 23:45:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14621; Thu, 22 Apr 93 23:45:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25136; Thu, 22 Apr 93 23:44:13 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 22 Apr 1993 23:44:11 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25127; Thu, 22 Apr 93 23:44:06 -0400 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA13824 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Fri, 23 Apr 1993 04:44:02 +0100 Date: Fri, 23 Apr 1993 04:44:02 +0100 From: James Cownie Message-Id: <199304230344.AA13824@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05068; Fri, 23 Apr 93 03:39:29 GMT To: mpi-pt2pt@cs.utk.edu In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Thu, 22 Apr 93 17:32:09 PDT <9304230032.AA10812@sodium.pnl.gov> Subject: Precise meaning of MPI_COMMIT Content-Length: 1192 My understanding (for what that's worth) of the purpose of the commit is that it allows the processing of the buffer descriptor itself. It does NOT bind any of the actual data, that is always done at the point of sending or receiveing the message. I could imagine it being very useful to have the commit if one had hardware similar to an IBM channel. The commit point would be the time when you could compile the channel program, as at that point you know all of the properties of the dat accesses required to compose the message. This would bind the addresses of the data, but the actual data movement would occur when the channel program were executed (i.e. on the actual send/recv). So the commit is telling the system that it now knows all about the layout of the message. It says nothing about the data content, which one assumes will change each of the many times the committed message descriptor is used. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 23 10:07:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26650; Fri, 23 Apr 93 10:07:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06759; Fri, 23 Apr 93 10:05:50 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Apr 1993 10:05:49 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06749; Fri, 23 Apr 93 10:05:48 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14391; Fri, 23 Apr 1993 10:05:47 -0400 Date: Fri, 23 Apr 1993 10:05:47 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304231405.AA14391@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: MPI_EXCHANGE? Thanks to Jim Cownie and Rik Littlefield for explaining the reason for having mpi_commit_buffer. Was there ever a vote on whether there should be a routine mpi_exchange for exchanging data between two processes? I think it would be a useful addition to the point-to-point routines. It would look something like this: mpi_exchange ( send_bdo_handle, recv_bdo_handle, other_proc_handle, tag, context ) where "bdo"="buffer descriptor object". I think a blocking version would be sufficient, though a nonblocking version is conceivable in which the exchange is initiated and we later check if its completed, or block until completion. mpi_exchange will help users avoid writing unsafe programs. David From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 23 10:19:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27131; Fri, 23 Apr 93 10:19:09 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08284; Fri, 23 Apr 93 10:17:36 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Apr 1993 10:17:35 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08276; Fri, 23 Apr 93 10:17:34 -0400 Received: from Electryon.Think.COM by mail.think.com; Fri, 23 Apr 93 10:17:32 -0400 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA09290; Fri, 23 Apr 93 10:17:26 EDT Date: Fri, 23 Apr 93 10:17:26 EDT Message-Id: <9304231417.AA09290@electryon.think.com> To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: <9304231405.AA14391@rios2.epm.ornl.gov> "walker@rios2.epm.ornl.gov" Subject: MPI_EXCHANGE? Date: Fri, 23 Apr 1993 10:05:47 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Thanks to Jim Cownie and Rik Littlefield for explaining the reason for having mpi_commit_buffer. Was there ever a vote on whether there should be a routine mpi_exchange for exchanging data between two processes? I think it would be a useful addition to the point-to-point routines. It would look something like this: mpi_exchange ( send_bdo_handle, recv_bdo_handle, other_proc_handle, tag, context ) where "bdo"="buffer descriptor object". I think a blocking version would be sufficient, though a nonblocking version is conceivable in which the exchange is initiated and we later check if its completed, or block until completion. mpi_exchange will help users avoid writing unsafe programs. David We have been dancing around the issue of bidirectional communciation for a while. Recall that we also are flirting with the notion of a cshift function. A generic send_and_receive function provides all of this functionality. MPI_send_and_receive( dest_proc, stag, scontext, sbdo, src_prc, rtag, rcontext, rbdo ) Looking at this, I suspect that scontext and rcontext would have to be the same. moose From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 23 11:08:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28371; Fri, 23 Apr 93 11:08:37 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14143; Fri, 23 Apr 93 11:07:25 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Apr 1993 11:07:24 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14135; Fri, 23 Apr 93 11:07:22 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA19885; Fri, 23 Apr 93 15:07:18 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01641; Fri, 23 Apr 93 09:05:52 MDT Date: Fri, 23 Apr 93 09:05:52 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304231505.AA01641@macaw.fsl.noaa.gov> To: walker@rios2.epm.ornl.gov Subject: Re: MPI_EXCHANGE? Cc: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu David writes: > Was there ever a vote on whether there should be a routine mpi_exchange > for exchanging data between two processes? I think it would be a useful > addition to the point-to-point routines. It would look something like this: > > mpi_exchange ( send_bdo_handle, > recv_bdo_handle, > other_proc_handle, > tag, > context ) > > where "bdo"="buffer descriptor object". I think a blocking version would be > sufficient, though a nonblocking version is conceivable in which the exchange > is initiated and we later check if its completed, or block until completion. > > mpi_exchange will help users avoid writing unsafe programs. > > David > I also like "exchange()". I buy the argument that exchange() helps users avoid writing unsafe programs. Whenever I use Express, I use exchange() at every opportunity. /* Begin REHASH of old arguments */ When we last discussed this in point-to-point, we got mired in details. Can send and receive buffers be the same? Should there be a version with both "source" and "destination" processes in the parameter list to support a more general "shift" (like Express' exchange()). If we allow one or more of these features, can exchange() be "fast"? If we can't make exchange() "fast", should it be left to users to build it out of the low-level MPI point-to-point routines? /* End REHASH of old arguments */ On January 6, the point-to-point subcomittee decided to dump this problem on the collective communication subcommittee. In the current collective communication, there are a whole bunch of group-based "shift()" routines. Maybe we need to answer the following questions: Are the collective-communication "shift()" routines general enough? (Examples? Counter-examples?) We COULD do "exchange()" by making a group of two and using one of the "cshift()" routines. Is this acceptable? If we had point-to-point versions of both "exchange()" and "shift()", could we avoid "ready-receive"? (This is a really odd thought... :-) I'll stop rambling for now... Tom Henderson From owner-mpi-pt2pt@CS.UTK.EDU Sun May 9 16:23:17 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA03559; Sun, 9 May 93 16:23:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12935; Sun, 9 May 93 16:21:49 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 9 May 1993 16:21:46 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12925; Sun, 9 May 93 16:21:39 -0400 Message-Id: <9305092021.AA12925@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 4833; Sun, 09 May 93 16:22:13 EDT Date: Sun, 9 May 93 16:22:12 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-03.DVI.* %%Pages: 35 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 2 36 df34 DI E /Fb 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E7E 8E12>3 D E /Fc 1 16 df15 D E /Fd 4 111 df<127012F8A3127005057C840E>58 D<15181578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F0 A2123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC007815181D1C7C99 26>60 D<12C012F0123C120FEA03C0EA00F0133EEB0F80EB01E0EB0078141EEC0780EC01E0EC00 78A2EC01E0EC0780EC1E001478EB01E0EB0F80013EC7FC13F0EA03C0000FC8FC123C12F012C01D 1C7C9926>62 D<381E03E0383F0FF038639838EBB01CEAC3E013C0138000075B1300A3000E5BA2 ECE08015C048EBE180EB01C1ECC300EB00E74813FE001813781A157F941D>110 D E /Fe 5 107 df0 D15 D<150C153C15F0EC03C0EC0F00143C14F0EB07 C0011FC7FC1378EA01E0EA0780001EC8FC127812E01278121EEA0780EA01E0EA0078131FEB07C0 EB00F0143C140FEC03C0EC00F0153C150C1500A8B612FCA21E277C9F27>20 D<12C012F0123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC0078151C 1578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F012C0C9 FCA8B612FCA21E277C9F27>I<12C0B3B3AD02317AA40E>106 D E /Ff 26 121 df<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C 01C0A2387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA8 7E007013E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>III73 D76 D<38FC01F8EAFE03A2383B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03 F8A3151C7F9B18>I<387E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A2 13311339A31319A2131D130DA3EA7F07EAFF87EA7F03151C7F9B18>III82 D<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>84 D<38FF83FEA3381C0070 B2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA007C171C809B18>I<387F8FE0139F 138F380E0700120FEA070E138EEA039C13DCEA01F8A26C5AA2137013F07F120113DCEA039E138E EA070F7F000E13801303001E13C0387F07F038FF8FF8387F07F0151C7F9B18>88 D97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3 E0EB80F0EB00701478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C 809B18>II101 D<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C13 1EEA0E0FEB0780130314C038FFC7F8A3151C7F9B18>107 DI110 DI<38 7F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 D<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E001319 7F9818>116 D<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>120 D E /Fg 73 126 df<127012F8B212701200A5127012F8A31270051E769D1A>33 D<1338137813F8EA01E0EA03C0EA0780EA0F00121E121C123C123812781270A312F05AA87E1270 A312781238123C121C121E7EEA0780EA03C0EA01E0EA00F8137813380D2878A21A>40 D<126012F012787E7E7EEA0780EA03C0120113E0120013F01370A313781338A813781370A313F0 13E0120113C01203EA0780EA0F00121E5A5A5A12600D287CA21A>I<13E0A40040134038F0E1E0 EAF8E3387EEFC0381FFF00EA07FC6C5A487EEA1FFF387EEFC038F8E3E0EAF0E13840E040000013 00A413157D991A>I<123C127E127FA3123F1207120F120E123E12FC12F812E0080D77851A>44 D<387FFFC0B512E0A26C13C013047D901A>I<127812FCA41278060676851A>I48 D<13C012011203A21207120F127F12FD 12791201B2EA7FFFA3101E7B9D1A>IIII<383FFFC05AA20070C7FCA8EA73F8EA7FFE7F387E0F80387803 C0EA3001000013E01300A3126012F01301EB03C038700780EA7C1F383FFF00EA1FFCEA07F0131E 7D9D1A>I<137F3801FF804813C03807C1E0EA0F01121E383C00C0003813001278127012F0EAE3 F8EAEFFEB5FC38FE0F8038F803C0EAF00114E01300A31270A2EA7801003813C0EA3C03381F0F80 380FFF006C5AEA01F8131E7D9D1A>I<12E0B512F8A338E001E014C0EA0003EB0780EB0F00130E 131E5B133813781370A213F05BA212015BA312035BA7151F7E9E1A>III<123C127EA4123C1200A9123C127C127EA3123E120E12 1E121C123C12F812F012E0071C77941A>59 D<387FFFF0B512F8A26C13F0C8FCA4387FFFF0B512 F8A26C13F0150C7E941A>61 D<1338137CA2136C13EEA313C6A2EA01C7A438038380A4380701C0 A213FFA24813E0EA0E00A4481370387F01FC38FF83FE387F01FC171E7F9D1A>65 DIIII<387FFFFCB5FC7E380E001CA41400A3EB0380A3EA0FFFA3EA0E03A390C7 FCA8EA7FE012FF127F161E7F9D1A>I<3801F8E0EA03FEEA07FFEA0F0FEA1E03EA3C011238EA78 001270A200F013005AA5EB0FF8A338F000E01270130112781238EA3C03121EEA0F0FEA07FFEA03 FEEA01F8151E7E9D1A>I<38FF83FEA3381C0070AA381FFFF0A3381C0070AB38FF83FEA3171E7F 9D1A>II<387F03F838FF87FC387F03F8381C01 E0EB03C014801307EB0F00131E131C133C5B5B7FEA1DFC121F139E130E130FEA1E07001C138013 0314C0EB01E0A2EB00F01470007F13FC38FF81FE387F00FC171E7F9D1A>75 DI<007E133FB4EB7F806C 1400381D80DCA313C1A2001C139CA213E3A2EB631C1377A21336A2133E131CA21300A7007F137F 39FF80FF80397F007F00191E809D1A>I<38FE03FE12FFA2381D8070A213C0121CA213E0A21360 1370A213301338A21318131CA2130C130EA21306A213071303A238FF81F0A21380171E7F9D1A> IIIII<3807F1C0EA1FFDEA3FFFEA7C1FEA7007EAF003EAE001A390C7FC7E1278123F EA1FF8EA0FFEEA01FF38000F80EB03C0130114E01300126012E0A2EAF001EB03C038FE0780B5FC EBFE00EAE3FC131E7D9D1A>I<387FFFFEB5FCA238E0380EA400001300B3A23803FF80A3171E7F 9D1A>I<38FF83FEA3381C0070B3A2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA00 7C171E7F9D1A>I<38FF01FEA3381C0070A3001E13F0000E13E0A3380701C0A438038380A43801 C700A4EA00C613EEA3136C137CA21338171E7F9D1A>I<00FE13FEEAFF01EAFE000070131C0078 133C00381338A7137C001C137013EEA513C6A2380DC760A31383A3000F13E0A2380701C0171E7F 9D1A>I<387F87F8A3380F01C03807038013833803870013C7EA01CE13EEEA00FCA21378A3137C A213FE13EEEA01CF13C7380387801383380703C01301380E00E0A2387F01FC38FF83FE387F01FC 171E7F9D1A>I<38FF01FEA3381C00706C13E0A2380701C0A213830003138013C700011300A2EA 00EEA2137CA21338AA48B4FCA3171E7F9D1A>I<383FFFF85AA23870007014F0EB01E014C0EA00 03EB0780EB0F00130E131E5B133813785B5B1201485A5B120748C7FC001E1338121C123C5A1270 B512F8A3151E7E9D1A>II<126012F07EA2127812 7CA27EA27EA27E7FA26C7EA26C7EA212017FA26C7EA2137CA27FA2131E131FA2EB0F80A2EB07C0 A2130314E0A21301EB00C013277DA21A>II<387F FFC0B512E0A26C13C013047D7E1A>95 D97 D<12FEA3120EA6133FEBFFC0000F13E0EBE1F0EB8070EB00781438000E133C141CA5000F133C14 381478EB80F0EBC3E0EBFFC0000E138038067E00161E7F9D1A>I<3801FF80000713C04813E0EA 1F01383C00C0481300127012F05AA57E1270007813707E381F01F0380FFFE06C13C00001130014 157D941A>IIII<3801F8FC3807FFFE5A381F0F 8C381C0380003C13C0EA3801A3EA3C03001C1380EA1F0FEBFF00485AEA39F80038C7FC123C121C 381FFF8014F04813F8387C00FC0070131C00F0131E48130EA36C131E0078133C383F01F8381FFF F06C13E00001130017217F941A>I<12FEA3120EA6133FEBFF80000F13C0EBE1E013801300A212 0EAB38FFE3FE13E713E3171E7F9D1A>II<12FEA3120EA6EB0FFCEB1FFEEB0FFCEB03C0EB0780EB0F00131E 5B5B13FC120F13DE138F380E0780EB03C0A2EB01E0EB00F038FFE3FE14FF14FE181E7F9D1A> 107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E 1EA2EA1C1CAC387F1F1F39FF9F9F80397F1F1F00191580941A>IIII<387F87 F038FF9FFCEA7FBF3803FC3CEBF018EBE000A25BA25BA9EA7FFFB5FC7E16157E941A>114 D<380FFB80EA3FFF5AEAF80FEAE003A300F8C7FCEA7FC0EA3FFCEA0FFF38007F80EB07C0EA6001 12E012F0130338FC0F80B512005BEAE7F812157C941A>I<13C01201A6387FFFE0B5FCA23801C0 00AA1470A314F0EBE1E0EA00FFEB7FC0EB3F00141C7F9B1A>I<38FE0FE0A3EA0E00AC1301A2EA 0F073807FFFE7EEA01FC17157F941A>I<387F83FC38FFC7FE387F83FC380E00E0A3380701C0A3 38038380A33801C700A3EA00EEA3137CA2133817157F941A>I<387FC7F8EBCFFCEBC7F8380703 C038038380EBC700EA01EFEA00FE137C13781338137C13EE120113C738038380000713C0EA0F01 387FC7FC00FF13FE007F13FC17157F941A>120 D<387FC3FC38FFC7FE387FC3FC380E00E0A27E EB01C013811203EB838013C31201EBC700EA00E7A213E61366136E133CA31338A35BA21230EA78 E01271EA7FC06C5A001EC7FC17207F941A>I123 D<126012F0B3B31260042775A21A>I<127CB4FC7FEA07C0EA01E01200AB7F137CEB3FE0131F13 3FEB7C0013F05BAB1201EA07C0B45A90C7FC127C13277DA21A>I E /Fh 22 121 df73 D77 D<90B512E015F890380F00 3C151E011E130FA21507A249130FA3150E49131E153C153815F09038F003E090B51280ECFE0001 F0C7FC485AA4485AA4485AA4EAFFF8A220227DA121>80 D97 D<137E48B4FC3803C380EA0703EA0E07121C003CC7FC12381278A35AA45BEA 7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01 C013F9EA01FDEA078F380F0780120E121CEA3C03383807001278A3EAF00EA214101418EB1C30EA 703C137C3838FC60383FCFC0380F078016237BA219>I<13F8EA03FCEA0F0EEA1E06123C1238EA 780CEAF038EAFFF01380EAF0005AA413021306EA701C1378EA3FE0EA0F800F157A9416>I<143C 147F14CF1301EB03861480A3EB0700A5130EEBFFF0A2EB0E00A25BA55BA55BA55BA45B1201A2EA 718012F390C7FC127E123C182D82A20F>II<13F0EA0FE0A21200A2485AA4485AA448C7FC131FEB7F80 EBE1C0380F80E0A21300120E381E01C0121CA338380380A21484EB07060070130C130E1418EB06 3038E007E0386003C017237DA219>I<136013F013E0A21300A8120EEA1F801233126312C3A3EA 0700A2120EA35A13201330EA3860A213C01239EA1F80EA0E000C217CA00F>I108 D<391E07C07C393F1FE1FE3963B873879039E03E038012C3EBC03CEB8038000790387807 00EB0070A3000EEBE00EA21610ED1C18261C01C0133015381660ED18C03A3803801F80D81801EB 0F0025157C9428>I<381E0780383F1FE0EA63B8EBE070EAC3C0A21380000713E01300A3380E01 C0A214C2EB0383001C1386EB0706140CEB0318003813F0381801E018157C941B>I<137E48B4FC 3803C380380701C0120E001C13E0123CA21278A338F003C0A21480130700701300130E5B6C5AEA 1FF0EA07C013157B9419>I<3803C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801 C03CA314780003137014F014E0EBE3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F 9419>I<381E0F80383F1FC03863B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A 121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC 6C7EEA007E130FA2EA7007EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA 0380A4EA0700EAFFF8A2EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E00 0D1F7C9E10>I<000F1330381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E03 80A214841486EB070CA2130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA 31C3EA61C1EAC1C0EAC380A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8 EA01E013157C9416>I<3803C1C0380FE7E0381C7E3038303C70EB38F01260146038007000A45B A214200070133038F1C060A200E113C038E3E180387E7F00EA3C3E14157D9416>120 D E /Fi 65 123 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80FC01387ED 83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D13 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA3127812F8AD1278127C A3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416>40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A313E0 A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I<1238127C12FE12FFA2127F12 3B1203A212071206A2120C121C12181270122008117C8610>44 DI<12 38127C12FEA3127C123807077C8610>I<13181378EA01F812FFA21201B3A7387FFFE0A213207C 9F1C>49 DII<14E013011303A2130713 0F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2380007 E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018C7FC A4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8386007 E0003013C0381C1F80380FFF00EA03F815207D9F1C>II57 D<1238127C12FEA3127C12381200A81238127C12FEA3127C123807167C95 10>I63 D<1470A214F8A3497EA249 7EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F140701E07F90B5FCA23901 8001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A>65 DIIIIIIII75 DIIIII< B512FEECFFC03907F007F0EC01F86E7E157E157FA6157E5D4A5AEC07F090B512C05D9038F00FE0 6E7E6E7E6E7EA81606EC00FEEDFF0CB538803FF8ED0FF027227EA12A>82 D<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400B4FC13E06C B4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA36C131EA26C 133C12FCB413F838C7FFE00080138018227DA11F>I<007FB61280A2397E03F80F007814070070 14030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127>IIII<3A7FFFC1FFF0A23A03FC00 0C006C6C5B000014386D5B90387F8060013F5B14C190381FE380010F90C7FC14F7EB07FE6D5AA2 6D7E13008081497F14BF9038031FE0496C7E130E90380C07F8496C7E133890383001FE496C7E13 E04848EB7F8049EB3FC03AFFFC03FFFEA227227FA12A>II<003FB512E0A29038801F C0383E003F003C14800038EB7F00485B5C1301386003FC5C130700005B495A131F5C133F495A91 C7FC5B491360485A12035B000714E0485A5B001FEB01C013C0383F8003007F1307EB003FB6FCA2 1B227DA122>I97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3 127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F 0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC 0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B38 1F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2 EAFFF0A20C247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC13 3E137EEA78FCEA7FF0EA1FC0102E83A311>III<3AFF87F00FE090399FFC3FF83A1F B87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I<38FF 87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FFC038 0F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E03807FF C03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2 EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A51207EB E300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01FC1A 167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C 6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<3AFFE3FF87F8A23A1F807C00C0 D80FC0EB0180147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB9038FF 07DC6C14F8EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>I<39FFF07FC0A2390F C01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F838 0381FC48C67E000E137E39FF81FFE0A21B167F951E>I<39FFF01FE0A2390FC00600A2EBE00E00 07130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA2130CA2 5B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>I<387FFFF0A2387C07E038700FC0 EA601F00E0138038C03F005B13FEC65A1201485AEBF0301207EA0FE0EBC070EA1F80003F1360EB 00E0EA7E03B5FCA214167E9519>I E /Fj 64 123 df<90380FC3E090387FEFF09038E07C7838 01C0F8D8038013303907007000A7B61280A23907007000B0387FE3FFA21D20809F1B>11 DII<90 380F80F890387FE7FE9038E06E063901C0FC0F380380F8380700F00270C7FCA6B7FCA239070070 07B03A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A2123812301260 1240060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A2 12E0AC1260A21270A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I< 12C012607E7E121C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A2 13005A12065A121C12185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A212381230 12601240060E7C840D>44 DI<127012F8A3127005057C840D>I<1303A2 13071306A2130E130CA2131C1318A213381330A213701360A213E013C0A212011380A312031300 A25A1206A2120E120CA2121C1218A212381230A212701260A212E05AA2102D7DA117>II<130EA2131E133EA2136E13EE13CEEA018E1203130E1206 120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>52 D<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A7 6CC7FC121F7D9D17>55 D57 D<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200 AA127012F8A312781218A41230A21260A21240051D7C930D>I63 D65 DI<90381FC04090387FF0C03801F8393803C00D380780 07380F0003121E003E1301123C127C1400127812F81500A8007814C0127CA2123C003EEB018012 1E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D9F21>IIII<90380FC02090387FF8603901F81CE03803E00638078003380F0001121E14005A127C156012 7812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E38078003EA03E03901F80E6039007FFC 2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90B5FCA2EB800FAD39FFF8FFF8A2 1D1F7E9E22>II76 DI<39FF807FF813C00007EB07809038E00300A2EA06F0A21378133CA2 131EA2130FA2EB078314C31303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0801D1F 7E9E22>III<3807E080EA0FF9EA1C1FEA300FEA70 07EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A213 0112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81F812217D9F19>83 D<007FB512E0A238780F010070130000601460A200E0147000C01430A400001400B23807FFFEA2 1C1F7E9E21>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800FC006A3913819E00ED8 03C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60A390393D801EC0A390 383F000F6D5CA3010E6DC7FCA32C207F9E2F>87 D<397FF83FF8A23907C00F800003EB06003801 E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C13078080130DEB1DF0EB18F8EB 3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FFE01FFEA21F1F7F9E 22>I92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00 E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>III I<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FC A216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I< 13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C 28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E 1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20 809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0 A223147F9326>IIII<3803E1 80EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA00 03A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318 A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01 F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA2 13FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14 E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E14 7F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA 01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E 13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3 007FC7FC123C151D7F9318>II E /Fk 37 121 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA25AA412F85A A97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01E0A2 13C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F123B12 03A21206A2120E120C12181270122008107C860F>44 D<1238127C12FEA3127C12381200A61238 127C12FEA3127C123807147C930F>58 D63 D<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F 48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E9E28>65 D<903807FC0290383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406 127EA200FE1400A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383F FF80903807FC001F1F7D9E26>67 DII73 D77 DIII 82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86CB4FC14C0 6C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE01E038E7 FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0140E00C0 1406A400001400B10007B512C0A21F1E7E9D24>I88 D97 D99 DII I104 D<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217EA00E>I107 DI<3AFE0FE03F80 90393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8FFEA227147D932C >I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D>I<48B4FC000713 C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F03807FFC0000113 0017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA6143E147E147CEB80 FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I<3AFFE7FE1FE0A2 3A1F00F007006E7ED80F801306A23907C1BC0CA214BE3903E31E18A23901F60F30A215B03900FC 07E0A290387803C0A3903830018023147F9326>119 D<38FFE1FFA2380F80706C6C5A6D5A3803 E180EA01F36CB4C7FC137E133E133F497E136FEBC7C0380183E0380381F0380701F8380E00FC39 FF81FF80A219147F931C>I E /Fl 73 124 df<90380F83E090387FE7F09038F07E783801C0F8 EA0380EC7000EA0700A8B612C0A23907007000B1397FE3FF80A21D2380A21C>11 DII<903807E03F90393FF0FF809039F03BC1C03A01C01F00E039 03803E01A23A07001C00C01600A7B712E0A23907001C011500B03A7FF1FFCFFEA2272380A229> I34 D<127012F812FCA2127C120CA41218A21230A212601240060F7CA20E>39 D<1330136013C0EA0180EA03005A1206120E120C121C12181238A212301270A3126012E0AE1260 1270A312301238A21218121C120C120E120612077EEA0180EA00C0136013300C327DA413>I<12 C012607E7E7E120E120612077E1380120113C0A2120013E0A313601370AE136013E0A313C01201 A21380120313005A1206120E120C5A5A5A5A0C327DA413>I<127012F812FCA2127C120CA41218 A21230A212601240060F7C840E>44 DI<127012F8A3127005057C840E> IIIIII<130EA2131EA2133EA2 136E13EE13CE1201138EEA030E12071206120E120C1218A212301270126012E0B512F8A238000E 00A73801FFF0A215217FA018>I<00101380EA1C07381FFF005B5B13F00018C7FCA613F8EA1BFE EA1F0F381C0780EA180314C0EA000114E0A4126012F0A214C0EAC0031260148038300700EA1C1E EA0FFCEA03F013227EA018>I<137E48B4FC3803C180380701C0EA0E03121CEB018048C7FCA212 7812701320EAF1FCEAF3FEEAF60738FC038000F813C0130112F014E0A51270A3003813C0130300 181380381C0700EA0E0EEA07FCEA01F013227EA018>I<12601270387FFFE0A214C0EA600038E0 018038C00300A21306C65AA25BA25BA25BA213E0A3485AA51203A86C5A13237DA118>III<127012F8A312701200AB127012F8A3127005157C940E>I< 127012F8A312701200AB127012F8A312781218A41230A3126012E01240051F7C940E>I61 D<497E497EA3497EA3497E130CA2EB1CF8EB1878A2EB383C 1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F0001F130139 FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003F0380780 0148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C1460123E12 1E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>IIII<903807F00890383FFC189038FC0E3838 01E0033903C001F83807800048C71278121E15385AA2007C14181278A212F81500A6EC1FFF1278 007CEB0078A2123CA27EA27E6C7E6C6C13F83801F0013900FC079890383FFE08903807F8002024 7DA226>I<39FFFC3FFFA239078001E0AD90B5FCA2EB8001AF39FFFC3FFFA220227EA125>II76 DI<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7F A2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000 EAFFF0156020227EA125>II< B512F014FC3807803FEC0F801407EC03C0A215E0A515C0A2EC0780140FEC3F00EBFFFC14F00180 C7FCADEAFFFCA21B227EA121>I82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA 3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF8 0138EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200 E0141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<39FFFC0FFFA239078001F8EC00 60B3A5000314406D13C0A20001EB01803900E00300EB7007EB3C1EEB1FF8EB07E020237EA125> I<3BFFF03FFC07FEA23B0F0007C001F00203EB00E01760D807806D13C0A33B03C007F001801406 A216032701E00C781300A33A00F0183C06A3903978383E0CEC301EA2161C90393C600F18A39039 1EC007B0A3010F14E0EC8003A36D486C5AA32F237FA132>87 D<397FF807FFA23907E003F00003 EB01C000015C3800F00301F890C7FCEB7806EB7C0EEB3E0CEB1E18EB1F38EB0FB0EB07E0A21303 1301497E80EB0778EB0E7CEB0C3EEB1C1EEB181F496C7EEB700701607F496C7E0001130101807F 00031300D80FC07F3AFFE007FFC0A222227FA125>I92 D 97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C141C141EA7141C 143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II<14 E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA38001278127012F0A712701278EA3801EA 3C03381E0EF0380FFCFEEA03F017237EA21B>II< 133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700B1EA7FF8A2102380A20F>I<14F03801F1 F83807FFB8380F1F38381E0F00EA1C07003C1380A5001C1300EA1E0FEA0F1EEA1FFCEA19F00018 C7FCA2121CEA1FFF6C13C04813E0383801F038700070481338A400701370007813F0381E03C038 0FFF803801FC0015217F9518>I<120E12FEA2121E120EAAEB1F80EB7FC0380FC1E0EB80F0EB00 70120EAE38FFE7FFA218237FA21B>I<121C121E123E121E121CC7FCA8120E12FEA2121E120EAF EAFFC0A20A227FA10E>II<120E12FEA2121E120EAAEB0FFCA2EB07E0EB0380EB0700 130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA217237FA21A >I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF809039C0F303 C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038FE7FC0 38FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II<380E1F8038FE7FE038FFC1F0380F0078120E143CA2141EA714 3CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FCA8EAFFE0A2171F7F941B>I<3801F060EA 07FCEA1F06381C03E0EA3C01EA7800A25AA712781301123C1303EA1F0EEA0FFCEA03F0C7FCA8EB 0FFEA2171F7E941A>I II<1206A5120EA3121E123EEAFFF8A2EA0E00AA13 0CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E1370AC14F0 1301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A26C13C0A338 038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2391E01C07C D81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D81980141DA2 EBF00F00001400A2497EEB600620157F9423>I<387FC1FFA2380780F8000313E03801C1C01480 3800E3001377133E133C131C133E13771367EBC3803801C1C0380380E0380700F0EA0F8038FFC1 FFA2181580941A>I<38FFC3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2 EA00E6A3137CA31338A21330A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>I< 383FFFC0A2383C038038380700EA300EEA701EEA603C13385BEA00F0485A3803C0C01380EA0700 5AEA1E01001C1380EA3803EA7007B5FCA212157F9416>II E /Fm 44 122 df<903901FF81FF011F01EF13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13 F8D807F013FCA2EE07E0020190C7FCA6B712F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7F A928>11 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FE A317277BA622>III<14075C5C5C5C5CA25B5B497E13 0F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F 13F8A31D277EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC038 0FFFF8EB80FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F 007814806CEBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C015 80A215003870001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201 A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F 90393FF007FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503 A3484891C7FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F8 90393FF007F0010FB512C0010391C7FC9038003FF829297CA832>II72 DI77 DII< B612F815FF16C03A03FE003FE0ED0FF0ED07F816FC150316FEA716FC150716F8ED0FF0ED3FE090 B61280EDFE0049C8FCB0B512F8A327297DA82F>I82 D<90387F80603903FFF0E0000F13FF381F807F383F001F003E1307007E1303127C00FC1301A214 007E7E6D130013F8EBFF806C13F814FE6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F 1407A200E013031401A37E15F06C13036C14E0B413079038E01FC090B5120000E05B38C01FF01D 297CA826>I<007FB712C0A39039807FC03FD87C00140700781503A20070150100F016E0A24815 00A5C71500B3A490B612E0A32B287EA730>I<48B47E000F13F0381F81FC486C7E147FA2EC3F80 A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F 83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<90 38FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80 FE001F5B380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14 F0121F397E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038 007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3 A3EAFFFEA30F2B7DAA14>105 D107 DI<3BFFC07F800FF090 3AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA301 E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F090 38DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E0390F F803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EFFF 80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85B A3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370 A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7EA2 6C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381FFF F0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03FF8 A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3AFFFE 03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F 8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<39FFFC0FFFA33907F003C06C6C 485AEA01FC6C6C48C7FCEBFF1E6D5AEB3FF86D5A130FA2130780497E497E131EEB3C7F496C7E49 6C7ED801E07FEBC00F00036D7E3AFFF01FFF80A3211B7F9A24>120 D<3AFFFE03FF80A33A07F0 007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2 EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA 0FC021277F9A24>I E /Fn 11 118 df49 D<913A03FF800380023FEBF00749B5EAFC0F0107ECFF1F01 1F9038803FBF903A3FF80007FFD9FFE07F48497F48497F4890C8127F4848153F49151F121F4915 0F123F5B007F1607A34992C7FC12FFAB127F7FEF0780A2123F7F001F160F6D1600120F6D5D6C6C 153E6C6D5C6C6D14FC6C6D495AD93FF8495A903A1FFF801FC0010790B55A01014AC7FCD9003F13 F80203138031337BB13C>67 D80 D97 D99 D105 D<2703F007F8EB0FF000FFD93FFFEB7FFE4A 6DB5FC903CF1F03FC3E07F80903CF3C01FE7803FC0260FF780EBEF0000079026000FFEEB1FE001 FE5C495CA2495CB2B500C1B50083B5FCA440207D9F45>109 D<3903F007F800FFEB3FFF4A7F90 39F1F03FC09039F3C01FE0380FF7800007496C7E13FE5BA25BB2B500C1B51280A429207D9F2E> II<1378A513F8A41201A212031207120F381FFFFE B5FCA33807F800AF140FA7141F3803FC1EEBFE3E3801FFFC38007FF0EB1FC0182E7EAD20>116 DI E /Fo 26 122 df<1238127C12FEA212FF127F 123B1203A41206A2120CA2121812381270122008137B8611>44 D<1318133813F8120712FF12F8 1200B3AD487E387FFFF0A214287CA71E>49 D<137F3801FFC0380781F0380E00F80018137C121E 003F137EEB803EA3381F007E000E137CC7FCA25C5C495AEB07C001FFC7FCA2EB01E06D7E147C80 A280A21580123C127EB4FCA31500485B007C133E00305B001C5B380F01F06CB45AC690C7FC1929 7EA71E>51 D<137F3801FFC03807C1E0380F0070001E7F001C133C003C131C48131EA200F87FA4 1580A41278141F127C003C133F121C001E136F6C13CF3807FF8F0001130FD8001013001300A214 1EA2121E003F5BA25C1470003E5B381801C0380E0780D807FEC7FCEA01F819297EA71E>57 D69 D<02FF13100107EBE03090391FC0707090387E001C01F8EB0EF048481303485A48481301484813 00A248C812705A123E1630127E127CA200FC1500A84AB5FC127C007E90380007F01503123EA212 3F7E6C7EA26C7E6C7E6C6C13076C7E017E131C90391FC07870903907FFE0100100EB8000282B7D A92F>71 D76 DI83 D87 D97 D99 D<140F49B4FCA2EB001F80AC137E3801FFCF3807C0EF380F003F 001E7F487FA2127C127812F8A81278127C123C5C6C5B6C5B3907C1EF803903FF8FF83800FE0F1D 2A7EA921>I<140F3901FC3F803907FF73C0380F07E3381E03C3003EEBE180393C01E000007C7F A6003C5BEA3E03001E5B381F078001FFC7FCEA39FC0030C8FCA21238123C383FFFC06C13F86C7F 487F3838003F48130FEC0780481303A40070EB070000785B6C131E380F80F86CB45AC613801A28 7E9A1E>103 D<120FEA1F80A213C01380A2EA0F00C7FCA8EA0780127FA2120F1207B3A2EAFFF8 A20D297FA811>105 D107 DI<3A0783F801FC 3AFF8FFE07FF903A9C0F0E07803B0FB0079803C03B07E003F001E001C013E0A2018013C0B13BFF FC7FFE3FFFA2301A7F9933>I<380783F838FF8FFCEB9C1E380FB00F3907E0078013C0A21380B1 39FFFCFFFCA21E1A7F9921>I<137F3801FFC03807C1F0380F0078001E7F487FA2487FA200F814 80A800781400007C5B003C131EA26C5B6C5B3807C1F03801FFC06C6CC7FC191A7E991E>I<3807 83F838FF8FFEEBBC1F390FE00780D807C013C090388003E0140115F0A2EC00F8A8EC01F0A215E0 EBC003EC07C09038E00F809038B83F00EB8FFCEB83F00180C7FCAAEAFFFCA21D267F9921>I<38 0787C038FF9FE0EBB9F0EA0FF1EA07E1EBC0E01400A25BAF7FEAFFFEA2141A7F9917>114 D<3807F840381FFFC0EA3C07EA7003EA6001EAE000A36C1300127EEA7FF0EA3FFC6CB4FC000713 8038003FC0130738C001E013007EA36C13C0EAF80138FE078038C7FF00EA83F8131A7E9918>I< 390780078000FF13FFA2000F130F00071307AF140FA2141F0003133F9038E077C03900FFC7FCEB 7F071E1A7F9921>117 D<3AFFF1FFC3FFA23A0F803E00FCD9001C13703807801E023E1360A226 03C03F13C01467A2D801E0EB818014C3A201F013C32600F1811300A201F913E790387B00E6A201 7F13FE013E137CA3011E1378011C1338A2281A7F992B>119 D<39FFF01FF8A2390F8007C00007 1480EC03006D5A00031306A26C6C5AA2EBF01C00001318A2EB7830A36D5AA26D5AA36D5AA36DC7 FCA21306A2130E130C1230EAFC18A25B1370EA78E0EA3FC06CC8FC1D267F9920>121 D E /Fp 10 118 df67 D80 D97 D99 D105 D<2701F803F8EB03F800FFD91FFFEB1FFF913B7C0FC07C0FC0913BE007E0E007E03C 07F9C003E1C0032601FB80D9F3807FD9FF00EBF70049D901FE6D7EA2495CA3495CB3A4486C496C 497EB500F0B500F0B512F0A344267EA549>109 D<3901F807F800FFEB1FFE9138781F809138E0 0FC03A07F9C007E03801FB80EBFF00496D7EA25BA35BB3A4486C497EB500F1B512E0A32B267EA5 30>II<1318A51338A41378A213F8A2120112031207001FB5FCB6FCA2D801F8C7FCB2EC0180AA38 00FC031500137CEB7E07EB3F0EEB0FFCEB03F019367EB421>116 DI E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 392 509 a Fp(P)n(oin)n(t)29 b(to)g(P)n(oin)n(t)g(Comm)n(unication)843 656 y Fo(Marc)20 b(Snir)554 731 y(William)d(Gropp)k(and)f(Ewing)f(Lusk)813 847 y(Ma)n(y)h(9,)f(1993)164 1068 y Fn(1)83 b(P)n(oin)n(t)27 b(to)h(P)n(oin)n(t)f(Comm)n(unication)164 1192 y Fm(1.1)70 b(In)n(tro)r(duction)164 1284 y Fl(This)15 b(section)f(is)h(a)g(draft)g(of)g (the)g(curren)o(t)f(prop)q(osal)i(for)f(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o (unic)o(a-)164 1344 y(tion.)21 b(It)15 b(do)q(es)h(not)g(y)o(et)f(include)f (a)i(description)f(of)h(the)f(F)l(ortran)h(77)g(and)g(C)g(bindings.)237 1404 y(I)c(ha)o(v)o(e)g(tried)f(to)i(indicate,)e(wherev)o(er)g(appropriate,)j (gaps)f(and)g(unresolv)o(ed)e(issues,)164 1465 y(using)17 b(small)d(t)o(yp)q (e.)164 1645 y Fk(Discussion:)54 b Fj(The)19 b(follo)o(wing)h(subsections)f (of)f(the)h(in)o(tro)q(duction)h(con)o(tain)f(general)g(in-)164 1705 y(formation)d(on)h(the)g(design)h(of)f(MPI)g(pro)q(cedures.)26 b(The)17 b(material)h(should)g(b)q(e)f(mo)o(v)o(ed)g(to)f(a)164 1766 y(general)g(in)o(tro)q(duction)g(for)e(the)i(en)o(tire)f(do)q(cumen)o (t.)164 2030 y Fm(1.2)70 b(Data)23 b(T)n(yp)r(es)164 2123 y Fi(1.2.1)55 b(Handle)164 2215 y Fl(MPI)19 b(pro)q(cedures)h(use)g(at)g(v)m (arious)h(places)e Fh(hand)r(les)p Fl(.)34 b(Handles)19 b(are)h(used)g(to)g (access)164 2275 y(opaque)c(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(ject)f(can)i (b)q(e)f(created,)f(up)q(dated)i(and)f(destro)o(y)o(ed)g(only)f(b)o(y)164 2336 y(b)o(y)h(calling)f(suitable)h(MPI)g(pro)q(cedures,)g(and)h(pro)o (viding)f(the)g(handle)h(as)g(parameter.)164 2396 y(Opaque)11 b(ob)s(jects)g(hide)g(from)f(the)h(user)g(the)g(in)o(ternal)f(represen)o (tation)g(used)i(for)f(v)m(arious)164 2456 y(MPI)17 b(ob)s(jects,)h(th)o(us)f (allo)o(wing)h(to)g(ha)o(v)o(e)f(similar)f(calls)h(in)g(C)h(and)h(F)l (ortran,)f(allo)o(wing)961 2599 y(1)p eop %%Page: 2 2 bop 164 307 a Fl(to)15 b(o)o(v)o(ercome)e(problems)g(with)i(the)g(t)o(yping)g (rules)f(in)h(these)f(languages,)j(and)e(allo)o(wing)164 367 y(for)j(future)f(extension)g(of)g(their)g(functionalit)o(y)l(.)23 b(Handles)17 b(are)h(of)f(t)o(yp)q(e)g Fg(void)25 b(*)17 b Fl(in)g(C)164 428 y(and)g(of)f(t)o(yp)q(e)g Fg(integer)e Fl(in)i(F)l(ortran.) 237 488 y(An)f(opaque)h(ob)s(ject)e(can)i(b)q(e)f Fg(persistent)d Fl(or)j Fg(ephemeral)p Fl(.)j(A)d(p)q(ersisten)o(t)f(ob)s(ject)164 548 y(p)q(ersists)e(un)o(til)f(destro)o(y)o(ed)f(b)o(y)h(an)i(explicit)c(op)q (eration.)21 b(An)11 b(ephemeral)e(ob)s(ject)i(is)h(go)q(o)q(d)164 608 y(for)17 b(a)g(single)g(use;)f(th)o(us)h(an)h(ephemeral)c(ob)s(ject)i (asso)q(ciated)i(with)f(a)g(comm)o(unicati)o(on)164 668 y(op)q(eration)f (disapp)q(ears)g(once)f(this)h(op)q(eration)g(is)f(completed)e(\(or)i(once)g (this)g(ob)s(ject)g(is)164 729 y(not)i(needed)e(an)o(ymore)g(for)i(the)f (completion)e(of)i(the)g(op)q(eration\).)237 789 y(An)j(opaque)g(ob)s(ject)f (is)h(created)f(b)o(y)g(a)h(call)f(to)i Fg(MPI)p 1213 789 16 2 v 17 w(CREATE)p Fl(,)c(and)j(destro)o(y)o(ed)f(b)o(y)164 849 y(a)23 b(call)f(to)i Fg(MPI)p 455 849 V 17 w(FREE)p Fl(.)d(Additional)h (MPI)g(functions)h(are)g(a)o(v)m(ailable)f(to)i(access)e(and)164 909 y(up)q(date)17 b(sp)q(eci\014c)f(opaque)g(ob)s(jects.)164 1029 y Fi(MPI)p 279 1029 17 2 v 20 w(CREA)-5 b(TE\(handle,)20 b(t)n(yp)r(e,)d(p)r(ersistence\))164 1121 y(OUT)i(handle)25 b Fl(handle)16 b(to)h(ob)s(ject)164 1219 y Fi(IN)h(t)n(yp)r(e)24 b Fl(state)d(v)m(alue)f(that)h(iden)o(ti\014es)e(the)i(t)o(yp)q(e)f(of)g(ob)s (ject)g(to)h(b)q(e)g(created)f(\(e.g.,)286 1279 y Fg(MPI)p 367 1279 16 2 v 17 w(COMMUNICATI)o(ON)o(,)j(MPI)p 847 1279 V 17 w(BUFFER,)g(MPI)p 1147 1279 V 18 w(CONTEXT)p Fl(,)13 b(etc.\).)164 1377 y Fi(IN)18 b(p)r(ersistence)24 b Fl(state)16 b(v)m(alue;)g(either)f Fg(MPI)p 1020 1377 V 17 w(PERSISTENT)e Fl(or)j Fg(MPI)p 1447 1377 V 18 w(EPHEMERAL)o Fl(.)164 1529 y Fi(MPI)p 279 1529 17 2 v 20 w(FREE\(handle\))164 1620 y(IN)i(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)237 1712 y(An)h(ob)s(ject)f(can)h(b)q(e)g(destro)o(y)o(ed)f (only)h(if)f(there)h(is)f(no)i(p)q(ending)f(op)q(eration)h(that)f(is)164 1772 y(using)h(this)g(ob)s(ject;)f(after)h(successful)f(return)g(of)h(the)g (routine,)f(the)h(handle)f(is)h(unde-)164 1832 y(\014ned.)164 1952 y Fi(MPI)p 279 1952 V 20 w(ASSOCIA)-5 b(TED\(handle,)21 b(t)n(yp)r(e\))164 2044 y(IN)d(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)164 2142 y Fi(OUT)j(t)n(yp)r(e)k Fl(state)237 2233 y(Returns)16 b(the)g(t)o(yp)q(e)f(of)h(the)g(ob)s(ject)f(the)h(handle)f(is)h(curren)o(tly) e(asso)q(ciated)j(with,)e(if)164 2293 y(suc)o(h)g(exists.)20 b(Returns)c(the)f(sp)q(ecial)g(t)o(yp)q(e)f Fg(MPI)p 1041 2293 16 2 v 18 w(NULL)g Fl(if)g(the)h(handle)h(is)f(not)h(curren)o(tly)164 2354 y(asso)q(ciated)h(with)f(an)o(y)g(ob)s(ject.)237 2414 y(MPI)h(ma)o(y)f(pro)o(vide)h(prede\014ned)h(opaque)g(ob)s(jects)f(and)h (prede\014ned,)g(static)f(han-)164 2474 y(dles)f(to)g(these)g(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(jects)g(ma)o(y)f(not)h(b)q(e)h(destro)o(y)o(ed.)961 2599 y(2)p eop %%Page: 3 3 bop 164 307 a Fi(List)13 b(of)g(handles)50 b Fl(An)11 b(MPI)g(call)f(ma)o(y)g (need)h(a)g(parameter)f(that)i(is)f(a)h Fh(list)h(of)g(hand)r(les)p Fl(.)164 367 y(In)k(C,)g(suc)o(h)g(list)g(will)f(b)q(e)h(a)h(record)f(with)g (one)h(comp)q(onen)o(t)e(b)q(eing)h(the)g(length)h(of)f(the)164 428 y(list,)j(the)g(other)g(comp)q(onen)o(ts)f(b)q(eing)h(an)h(arra)o(y)f(of) g(p)q(oin)o(ters.)33 b(In)20 b(F)l(ortran,)h(the)f(list)164 488 y(will)15 b(b)q(e)h(an)h(arra)o(y)g(of)f(in)o(tegers,)f(the)h(\014rst)h (one)f(of)h(whic)o(h)e(is)h(the)g(length)g(of)h(the)f(list.)164 657 y Fk(Discussion:)54 b Fj(The)19 b(mec)o(hanism)h(for)e(opaque)h(ob)s (jects)f(used)h(here)g(follo)o(ws)g(the)g(POSIX)164 714 y(F)l(ortran)14 b(binding)j(standard.)j(An)15 b(alternativ)o(e)h(c)o(hoice)g(is)g(to)f(ha)o (v)o(e)f(di\013eren)o(t)i(t)o(yp)q(e)f(declara-)164 770 y(tions)g(for)f(eac)o (h)h(t)o(yp)q(e)h(of)e(opaque)h(ob)s(ject.)k(Then,)d(opaque)f(ob)s(jects)f (are)h(created/destro)o(y)o(ed)164 826 y(lik)o(e)e(regular)f(v)m(ariables,)h (rather)f(than)f(b)o(y)h(MPI)g(calls;)i(they)e(are)f(still)j(accessed)e(and)g (up)q(dated)164 883 y(only)k(via)f(MPI)g(functions.)164 1193 y Fi(1.2.2)55 b(State)164 1286 y Fl(MPI)12 b(pro)q(cedures)i(use)e(at)i(v)m (arious)f(places)g(argumen)o(ts)f(with)h Fh(state)g Fl(t)o(yp)q(es.)20 b(The)13 b(v)m(alues)164 1346 y(of)j(suc)o(h)f(data)i(t)o(yp)q(e)e(are)g(all) g(iden)o(ti\014ed)g(b)o(y)g(names,)f(and)i(no)g(op)q(eration)h(is)e (de\014ned)g(on)164 1406 y(them.)k(F)l(or)14 b(example,)f(the)h Fg(MPI)p 759 1406 16 2 v 17 w(CREATE)e Fl(routine)i(has)i(a)e(state)h(t)o(yp) q(e)f(parameter)f(with)164 1466 y(v)m(alues)j Fg(MPI)p 390 1466 V 18 w(PERSISTEN)o(T)d Fl(and)k Fg(MPI)p 853 1466 V 17 w(EPHEMERAL)p Fl(.)237 1527 y(An)e Fg(enumeratio)o(n)d Fl(declared)j(in)g(an) h(included)e(MPI.h)g(\014le)h(will)f(b)q(e)i(used)f(in)g(C)h(for)164 1587 y(state)g(datat)o(yp)q(es.)22 b(The)17 b(F)l(ortran)f(77)h(mec)o(hanism) c(needs)j(to)h(b)q(e)f(decided.)164 1767 y Fk(Discussion:)67 b Fj(Named)22 b(in)o(teger)h(constan)o(ts)e(can)h(b)q(e)g(used)h(in)g(F)l (ortran)d(90,)j(using)g(the)164 1827 y Ff(PARAMETER)17 b Fj(mec)o(hanism.)28 b(The)18 b(constan)o(t)f(declarations)i(can)f(b)q(e)g(made)g(a)o(v)m(ailable) i(via)e(an)164 1888 y Ff(INCLUDE)e Fj(\014le.)25 b(F)l(ortran)16 b(77)g(do)q(es)h(not)f(seem)h(to)f(o\013er)g(an)o(y)h(con)o(v)o(enien)o(t)g (mec)o(hanism.)25 b(One)164 1948 y(p)q(ossibilit)o(y)c(is)e(to)f(sp)q(ecify)j (explicit)g(in)o(teger)e(v)m(alues,)h(and)f(allo)o(w)g(the)g(use)g(of)f (named)i(con-)164 2008 y(stan)o(ts)15 b(with)i(those)f(F)l(ortran)f(77)h (compilers)i(that)d(supp)q(ort)i(them)f(con)o(v)o(enien)o(tly)l(.)25 b(Another)164 2068 y(p)q(ossibilit)o(y)17 b(is)f(to)f(use)g(c)o(haracter)g (strings,)f(rather)h(than)g(in)o(tegers.)164 2319 y Fi(1.2.3)55 b(Named)18 b(constan)n(ts)164 2411 y Fl(MPI)j(pro)q(cedures)i(sometime)o(s)c (assign)k(a)g(sp)q(ecial)e(meaning)g(to)h(a)g(sp)q(ecial)g(v)m(alue)g(of)164 2471 y(a)f(basic)g(t)o(yp)q(e)f(parameter;)h(e.g.)34 b Fg(tag)20 b Fl(is)h(an)g(in)o(teger)e(v)m(alued)i(parameter)f(of)h(p)q(oin)o(t-)961 2599 y(3)p eop %%Page: 4 4 bop 164 307 a Fl(to-p)q(oin)o(t)23 b(comm)o(unic)o(ation)d(op)q(erations,)25 b(with)d(a)h(sp)q(ecial)f Fg(DONTCARE)d Fl(v)m(alue.)39 b(Suc)o(h)164 367 y(parameters)14 b(will)f(ha)o(v)o(e)h(a)h(range)g(of)g(regular)f(v)m (alues,)h(whic)o(h)f(is)g(a)h(prop)q(er)g(subrange)h(of)164 428 y(the)i(range)h(of)g(v)m(alues)f(of)h(the)f(corresp)q(onding)h(basic)f(t) o(yp)q(e;)h(sp)q(ecial)f(v)m(alues)g(\(suc)o(h)g(as)164 488 y(DONTCARE\))d(will)e(b)q(e)i(outside)g(the)g(regular)g(range.)21 b(The)15 b(range)g(of)g(regular)g(v)m(alues)164 548 y(can)d(b)q(e)g(queried,) f(and)h(sometimes)d(set,)j(using)g(en)o(vironmen)o(t)d(inquiry)i(or)h(en)o (vironmen)o(t)164 608 y(setting)21 b(functions)g(\(Section)f Fi(??)p Fl(\).)37 b(The)21 b(sp)q(ecial)f(v)m(alues)h(are)g(pro)o(vided)g(b)o (y)f(named)164 668 y(constan)o(t,)c(that)h(are)f(made)f(a)o(v)m(ailable)h (via)g(an)g(MPI.h)g(include)f(\014le)g(in)h(a)h(C)f(binding.)164 838 y Fk(Discussion:)237 894 y Fj(Need)e(to)e(agree)g(on)h(a)g(F)l(ortran)f (mec)o(hanism)h(for)f(named)h(constan)o(ts)f(\(see)h(the)g(discussion)164 951 y(ab)q(o)o(v)o(e\).)237 1007 y(Implemen)o(ters)19 b(should)g(try)e(to)g (detect)h(illegal)i(uses)e(of)g(\\sp)q(ecial)h(v)m(alues".)29 b(Th)o(us,)18 b(the)164 1064 y(use)e(of)e(the)i Ff(DONTCARE)e Fj(v)m(alue)i(to)f(tag)f(a)h(message)g(sen)o(t)f(should)j(b)q(e)f(\015agged)f (as)f(an)h(error.)164 1374 y Fi(1.2.4)55 b(Choice)164 1466 y Fl(MPI)13 b(functions)i(sometime)o(s)d(use)i(parameters)f(with)g(a)i Fh(choic)n(e)f Fl(\(or)h(union\))f(data)h(t)o(yp)q(e.)164 1527 y(I.e.,)10 b(distinct)g(calls)g(to)i(the)e(same)g(routine)h(ma)o(y)e(pass)j (b)o(y)f(reference)e(actual)i(parameters)164 1587 y(of)18 b(di\013eren)o(t)f (t)o(yp)q(es.)25 b(The)17 b(mec)o(hanism)e(for)i(pro)o(viding)h(suc)o(h)f (parameters)g(will)f(di\013er)164 1647 y(from)e(language)i(to)g(language.)22 b(In)15 b(C,)g(a)h(formal)e(parameter)g(of)i(t)o(yp)q(e)e Fg(void)25 b(*)15 b Fl(will)f(b)q(e)164 1707 y(used,)i(with)g(an)h(actual)f(p)q(oin)o (ter)g(parameter.)k(in)c(F)l(ortran,)g(w)o(e)g(shall)g(c)o(heat.)164 1876 y Fk(Discussion:)237 1933 y Fj(The)i(F)l(ortran)e(77)h(standard)g(sp)q (eci\014es)i(that)d(the)i(t)o(yp)q(e)f(of)g(actual)h(argumen)o(ts)e(need)i (to)164 1989 y(agree)f(with)g(the)g(t)o(yp)q(e)h(of)e(dumm)o(y)h(argumen)o (ts;)g(no)g(construct)g(equiv)m(alen)o(t)i(to)e(C)f(p)q(oin)o(ters)164 2046 y(is)e(a)o(v)m(ailable.)21 b(Th)o(us,)14 b(it)g(w)o(ould)g(seem)g(that)f (there)g(is)i(no)e(standard)g(conforming)h(mec)o(hanism)164 2102 y(to)g(supp)q(ort)h(c)o(hoice)g(parameters.)k(Ho)o(w)o(ev)o(er,)14 b(most)g(F)l(ortran)f(compiler)j(either)f(don't)g(c)o(hec)o(k)164 2159 y(t)o(yp)q(e)i(consistency)h(of)f(calls)h(to)e(external)h(routines,)h (or)e(supp)q(ort)h(a)g(sp)q(ecial)i(mec)o(hanism)e(to)164 2215 y(link)i(foreign)e(\(e.g.,)g(C\))f(routines.)27 b(I)18 b(suggest)e(that)h(w)o (e)g(accept)h(this)f(nonconformit)o(y)h(with)164 2272 y(F)l(ortran)10 b(77)h(standard.)18 b(I.e.,)12 b(w)o(e)f(accept)h(that)f(the)g(same)h (routine)g(ma)o(y)e(b)q(e)j(passed)e(an)h(actual)164 2328 y(parameter)i(of)h (a)g(di\013eren)o(t)g(t)o(yp)q(e)h(at)e(distinct)j(calls.)237 2385 y(Generic)g(routines)g(can)f(b)q(e)h(used)f(in)h(F)l(ortran)e(90)g(to)h (pro)o(vide)g(a)g(standard)g(conforming)164 2441 y(solution.)j(This)12 b(solution)g(will)g(b)q(e)g(consisten)o(t)f(with)g(our)g(nonstandard)g (conforming)g(F)l(ortran)961 2599 y Fl(4)p eop %%Page: 5 5 bop 164 307 a Fj(77)15 b(solution.)164 632 y Fm(1.3)70 b(Pro)r(cesses)164 725 y Fl(An)16 b(MPI)h(program)f(is)h(executed)e(b)o(y)h(sev)o(eral)g (autonomous)h(pro)q(cesses)g(that)g(execute)164 785 y(eac)o(h)f(their)g(o)o (wn)i(co)q(de,)e(in)h(a)g(MIMD)f(st)o(yle.)22 b(The)17 b(co)q(des)g(executed) f(b)o(y)g(eac)o(h)g(pro)q(cess)164 845 y(need)21 b(not)h(b)q(e)f(iden)o (tical.)35 b(The)21 b(pro)q(cesses)h(comm)o(unic)o(ate)d(via)i(calls)f(to)i (MPI)f(com-)164 905 y(m)o(unication)13 b(primitiv)o(es.)18 b(T)o(ypically)l(,)13 b(eac)o(h)i(pro)q(cessor)h(executes)e(in)h(its)g(o)o (wn)h(address)164 965 y(space,)i(although)g(shared-memory)e(implem)o(en)o (tati)o(ons)g(of)i(MPI)f(are)h(p)q(ossible.)26 b(This)164 1025 y(do)q(cumen)o(t)20 b(sp)q(eci\014es)h(the)f(b)q(eha)o(vior)h(of)h(a)f (parallel)g(program)g(assuming)g(that)g(only)164 1086 y(MPI)c(calls)f(are)i (used)f(for)h(comm)o(uni)o(cation.)k(The)17 b(in)o(teraction)f(of)i(an)f(MPI) g(program)164 1146 y(with)k(other)g(p)q(ossible)g(means)f(of)h(comm)o (unication)d(\(e.g.,)j(shared)h(memory\))c(is)i(not)164 1206 y(sp)q(eci\014ed.)29 b(In)18 b(particular,)h(it)g(is)f(assumed)h(that)g (message)f(bu\013ers)i(at)f(distinct)f(pro-)164 1266 y(cessors)f(are)f (disjoin)o(t.)237 1326 y(MPI)k(do)q(es)g(not)h(sp)q(ecify)e(the)h(execution)f (mo)q(del)g(for)h(eac)o(h)f(pro)q(cess.)34 b(A)19 b(pro)q(cess)164 1387 y(can)e(b)q(e)f(sequen)o(tial,)f(or)i(can)g(b)q(e)f(m)o(ultithreaded,)e (with)i(threads)h(p)q(ossibly)g(executing)164 1447 y(concurren)o(tly)l(.)23 b(Care)18 b(has)g(b)q(een)f(tak)o(en)g(to)h(mak)o(e)d(MPI)i(\\thread-safe",)i (b)o(y)d(a)o(v)o(oiding)164 1507 y(the)g(use)g(of)h(implici)o(t)d(global)i (states.)237 1567 y(The)e(initial)e(allo)q(cation)h(of)h(pro)q(cesses)g(to)g (an)g(MPI)e(computation)h(and)h(their)f(bind-)164 1627 y(ing)f(to)h(ph)o (ysical)f(pro)q(cessors)h(is)g(not)g(sp)q(eci\014ed)e(b)o(y)h(the)h(program)f (itself.)19 b(It)12 b(is)g(exp)q(ected)164 1688 y(that)17 b(v)o(endors)f (will)g(pro)o(vide)g(mec)o(hanism)o(s)e(to)j(do)g(so)g(either)f(at)h(load)g (time)d(or)j(at)g(run)164 1748 y(time.)26 b(Suc)o(h)18 b(mec)o(hanism)o(s)e (will)i(allo)o(w)g(to)h(sp)q(ecify)e(the)i(initial)e(n)o(um)o(b)q(er)f(of)j (required)164 1808 y(pro)q(cesses,)c(the)g(co)q(de)g(to)g(b)q(e)f(executed)g (b)o(y)g(eac)o(h)g(initial)g(pro)q(cess,)h(and)g(the)g(allo)q(cation)164 1868 y(of)d(pro)q(cesses)h(to)g(pro)q(cessors.)21 b(Also,)12 b(the)g(curren)o(t)f(prop)q(osal)j(do)q(es)f(not)f(pro)o(vide)f(for)i(dy-)164 1928 y(namic)j(creation)h(or)g(deletion)g(of)g(pro)q(cesses)h(during)g (program)f(execution,)f(although)164 1989 y(it)g(is)f(in)o(tended)g(to)i(b)q (e)f(consisten)o(t)g(with)g(suc)o(h)f(extension.)21 b(Finally)l(,)14 b(the)i(curren)o(t)f(pro-)164 2049 y(p)q(osal)h(do)q(es)g(not)f(sp)q(ecify)f (a)i(naming)e(sc)o(heme)e(for)k(pro)q(cesses.)21 b(W)l(e)15 b(prop)q(ose)h(to)f(alw)o(a)o(ys)164 2109 y(iden)o(tify)j(pro)q(cesses)j (according)g(to)f(their)g(relativ)o(e)e(rank)j(in)f(a)g(con)o(text)g (\(group\),)h(so)164 2169 y(that,)f(e\013ectiv)o(ely)l(,)d(pro)q(cesses)i (are)h(iden)o(ti\014ed)d(b)o(y)i(consecutiv)o(e)f(in)o(tegers.)29 b(Absolute,)164 2229 y(system-wide)13 b(unique)h(pro)q(cess)h(id's)g(are)f (\(will)g(b)q(e\))h(needed)f(only)g(if)g(dynamic)f(pro)q(cess)164 2290 y(creation)h(is)g(to)h(b)q(e)f(supp)q(orted)i(\(in)e(suc)o(h)g(ev)o(en)o (tualit)o(y)e(w)o(e)h(prop)q(ose)j(to)f(use)f(handles)h(to)164 2350 y(opaque)i Fh(pr)n(o)n(c)n(ess)f(structur)n(es)g Fl(for)g(that)h(purp)q (ose\).)961 2599 y(5)p eop %%Page: 6 6 bop 164 307 a Fm(1.4)70 b(Con)n(texts)164 520 y Fk(Discussion:)46 b Fj(This)18 b(section)f(con)o(tains)g(a)f(prop)q(osal)h(for)f(use)i(of)e (con)o(texts)g(that)g(will)i(sub-)164 580 y(sume)13 b(groups.)19 b(It)13 b(b)q(orro)o(ws)g(hea)o(vily)h(on)f(the)g(curren)o(t)g(group)g(prop)q (osal.)20 b(This)14 b(prop)q(osal)f(has)164 640 y(not)i(y)o(et)g(b)q(een)h (discussed)h(in)f(MPI)f(meetings.)237 821 y Fl(A)h Fi(con)n(text)f Fl(consists)i(of:)237 923 y Fe(\017)24 b Fl(A)d(set)g(of)h(pro)q(cesses)g (that)g(curren)o(tly)d(b)q(elong)j(to)g(the)f(con)o(text)g(\(p)q(ossibly)g (all)286 983 y(pro)q(cesses,)16 b(or)h(a)g(prop)q(er)f(subset\).)237 1084 y Fe(\017)24 b Fl(A)19 b Fi(ranking)g Fl(of)h(the)f(pro)q(cesses)g (within)g(that)g(con)o(text,)g(i.e.,)f(a)h(n)o(um)o(b)q(ering)f(of)286 1145 y(the)g(pro)q(cesses)g(in)g(that)g(con)o(text)g(from)f(0)h(to)g Fd(n)13 b Fe(\000)f Fl(1,)18 b(where)g Fd(n)g Fl(is)g(the)g(n)o(um)o(b)q(er) 286 1205 y(of)e(pro)q(cesses)h(in)f(that)h(con)o(text.)237 1307 y(A)f(pro)q(cess)h(ma)o(y)e(b)q(elong)h(to)h(sev)o(eral)e(con)o(texts)h (at)g(the)g(same)g(time.)237 1367 y(An)o(y)i(in)o(terpro)q(cess)f(comm)o (unication)f(o)q(ccurs)i(within)g(a)h(con)o(text,)e(and)i(messages)164 1427 y(sen)o(t)i(within)f(one)h(con)o(text)f(can)h(b)q(e)g(receiv)o(ed)e (only)i(within)f(the)h(same)f(con)o(text.)34 b(A)164 1487 y(con)o(text)13 b(is)h(sp)q(eci\014ed)g(using)h(a)f Fh(c)n(ontext)j(hand)r(le)f Fl(\(i.e.,)d(a)h(handle)g(to)h(an)g(opaque)f(ob)s(ject)164 1547 y(that)23 b(iden)o(ti\014es)d(a)j(con)o(text\).)38 b(Con)o(text)22 b(handles)g(cannot)h(b)q(e)f(transferred)g(for)h(one)164 1608 y(pro)q(cess)18 b(to)g(another;)g(they)f(can)h(b)q(e)f(used)h(only)f(on)h (the)f(pro)q(cess)h(where)f(they)g(where)164 1668 y(created.)237 1728 y(F)l(ollo)o(ws)f(examples)e(of)j(p)q(ossible)f(uses)h(for)f(con)o (texts.)164 1858 y Fi(Lo)r(osely)h(sync)n(hronous)i(library)g(call)h(in)n (terface)49 b Fl(Consider)16 b(the)f(case)h(where)g(a)164 1918 y(parallel)e(application)i(executes)e(a)h(\\parallel)g(call")g(to)h(a)f (library)g(routine,)g(i.e.,)e(where)164 1978 y(all)i(pro)q(cesses)i(transfer) f(con)o(trol)g(to)g(the)g(library)f(routine.)21 b(If)16 b(the)f(library)h(w)o (as)g(dev)o(el-)164 2038 y(op)q(ed)k(separately)l(,)f(then)h(one)f(should)h (b)q(ew)o(are)f(of)h(the)f(p)q(ossibilit)o(y)g(that)g(the)h(library)164 2099 y(co)q(de)12 b(ma)o(y)f(receiv)o(e)f(b)o(y)i(mistak)o(e)e(messages)h (send)i(b)o(y)e(the)h(caller)f(co)q(de,)i(and)g(vice-v)o(ersa.)164 2159 y(T)l(o)19 b(prev)o(en)o(t)f(suc)o(h)g(o)q(ccurrence)h(one)g(migh)o(t)e (use)i(a)g(barrier)f(sync)o(hronization)h(b)q(efore)164 2219 y(and)f(after)f(the)h(parallel)e(library)h(call.)24 b(Instead,)17 b(one)h(can)g(allo)q(cate)f(a)h(di\013eren)o(t)e(con-)164 2279 y(text)e(to)g(the)h(library)l(,)e(th)o(us)h(prev)o(en)o(ting)f(un)o(w)o(an)o (ted)h(in)o(terference.)k(No)o(w,)d(the)f(transfer)164 2339 y(of)j(con)o(trol)e(to)i(the)f(library)g(need)f(not)i(b)q(e)f(sync)o (hronized.)961 2599 y(6)p eop %%Page: 7 7 bop 164 307 a Fi(F)-5 b(unctional)25 b(decomp)r(osition)e(and)g(mo)r(dular)g (co)r(de)f(dev)n(elopmen)n(t)49 b Fl(Often,)164 367 y(a)18 b(parallel)f(application)g(is)h(dev)o(elop)q(ed)e(b)o(y)h(in)o(tegrating)h (sev)o(eral)e(distinct)h(functional)164 428 y(mo)q(dules,)h(that)h(is)g(eac)o (h)f(dev)o(elop)q(ed)g(separately)l(.)28 b(Eac)o(h)19 b(mo)q(dule)f(is)g(a)h (parallel)f(pro-)164 488 y(gram)13 b(that)h(runs)g(on)g(a)g(dedicated)f(set)h (of)g(pro)q(cesses,)g(and)g(the)g(computation)f(consists)164 548 y(of)j(phases)h(where)f(mo)q(dules)g(compute)e(separately)l(,)i(in)o (termixe)o(d)e(with)i(global)g(phases)164 608 y(where)i(all)g(pro)q(cesses)h (comm)o(unicate.)25 b(It)18 b(is)g(con)o(v)o(enien)o(t)f(to)i(allo)o(w)f(eac) o(h)g(mo)q(dule)g(to)164 668 y(use)g(its)g(o)o(wn)g(priv)m(ate)g(pro)q(cess)h (n)o(um)o(b)q(ering)d(sc)o(heme,)g(for)i(the)g(in)o(tramo)q(dule)f(compu-)164 729 y(tation.)27 b(This)18 b(is)f(ac)o(hiev)o(ed)g(b)o(y)g(using)h(a)h(priv)m (ate)e(mo)q(dule)g(con)o(text)g(for)h(in)o(tramo)q(dule)164 789 y(computation,)d(and)i(a)g(global)f(con)o(text)g(for)g(in)o(termo)q(dule) e(comm)o(unic)o(ation.)164 919 y Fi(Collectiv)n(e)f(comm)n(unication)51 b Fl(MPI)10 b(supp)q(orts)i(collectiv)o(e)c(comm)o(unic)o(ation)h(within)164 979 y(dynamically)h(created)i(groups)h(of)g(pro)q(cesses.)20 b(Eac)o(h)13 b(suc)o(h)f(group)h(can)g(b)q(e)f(represen)o(ted)164 1039 y(b)o(y)h(a)i(distinct)e(con)o(text.)19 b(This)c(pro)o(vides)e(a)h (simple)e(mec)o(hanism)f(to)j(ensure)g(that)g(com-)164 1099 y(m)o(unication)20 b(that)i(p)q(ertains)g(to)g(collectiv)o(e)c(comm)o (unication)h(within)i(one)h(group)g(is)164 1159 y(not)17 b(confused)f(with)g (collectiv)o(e)e(comm)o(uni)o(cation)g(within)h(another)i(group.)164 1289 y Fi(Ligh)n(t)n(w)n(eigh)n(t)i(gang)f(sc)n(heduling)50 b Fl(Consider)15 b(an)g(en)o(vironmen)o(t)d(where)i(pro)q(cesses)164 1349 y(are)j(m)o(ultith)o(treaded.)k(Con)o(texts)c(can)h(b)q(e)f(used)g(to)h (pro)o(vide)e(a)i(mec)o(hanism)c(whereb)o(y)164 1410 y(all)21 b(pro)q(cesses)i(are)f(time-shared)e(b)q(et)o(w)o(een)h(sev)o(eral)g (parallel)g(executions,)h(and)h(can)164 1470 y(con)o(text)h(switc)o(h)h(from) f(one)h(parallel)g(execution)f(to)h(another,)j(in)d(a)g(lo)q(osely)h(syn-)164 1530 y(c)o(hronous)d(manner.)39 b(A)22 b(thread)h(is)f(allo)q(cated)h(on)g (eac)o(h)f(pro)q(cess)h(to)g(eac)o(h)f(parallel)164 1590 y(execution,)14 b(and)i(a)g(di\013eren)o(t)e(con)o(text)g(is)i(used)f(to)h(iden)o(tify)d(eac) o(h)i(parallel)f(execution.)164 1650 y(Th)o(us,)f(tra\016c)g(from)e(one)i (execution)f(cannot)i(b)q(e)f(confused)f(with)h(tra\016c)g(from)e(another)164 1711 y(execution.)31 b(The)20 b(blo)q(c)o(king)g(and)h(un)o(blo)q(c)o(king)e (of)h(threads)h(due)f(to)g(comm)o(unicati)o(on)164 1771 y(ev)o(en)o(ts)14 b(pro)o(vide)g(a)i(\\lazy")f(con)o(text)f(switc)o(hing)h(mec)o(hanism)o(.)j (This)d(can)h(b)q(e)f(extended)164 1831 y(to)j(the)f(case)g(where)g(the)h (parallel)e(executions)h(are)g(spanning)i(distinct)d(pro)q(cess)i(sub-)164 1891 y(sets.)j(\(MPI)16 b(do)q(es)h(not)g(require)e(m)o(ultithreaded)e(pro)q (cesses.\))164 2061 y Fk(Discussion:)39 b Fj(A)14 b(con)o(text)f(handle)i (migh)o(t)f(b)q(e)g(implemen)o(ted)i(as)e(a)f(p)q(oin)o(ter)h(to)g(a)f (structure)164 2117 y(that)f(consists)h(of)g(con)o(text)f(lab)q(el)j(\(that)d (is)h(carried)h(b)o(y)f(messages)f(sen)o(t)h(within)h(this)g(con)o(text\))164 2173 y(and)20 b(a)g(con)o(text)g(mem)o(b)q(er)g(table,)i(that)d(translates)h (pro)q(cess)h(ranks)e(within)j(a)e(con)o(text)f(to)164 2230 y(absolute)g(addresses)g(or)g(to)f(routing)h(information.)31 b(Of)19 b(course,)h(other)f(implemen)o(tations)164 2286 y(are)c(p)q(ossible,) i(including)h(implemen)o(tations)e(that)f(do)g(not)g(require)h(eac)o(h)g(con) o(text)f(mem)o(b)q(er)164 2343 y(to)g(store)f(a)h(full)h(list)g(of)f(the)g (con)o(text)g(mem)o(b)q(ers.)237 2399 y(Con)o(texts)g(can)h(b)q(e)g(used)g (only)h(on)e(the)h(pro)q(cess)g(where)g(they)g(w)o(ere)f(created.)22 b(Since)17 b(the)164 2456 y(con)o(text)e(carries)g(information)h(on)f(the)g (group)g(of)g(pro)q(cesses)h(that)e(b)q(elong)i(to)f(this)h(con)o(text,)961 2599 y Fl(7)p eop %%Page: 8 8 bop 164 307 a Fj(a)14 b(pro)q(cess)g(can)g(send)g(a)g(message)g(within)h(a)e (con)o(text)h(only)g(to)g(other)f(pro)q(cesses)i(that)e(b)q(elong)164 364 y(to)i(that)g(con)o(text.)20 b(Th)o(us,)15 b(eac)o(h)h(pro)q(cess)f (needs)i(to)e(k)o(eep)g(trac)o(k)g(only)h(of)f(the)h(con)o(texts)f(that)164 420 y(where)j(created)f(at)g(that)g(pro)q(cess;)h(the)g(total)f(n)o(um)o(b)q (er)h(of)f(con)o(texts)f(p)q(er)i(pro)q(cess)g(is)g(lik)o(ely)164 477 y(to)d(b)q(e)g(small.)237 533 y(The)23 b(only)f(di\013erence)i(I)e(see)h (b)q(et)o(w)o(een)f(this)h(curren)o(t)f(de\014nition)i(of)e(con)o(text,)h (whic)o(h)164 589 y(subsumes)18 b(the)f(group)h(concept,)g(and)f(a)g(pared)h (do)o(wn)f(de\014nition,)j(if)e(that)e(I)i(assume)f(here)164 646 y(that)g(pro)q(cess)g(n)o(um)o(b)q(ering)h(is)g(relativ)o(e)g(to)f(the)g (con)o(text,)g(rather)g(then)g(b)q(eing)i(global,)f(th)o(us)164 702 y(requiring)f(a)e(con)o(text)g(mem)o(b)q(er)h(table.)22 b(I)16 b(argue)g(that)f(this)h(is)g(not)g(m)o(uc)o(h)f(added)i(o)o(v)o (erhead,)164 759 y(and)e(giv)o(es)h(m)o(uc)o(h)f(additional)i(needed)f (functionalit)o(y)l(.)239 853 y Fc(\017)24 b Fj(If)14 b(a)f(new)g(con)o(text) g(is)h(created)g(b)o(y)f(cop)o(ying)h(a)f(previous)h(con)o(text,)f(then)h (one)f(do)q(es)h(not)286 909 y(need)19 b(a)e(new)h(mem)o(b)q(er)g(table;)i (rather,)d(one)h(needs)h(just)e(a)h(new)g(con)o(text)f(lab)q(el)j(and)286 966 y(a)e(new)h(p)q(oin)o(ter)g(to)e(the)i(same)f(old)h(con)o(text)f(mem)o(b) q(er)g(table.)30 b(This)19 b(holds)g(true,)g(in)286 1022 y(particular,)c(for) g(con)o(texts)f(that)h(include)j(all)e(pro)q(cesses.)239 1116 y Fc(\017)24 b Fj(A)13 b(con)o(text)g(mem)o(b)q(er)h(table)g(mak)o(es)f(sure) h(that)f(a)g(message)g(is)h(sen)o(t)f(only)h(to)f(a)h(pro)q(cess)286 1172 y(that)f(can)h(execute)g(in)h(the)f(con)o(text)f(of)g(the)h(message.)19 b(The)14 b(alternativ)o(e)g(mec)o(hanism,)286 1229 y(whic)o(h)e(is)g(c)o(hec) o(king)g(at)f(reception,)i(is)f(less)g(e\016cien)o(t,)g(and)g(requires)g (that)f(eac)o(h)g(con)o(text)286 1285 y(lab)q(el)18 b(b)q(e)g(system-wide)f (unique.)26 b(This)17 b(requires)g(that,)f(to)g(the)h(least,)g(all)g(pro)q (cesses)286 1342 y(in)e(a)f(con)o(text)f(execute)i(a)f(collectiv)o(e)i (agreemen)o(t)d(algorithm)i(at)e(the)h(creation)h(of)e(this)286 1398 y(con)o(text.)239 1492 y Fc(\017)24 b Fj(The)14 b(use)g(of)f(relativ)o (e)h(addressing)g(within)h(eac)o(h)f(con)o(text)f(is)h(needed)h(to)e(supp)q (ort)h(true)286 1548 y(mo)q(dular)19 b(dev)o(elopmen)o(t)g(of)f(sub)q (computations)h(that)e(execute)i(on)f(a)g(subset)h(of)f(the)286 1605 y(pro)q(cesses.)h(There)14 b(is)f(also)g(a)g(big)h(adv)m(an)o(tage)e(in) i(using)g(the)f(same)g(con)o(text)f(construct)286 1661 y(for)i(collectiv)o(e) j(comm)o(unications)f(as)f(w)o(ell.)164 2009 y Fi(1.4.1)55 b(Con)n(text)19 b(Op)r(erations)164 2101 y Fl(A)h(global)i(con)o(text)e Fi(ALL)g Fl(is)h(prede\014ned.)35 b(All)20 b(pro)q(cesses)h(b)q(elong)h(to)f (this)g(con)o(text)164 2162 y(when)d(computation)g(starts.)28 b(MPI)18 b(do)q(es)h(not)g(sp)q(ecify)e(ho)o(w)i(pro)q(cesses)g(are)f (initially)164 2222 y(rank)o(ed)h(within)f(the)h(con)o(text)f(ALL.)h(It)g(is) g(exp)q(ected)f(that)h(the)g(start-up)h(pro)q(cedure)164 2282 y(used)d(to)g(initiate)f(an)i(MPI)e(program)h(\(at)g(load-time)f(or)h (run-time\))e(will)h(pro)o(vide)g(in-)164 2342 y(formation)d(or)h(con)o(trol) g(on)g(this)g(initial)e(ranking)j(\(e.g.,)e(b)o(y)g(sp)q(ecifying)g(that)h (pro)q(cesses)164 2402 y(are)20 b(rank)o(ed)g(according)g(to)h(their)e (pid's,)h(or)h(according)f(to)g(the)g(ph)o(ysical)f(addresses)961 2599 y(8)p eop %%Page: 9 9 bop 164 307 a Fl(of)15 b(the)g(executing)f(pro)q(cessors,)i(or)f(according)g (to)h(a)f(n)o(um)o(b)q(ering)e(sc)o(heme)g(sp)q(eci\014ed)i(at)164 367 y(load)i(time\).)164 548 y Fk(Discussion:)24 b Fj(If)17 b(w)o(e)g(think)h(of)f(adding)h(new)f(pro)q(cesses)h(at)e(run-time,)i(then)f Ff(ALL)g Fj(con)o(v)o(eys)164 608 y(the)e(wrong)g(impression,)h(since)g(it)g (is)f(just)g(the)g(initial)j(set)d(of)f(pro)q(cesses.)237 789 y Fl(The)i(follo)o(wing)g(op)q(erations)i(are)e(a)o(v)m(ailable)g(for)g (creating)g(new)g(con)o(texts.)164 909 y Fi(MPI)p 279 909 17 2 v 20 w(COPY)p 461 909 V 22 w(CONTEXT\(new)n(con)n(text,)i(con)n(text\))237 969 y Fl(Create)f(a)g(new)f(con)o(text)g(that)h(includes)e(all)h(pro)q (cesses)i(in)e(the)g(old)h(con)o(text.)k(The)164 1029 y(rank)c(of)f(the)g (pro)q(cesses)h(in)f(the)h(previous)f(con)o(text)f(is)h(preserv)o(ed.)21 b(The)16 b(call)g(m)o(ust)f(b)q(e)164 1090 y(executed)j(b)o(y)i(all)f(pro)q (cesses)i(in)e(the)h(old)g(con)o(text.)31 b(It)19 b(is)h(a)g(blo)q(c)o(king)f (call:)28 b(No)20 b(call)164 1150 y(returns)c(un)o(til)f(all)h(pro)q(cesses)h (ha)o(v)o(e)e(called)h(the)g(function.)21 b(The)16 b(parameters)f(are)164 1264 y Fi(OUT)k(new)n(con)n(text)24 b Fl(handle)12 b(to)h(newly)f(created)g (con)o(text.)19 b(The)13 b(handle)f(should)h(not)286 1324 y(b)q(e)j(asso)q (ciated)i(with)e(an)g(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1426 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 1649 y Fk(Discussion:)35 b Fj(I)11 b(considered)h(adding)f(a)f(string)g (parameter,)g(to)g(pro)o(vide)h(a)e(unique)j(iden)o(ti\014er)164 1706 y(to)j(the)h(next)f(con)o(text.)21 b(But,)15 b(in)i(an)e(en)o(vironmen)o (t)h(where)g(pro)q(cesses)g(are)f(single)i(threaded,)164 1762 y(this)j(is)h(not)f(m)o(uc)o(h)g(help:)31 b(Either)20 b(all)h(pro)q(cesses)g (agree)f(on)g(the)g(order)g(they)g(create)g(new)164 1819 y(con)o(texts,)15 b(or)h(the)g(application)i(deadlo)q(c)o(ks.)23 b(A)16 b(k)o(ey)g(ma)o(y)f (help)i(in)g(an)f(en)o(vironmen)o(t)h(where)164 1875 y(pro)q(cesses)f(are)g (m)o(ultithreaded,)h(to)e(distinguish)j(call)f(from)e(distinct)i(threads)f (of)f(the)h(same)164 1932 y(pro)q(cess;)f(but)g(it)h(migh)o(t)f(b)q(e)h (simpler)g(to)f(use)g(a)g(m)o(utex)g(algorithm)g(at)g(eac)o(h)g(pro)q(cess.) 237 1992 y Fk(Implemen)o(tation)23 b(note:)k Fj(No)19 b(comm)o(unication)h (is)f(needed)i(to)d(create)h(a)f(new)i(con-)164 2052 y(text,)g(b)q(ey)o(ond)g (a)f(barrier)h(sync)o(hronization;)i(all)e(pro)q(cesses)g(can)g(agree)f(to)g (use)h(the)f(same)164 2112 y(naming)c(sc)o(heme)f(for)g(successiv)o(e)i (copies)f(of)f(the)g(same)g(con)o(text.)19 b(Also,)c(no)f(new)g(rank)g(table) 164 2172 y(is)i(needed,)g(just)f(a)g(new)g(con)o(text)g(lab)q(el)h(and)g(a)f (new)g(p)q(oin)o(ter)h(to)e(the)i(same)f(old)g(table.)164 2413 y Fi(MPI)p 279 2413 V 20 w(NEW)p 438 2413 V 20 w(CONTEXT\(new)n(con)n(text,)j (con)n(text,)g(k)n(ey)-5 b(,)18 b(index\))961 2599 y Fl(9)p eop %%Page: 10 10 bop 164 307 a Fi(OUT)19 b(new)n(con)n(text)24 b Fl(handle)13 b(to)g(newly)g(created)f(con)o(text)g(at)i(calling)e(pro)q(cess.)21 b(This)286 367 y(handle)16 b(should)h(not)g(b)q(e)f(asso)q(ciated)h(with)f (an)h(ob)s(ject)f(b)q(efore)g(the)g(call.)164 469 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 571 y Fi(IN)i(k)n(ey)24 b Fl(in)o(teger)164 672 y Fi(IN)18 b(index)25 b Fl(in)o(teger)237 787 y(A)20 b(new)g(con)o(text)f(is)g(created)h(for)g(eac)o(h)f(distinct)g(v)m (alue)h(of)g Fg(key)p Fl(;)g(this)g(con)o(text)f(is)164 847 y(shared)e(b)o(y)g(all)f(pro)q(cesses)h(that)h(made)d(the)i(call)f(with)h (this)g(k)o(ey)e(v)m(alue.)23 b(Within)16 b(eac)o(h)164 907 y(new)k(con)o(text)g(the)g(pro)q(cesses)h(are)g(rank)o(ed)f(according)h(to)g (the)f(order)h(of)f(the)h Fg(index)164 967 y Fl(v)m(alues)16 b(they)g(pro)o(vided;)f(in)h(case)g(of)h(ties,)e(pro)q(cesses)i(are)f(rank)o (ed)g(according)g(to)h(their)164 1027 y(rank)f(in)g(the)g(old)h(con)o(text.) 237 1088 y(This)e(call)e(is)h(blo)q(c)o(king:)20 b(No)14 b(call)f(returns)i (un)o(til)e(all)h(pro)q(cesses)g(in)g(the)g(old)h(con)o(text)164 1148 y(executed)g(the)h(call.)237 1208 y(P)o(articular)g(uses)g(of)h(this)f (function)g(are:)237 1268 y(\(i\))h(Reordering)h(pro)q(cesses:)25 b(All)16 b(pro)q(cesses)i(pro)o(vide)f(the)h(same)e Fg(key)h Fl(v)m(alue,)g(and)164 1328 y(pro)o(vide)e(their)h(index)f(in)h(the)g(new)g (order.)237 1389 y(\(ii\))e(Splitting)g(a)h(con)o(text)f(in)o(to)h(sub)q(con) o(texts,)f(while)g(preserving)g(the)h(old)g(relativ)o(e)164 1449 y(order)23 b(among)f(pro)q(cesses:)35 b(All)21 b(pro)q(cesses)i(pro)o (vide)f(the)g(same)g Fg(index)f Fl(v)m(alue,)i(and)164 1509 y(pro)o(vide)15 b(a)i(k)o(ey)e(iden)o(tifying)g(their)g(new)h(sub)q(con)o (text.)164 1629 y Fi(MPI)p 279 1629 17 2 v 20 w(RANK\(rank,)i(con)n(text\)) 164 1743 y(OUT)h(rank)24 b Fl(in)o(teger)164 1845 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 1959 y(Return)h(the)g(rank)h (of)f(the)g(calling)g(pro)q(cess)g(within)g(the)g(sp)q(eci\014ed)g(con)o (text.)164 2080 y Fi(MPI)p 279 2080 V 20 w(SIZE\(size,)j(con)n(text\))164 2194 y(OUT)g(size)24 b Fl(in)o(teger)164 2296 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 2410 y(Return)h(the)g(n)o(um)o(b)q(er)f(of)h (pro)q(cesses)h(that)g(b)q(elong)f(to)h(the)f(sp)q(eci\014ed)g(con)o(text.) 949 2599 y(10)p eop %%Page: 11 11 bop 164 307 a Fi(Usage)25 b(note)49 b Fl(Use)21 b(of)i(con)o(texts)e(for)i (libraries:)32 b(Eac)o(h)22 b(library)f(ma)o(y)g(pro)o(vide)g(an)164 367 y(initialization)15 b(routine)i(that)g(is)g(to)g(b)q(e)g(called)f(b)o(y)h (all)f(pro)q(cesses,)h(and)h(that)f(generate)164 428 y(a)g(con)o(text)e(for)i (the)f(use)g(of)g(that)h(library)l(.)237 488 y(Use)11 b(of)h(con)o(texts)e (for)i(functional)f(decomp)q(osition:)18 b(A)11 b(harness)h(program,)g (running)164 548 y(in)18 b(the)g(con)o(text)g Fg(ALL)f Fl(generates)i(a)g (sub)q(con)o(text)f(for)h(eac)o(h)f(mo)q(dule)f(and)i(then)f(starts)164 608 y(the)e(submo)q(dule)f(within)h(the)g(corresp)q(onding)h(con)o(text.)237 668 y(Use)k(of)g(con)o(texts)g(for)g(collectiv)o(e)e(comm)o(unic)o(ation:)29 b(A)21 b(con)o(text)f(is)h(created)g(for)164 729 y(eac)o(h)16 b(group)h(of)f(pro)q(cesses)h(where)f(collectiv)o(e)e(comm)o(uni)o(cation)g (is)i(to)g(o)q(ccur.)237 789 y(Use)i(of)g(con)o(texts)g(for)g(con)o (text-switc)o(hing)f(among)h(sev)o(eral)f(parallel)g(executions:)164 849 y(A)h(pream)o(ble)e(co)q(de)i(is)g(used)g(to)g(generate)g(a)h(di\013eren) o(t)e(con)o(text)g(for)i(eac)o(h)e(execution;)164 909 y(this)j(pream)o(ble)d (co)q(de)j(needs)g(to)g(use)f(a)i(m)o(utual)d(exclusion)g(proto)q(col)j(to)f (mak)o(e)e(sure)164 969 y(eac)o(h)e(thread)g(claims)e(the)i(righ)o(t)g(con)o (text.)164 1139 y Fk(Discussion:)59 b Fj(If)20 b(pro)q(cess)g(handles)h(are)e (made)h(explicit)i(in)e(MPI,)g(then)g(an)f(additional)164 1195 y(function)c(needed)h(is)f Fk(MPI)p 646 1195 16 2 v 18 w(PR)o(OCESS\(pro)q (cess,)i(con)o(text,)f(rank\))p Fj(,)e(whic)o(h)i(returns)e(a)164 1252 y(handle)i(to)f(the)g(pro)q(cess)h(iden)o(ti\014ed)h(b)o(y)e(the)g Ff(rank)g Fj(and)g Ff(context)g Fj(parameters.)237 1312 y(A)10 b(p)q(ossible)i(addition)f(is)g(a)f(function)h(of)f(the)g(form)f Fk(MPI)p 1199 1312 V 18 w(CREA)l(TE)p 1434 1312 V 19 w(CONTEXT\(new)o(con)o (text,)164 1372 y(list)p 236 1372 V 20 w(of)p 298 1372 V 19 w(pro)q(cess)p 484 1372 V 18 w(handles\))19 b Fj(whic)o(h)g(creates)f(a)g (new)h(con)o(text)e(out)h(of)g(an)g(explicit)i(list)f(of)164 1432 y(mem)o(b)q(ers)e(\(and)g(rank)g(them)g(in)h(their)g(order)f(of)g(o)q (ccurrence)h(in)g(the)f(list\).)26 b(This,)18 b(coupled)164 1492 y(with)13 b(a)f(mec)o(hanism)h(for)e(requiring)j(the)e(spa)o(wning)h(of) f(new)g(pro)q(cesses)h(to)f(the)g(computation,)164 1553 y(will)20 b(allo)o(w)e(to)g(create)g(a)g(new)g(all)h(inclusiv)o(e)i(con)o(text)c(that)h (includes)i(the)e(additional)i(pro-)164 1613 y(cesses.)k(Ho)o(w)o(ev)o(er,)16 b(I)h(opp)q(ose)g(the)f(idea)h(of)g(requiring)g(dynamic)h(pro)q(cess)f (creation)f(as)g(part)164 1673 y(of)f(MPI.)f(Man)o(y)h(implemen)o(ters)h(w)o (an)o(t)e(to)g(run)i(MPI)f(in)h(an)f(en)o(vironmen)o(t)g(where)g(pro)q (cesses)164 1733 y(are)g(statically)h(allo)q(cated)g(at)f(load-time.)164 1983 y Fi(1.4.2)55 b(Error)18 b(Handling)164 2076 y Fl(It)12 b(is)g(assumed)g(that)g(MPI)g(is)g(implem)o(en)o(te)o(d)e(on)i(top)h(of)g(an) f(error-free)g(comm)o(unicati)o(on)164 2136 y(subsystem:)18 b(A)12 b(message)g(sen)o(t)g(is)g(alw)o(a)o(ys)g(receiv)o(ed)e(correctly)l(,) i(and)h(the)f(user)g(do)q(es)h(not)164 2196 y(need)e(to)g(c)o(hec)o(k)e(for)i (transmission)g(errors,)g(time-outs,)g(and)g(the)g(lik)o(es.)18 b(In)11 b(other)g(w)o(ords,)164 2256 y(MPI)21 b(do)q(es)h(not)g(pro)o(vide)e (mec)o(hanisms)f(to)i(deal)g(with)h(failures)e(in)i(the)f(underlying)164 2316 y(comm)o(unic)o(ation)15 b(subsystem)g({)j(it)e(is)h(the)f(resp)q (onsibilit)o(y)g(of)h(the)f(MPI)h(implem)o(en)n(ter)164 2377 y(to)i(insulate)g(the)g(user)f(from)g(suc)o(h)h(errors)g(\(or)g(to)h (re\015ect)e(them)f(as)j(global)f(program)164 2437 y(failures\).)i(The)16 b(same)f(holds)i(true)f(for)g(no)q(de)h(failures.)949 2599 y(11)p eop %%Page: 12 12 bop 237 307 a Fl(Of)15 b(course,)h(MPI)f(programs)g(ma)o(y)f(still)g(b)q(e)i (erroneous.)21 b(A)15 b Fi(program)j(error)d Fl(can)164 367 y(o)q(ccur)i(when)h(an)f(MPI)g(call)g(is)g(called)f(with)h(an)h(incorrect)e (parameter)g(\(non-existing)164 428 y(destination)f(in)g(a)h(send)g(op)q (eration,)g(bu\013er)f(to)q(o)i(small)c(in)i(a)h(receiv)o(e)d(op)q(eration,)j (etc.\))164 488 y(This)k(t)o(yp)q(e)g(of)h(error)f(w)o(ould)h(o)q(ccur)f(in)g (an)o(y)g(impleme)o(n)o(tation.)31 b(In)20 b(addition,)h(a)g Fi(re-)164 548 y(source)g(error)e Fl(ma)o(y)e(o)q(ccur)i(when)g(a)h(program)f (exceeds)e(the)i(amoun)o(t)f(of)i(a)o(v)m(ailable)164 608 y(system)e (resources)i(\(n)o(um)o(b)q(er)e(of)i(p)q(ending)h(messages,)f(system)e (bu\013ers,)j(etc.\).)31 b(The)164 668 y(o)q(ccurrence)13 b(of)i(this)f(t)o (yp)q(e)f(of)i(error)f(dep)q(ends)g(on)h(the)f(amoun)o(t)f(of)i(a)o(v)m (ailable)e(resources)164 729 y(in)19 b(the)g(system)f(and)j(the)e(resource)g (allo)q(cation)h(mec)o(hanism)c(used;)21 b(this)e(ma)o(y)f(di\013er)164 789 y(from)f(system)g(to)h(system.)26 b(The)18 b(recomme)o(nded)d(impleme)o (n)o(tation)h(pro\014le)h(pro)o(vides)164 849 y(sev)o(eral)d(mec)o(hanism)o (s)f(to)i(alleviate)e(the)i(p)q(ortabilit)o(y)f(problem)f(this)i(represen)o (ts.)20 b(One)164 909 y(can)c(also)h(write)f Fi(safe)g Fl(programs,)g(that)h (are)f(not)h(sub)s(ject)e(to)i(resource)f(errors.)237 969 y(All)21 b(MPI)g(pro)q(cedure)h(calls)f(return)h(an)h(error)e(parameter)g(that)h (indicates)g(suc-)164 1029 y(cessful)g(completion)f(of)i(the)f(op)q(eration,) j(or)e(the)f(error)h(condition)f(that)h(o)q(ccurred,)164 1090 y(otherwise.)237 1150 y(The)h(recommende)o(d)d(implem)o(en)o(tation)g (pro\014le)j(in)f(a)h(POSIX)f(en)o(vironmen)o(t)e(is)164 1210 y(for)h(an)o(y)f(MPI)g(routine)g(that)g(encoun)o(ters)g(a)h(reco)o(v)o (erable)e(error)h(to)h(store)f(an)h(error)164 1270 y(n)o(um)o(b)q(er)f(in)h (a)h(global)g(v)m(ariable)f(\()p Fh(errno)h Fl(in)f(a)h(C)g(en)o(vironmen)o (t\))d(and)j(generate)f(an)164 1330 y Fh(MPI)d(err)n(or)e(signal)p Fl(,)i(using)g(a)f(sp)q(ecial)g(signal)g(v)m(alue.)27 b(The)18 b(default)g(handler)g(for)g(this)164 1391 y(signal)13 b(terminates)e(the)h (execution)g(of)h(all)f(in)o(v)o(olv)o(ed)e(pro)q(cesses,)k(with)f(a)g (suitable)f(error)164 1451 y(message)19 b(b)q(eing)i(returned)f(to)g(the)g (user.)33 b(Ho)o(w)o(ev)o(er,)19 b(the)h(user)h(can)f(pro)o(vide)f(his)i(or) 164 1511 y(her)14 b(o)o(wn)g(signal)h(handling)f(routine.)20 b(In)14 b(particular,)g(the)g(user)g(can)g(sp)q(ecify)f(a)i(\\no)q(op")164 1571 y(signal)i(handler,)f(th)o(us)g(relegating)g(all)g(error)h(handling)f (to)h(the)g(user)f(co)q(de,)g(using)h(the)164 1631 y(error)f(parameters)f (returned)h(b)o(y)g(the)g(MPI)g(calls.)237 1692 y(MPI)f(calls)f(ma)o(y)g (initiate)g(op)q(erations)i(that)g(con)o(tin)o(ue)e(async)o(hronously)h (after)g(the)164 1752 y(call)k(returned.)30 b(Th)o(us,)20 b(the)f(op)q (eration)i(ma)o(y)d(return)h(with)g(a)h(co)q(de)g(indicating)f(suc-)164 1812 y(cessful)14 b(completion,)e(y)o(et)i(later)g(cause)h(an)g(error)f (exception)g(to)h(b)q(e)f(raised.)21 b(If)14 b(there)g(is)164 1872 y(a)i(subsequen)o(t)f(call)g(that)h(relates)f(to)g(the)h(same)e(op)q (eration)i(\(e.g.,)f(a)h(call)e(that)i(v)o(eri\014es)164 1932 y(that)f(an)f(async)o(hronous)i(op)q(eration)f(has)g(completed\))d(then)i (the)g(error)g(parameter)f(as-)164 1993 y(so)q(ciated)18 b(with)f(this)g (call)g(will)f(b)q(e)i(used)f(to)h(indicate)e(the)h(nature)h(of)g(the)f (error.)24 b(In)18 b(a)164 2053 y(few)13 b(cases,)h(the)g(error)f(ma)o(y)f(o) q(ccur)i(after)g(all)f(calls)g(that)h(relate)f(to)h(the)f(op)q(eration)i(ha)o (v)o(e)164 2113 y(completed,)e(so)k(that)g(no)f(error)g(parameter)f(can)h(b)q (e)h(used)f(to)g(indicate)f(the)h(nature)h(of)164 2173 y(the)h(error)h (\(e.g.,)f(an)h(error)f(in)h(a)g(send)f(with)h(the)f(ready)g(mo)q(de\).)28 b(In)18 b(suc)o(h)g(cases,)h(an)164 2233 y(error)d(will)f(b)q(e)i (undetected,)e(if)g(the)h(user)h(disabled)e(the)h(MPI)g(error)g(signal.)164 2414 y Fk(Discussion:)40 b Fj(The)16 b(alternativ)o(e)f(c)o(hoice)h(is)g(to)f (ha)o(v)o(e)f(fatal)h(and)h(non-fatal)f(signals.)949 2599 y Fl(12)p eop %%Page: 13 13 bop 164 452 a Fm(1.5)70 b(Messages)164 544 y Fl(A)16 b(message)f(consists)i (of)g(an)f Fh(envelop)n(e)i Fl(and)f Fh(data)p Fl(.)164 674 y Fi(1.5.1)55 b(Data)164 766 y Fl(The)17 b(data)h(part)g(of)f(a)h(message)e (consists)i(of)f(a)h(sequence)e(of)h(v)m(alues,)g(eac)o(h)g(of)g(a)h(basic) 164 826 y(datat)o(yp)q(e)k(in)f(the)g(host)h(language.)37 b(Th)o(us,)22 b(in)f(F)l(ortran,)i(a)f(message)e(consists)i(of)g(a)164 887 y(sequence)14 b(of)h(v)m(alues)h(that)f(are)g(eac)o(h)g(of)g(t)o(yp)q(e)g Fg(INTEGER)p Fl(,)d Fg(REAL)p Fl(,)h Fg(DOUBLE)24 b(PRECISION)o Fl(,)164 947 y Fg(COMPLEX)p Fl(,)16 b Fg(LOGICAL)p Fl(,)f(or)k(\(length)g (1\))f Fg(CHARACTER)p Fl(.)d(A)k(message)e(ma)o(y)g(mix)g(v)m(alues)i(of)164 1007 y(di\013eren)o(t)c(t)o(yp)q(es.)164 1188 y Fk(Discussion:)40 b Fj(Ma)o(y)15 b(also)g(need)h Ff(DOUBLE)23 b(COMPLEX)14 b Fj(in)i(F)l(ortran.)164 1438 y Fi(1.5.2)55 b(En)n(v)n(elop)r(e)164 1530 y Fl(The)16 b(follo)o(wing)g(information)f(is)h(asso)q(ciated)i(with)e (eac)o(h)f(message:)164 1632 y Fi(source)24 b Fl(The)16 b(rank)h(the)f (sending)g(pro)q(cess)164 1734 y Fi(destination)25 b Fl(The)17 b(rank)f(of)h(the)f(receiving)e(pro)q(cess)164 1835 y Fi(tag)24 b Fl(User)16 b(de\014ned)164 1937 y Fi(con)n(text)24 b Fl(handle)237 2039 y(The)15 b(range)g(of)g(v)m(alid)f(v)m(alues)h(for)g(the)f Fi(source)g Fl(and)h Fi(destination)h Fl(\014elds)e(is)h Fg(0)25 b(...)164 2099 y(n-1)p Fl(,)16 b(where)h Fg(n)f Fl(is)h(the)g(n)o(um)o(b)q (er)f(of)h(pro)q(cesses)h(in)f(the)g(curren)o(t)f(con)o(text.)23 b(The)17 b(ranges)164 2159 y(of)j(v)m(alid)g(v)m(alues)g(for)h Fg(tag)e Fl(is)g(impleme)o(n)o(tation)e(dep)q(enden)o(t,)k(and)f(can)g(b)q(e) h(found)f(b)o(y)164 2219 y(calling)13 b(a)h(suitable)g(query)f(function,)h (as)g(describ)q(ed)g(in)f(Section)h Fi(??)p Fl(.)21 b Fg(Context)11 b Fl(should)164 2279 y(b)q(e)16 b(a)h(con)o(text)e(shared)i(b)o(y)f(b)q(oth)h (source)f(and)h(destination.)237 2340 y(The)h Fg(tag)e Fl(\014eld)h(can)h(b)q (e)g(arbitrarily)e(set)i(b)o(y)f(the)g(application,)g(and)h(can)g(b)q(e)g (used)164 2400 y(to)f(distinguish)f(di\013eren)o(t)f(messages.)949 2599 y(13)p eop %%Page: 14 14 bop 237 307 a Fl(The)12 b(actual)g(mec)o(hanism)d(used)j(to)g(asso)q(ciate)h (an)f(en)o(v)o(elop)q(e)f(with)g(a)i(message)e(is)g(im-)164 367 y(plemen)o(tation)h(dep)q(enden)o(t;)i(some)f(of)i(the)f(information)f (\(e.g.,)h Fi(sender)f Fl(or)i Fi(receiv)n(er)p Fl(\))164 428 y(ma)o(y)g(b)q(e)h(implicit,)c(and)17 b(need)f(not)h(b)q(e)f(explicitly)e (carried)h(b)o(y)h(a)h(message.)164 572 y Fm(1.6)70 b(Data)23 b(Bu\013ers)164 664 y Fl(The)18 b(basic)h(p)q(oin)o(t)f(to)h(p)q(oin)o(t)f (comm)o(unicati)o(on)e(op)q(erations)k(are)e Fi(send)g Fl(and)h Fi(receiv)n(e)p Fl(.)164 724 y(A)13 b Fi(send)h Fl(op)q(eration)h(creates)f (a)g(message;)g(the)f(message)h(data)g(is)g(assem)o(bled)e(from)h(the)164 785 y Fi(send)k(bu\013er)p Fl(.)j(A)14 b Fi(receiv)n(e)f Fl(op)q(eration)j (consumes)d(a)i(message;)f(the)g(message)g(data)h(is)164 845 y(mo)o(v)o(ed)g(in)o(to)i(the)f Fi(receiv)n(e)j(bu\013er)p Fl(.)k(The)17 b(sp)q(eci\014cation)g(of)g(send)g(or)h(receiv)o(e)c(bu\013ers) 164 905 y(uses)i(the)g(same)g(syn)o(tax.)237 965 y(A)j(bu\013er)g(consists)h (of)g(a)f(sequence)f Fi(bu\013er)k(comp)r(onen)n(ts)p Fl(.)30 b(Eac)o(h)19 b(comp)q(onen)o(t)164 1025 y(consists)14 b(of)h(a)f(sequence)f (v)m(ariables)h(of)h(the)f(same)f(basic)h(t)o(yp)q(e.)20 b(There)13 b(are)h(three)g(basic)164 1086 y(t)o(yp)q(es)i(of)h(bu\013er)f(comp)q(onen)o (ts:)164 1199 y Fi(blo)r(c)n(k)25 b Fl(A)15 b(sequence)h(of)g(con)o(tiguous)h (v)m(alues)f(of)h(the)f(same)f(basic)h(t)o(yp)q(e,)f(sp)q(eci\014ed)h(b)o(y) 286 1301 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1382 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e(\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1462 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)164 1564 y Fi(v)n(ector)24 b Fl(A)14 b(sequence)f(of)i(equally)f(spaced)h(and)g(equally)e(sized)h(blo)q (c)o(ks)h(of)g(elemen)o(ts)d(of)286 1624 y(the)k(same)f(basic)h(t)o(yp)q(e,)g (sp)q(eci\014ed)f(b)o(y)286 1726 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1807 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e (\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1887 y Fi(stride)24 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(b)q(et)o(w)o(een)h(the)h(start)h (of)f(eac)o(h)g(blo)q(c)o(k)286 1968 y Fi(len)n(blk)25 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(in)h(eac)o(h)h(blo)q(c)o(k)g(\()p Fg(lenblk)c Fe(\024)h Fg(stride)p Fl(\))286 2049 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)286 2151 y(Note)10 b(that)i(a)f (constan)o(t)g(stride)f(b)q(ecomes)g(con)o(tiguous)h(when)g Fg(stride)24 b(=)h(lenblk)p Fl(.)286 2211 y(A)c(v)o(ector)g(bu\013er)g(comp)q (onen)o(t)g(can)h(b)q(e)f(an)h(arbitrary)g(submatrix)e(of)i(a)g(t)o(w)o(o-) 286 2271 y(dimensional)14 b(matrix.)164 2372 y Fi(indexed)24 b Fl(A)16 b(sequence)f(of)i(elemen)o(t)o(s)d(of)j(the)f(same)f(basic)h(t)o (yp)q(e,)f(sp)q(eci\014ed)h(b)o(y)286 2474 y Fi(start)24 b Fl(initial)15 b(elemen)o(t)949 2599 y(14)p eop %%Page: 15 15 bop 286 307 a Fi(list)p 363 307 17 2 v 21 w(of)p 429 307 V 20 w(indices)26 b Fl(List)12 b(of)g(displacemen)o(ts)d(of)j(the)f(elemen)o (ts)e(in)j(the)f(bu\013er)h(com-)393 367 y(p)q(onen)o(ts,)17 b(relativ)o(e)d(to)j(the)f(initial)f(elemen)n(t.)286 443 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)237 534 y(F)l(or)f(example,)d(if)i(a)h(F)l(ortran)g(arra)o(y)g(is)f(declared)g (as)h Fg(double)24 b(precision)e(a\(10\))p Fl(,)164 594 y(then)15 b(the)f(tuple)g Fd(<)p Fg(\(a\(3\),)24 b(\(0,3,6,2\))o(,)f(DOUBLE)p Fd(>)12 b Fl(sp)q(eci\014es)j(a)g(bu\013er)g(comp)q(onen)o(t)164 655 y(with)h(en)o(tries)f Fg(a\(3\))24 b(a\(6\),)g(a\(9\),)g(a\(5\))p Fl(.)164 824 y Fk(Discussion:)56 b Fj(Do)19 b(w)o(e)g(allo)o(w)g(en)o(tries)h (to)f(b)q(e)h(rep)q(eated)f(in)h(an)g(indexed)g(bu\013er)g(comp)q(o-)164 880 y(nen)o(t?)237 937 y(Do)15 b(w)o(e)g(allo)o(w)g(di\013eren)o(t)g (bu\013er)h(comp)q(onen)o(ts)f(to)f(o)o(v)o(erlap?)237 993 y(Do)e(w)o(e)g(require)h(in)g(an)f(v)o(ector)g(bu\013er)g(comp)q(onen)o(t)g (that)g Ff(len)g Fj(b)q(e)g(a)g(m)o(ultiple)i(of)e Ff(lenblk)p Fj(?)237 1234 y Fl(A)j(bu\013er)h(is)f(describ)q(ed)f(b)o(y)h(an)h(opaque)f (ob)s(ject)g(accessed)g(via)g(a)h Fi(bu\013er)h(handle)p Fl(.)164 1294 y(Suc)o(h)f(ob)s(ject)g(is)g(created)f(and)i(destro)o(y)o(ed)f(via)g (calls)f(to)i Fg(MPI)p 1292 1294 16 2 v 17 w(CREATE)e Fl(and)h Fg(MPI)p 1652 1294 V 18 w(FREE)p Fl(.)164 1354 y(It)k(is)h(asso)q(ciated)h (with)f(successiv)o(e)e(bu\013er)i(comp)q(onen)o(ts)f(b)o(y)g(calling)h(in)f (succession)164 1415 y(one)f(of)g(the)g(functions)g Fg(MPI)p 695 1415 V 17 w(ADD)p 790 1415 V 18 w(BLOCK)p Fl(,)d Fg(MPI)p 1046 1415 V 18 w(ADD)p 1142 1415 V 17 w(VECTOR)h Fl(or)i Fg(MPI)p 1472 1415 V 17 w(ADD)p 1567 1415 V 18 w(INDEX)p Fl(,)d(in)164 1475 y(order)h(to)g(app)q(end)h(a)g(comp)q(onen)o(t)e(to)h(the)g(bu\013er)g (asso)q(ciated)h(with)f(a)g(bu\013er)g(handle.)164 1535 y(A)f(bu\013er)h(ob)s (ject)f(can)g(b)q(e)h(destro)o(y)o(ed)e(only)h(if)g(there)g(is)g(no)h(p)q (ending)g(comm)o(unicati)o(on)164 1595 y(op)q(eration)j(using)g(it.)31 b(After)19 b(a)h(bu\013er)g(ob)s(ject)f(is)g(destro)o(y)o(ed)g(the)g(asso)q (ciated)i(bu\013er)164 1655 y(handle)16 b(is)g(unde\014ned.)164 1776 y Fi(MPI)p 279 1776 17 2 v 20 w(ADD)p 427 1776 V 21 w(BLOCK\()i (bu\013er,)g(start,)g(len,)h(datat)n(yp)r(e\))237 1896 y Fl(App)q(end)d(a)h (blo)q(c)o(k)f(comp)q(onen)o(t)f(to)i(bu\013er.)k(The)16 b(parameters)g(are:) 164 1981 y Fi(INOUT)i(bu\013er)24 b Fl(bu\013er)17 b(handle)164 2077 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g (elemen)o(t)e(\(c)o(hoice)i(t)o(yp)q(e\))164 2173 y Fi(IN)j(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o(teger\))164 2269 y Fi(IN)18 b(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)164 2414 y Fi(MPI)p 279 2414 V 20 w(ADD)p 427 2414 V 21 w(VEC\()i(bu\013er,)f(len,)g(stride,)g(len)n(blk,)i(datat)n(yp) r(e)e(\))949 2599 y Fl(15)p eop %%Page: 16 16 bop 237 307 a Fl(App)q(end)16 b(a)h(v)o(ector)e(bu\013er)i(comp)q(onen)o(t)e (to)i(bu\013er.)k(The)c(parameters)e(are:)164 409 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 511 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g(elemen)o(t)e(\(c)o(hoice)i(t)o(yp)q(e\))164 612 y Fi(IN)j(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o (teger\))164 714 y Fi(IN)18 b(stride)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(b)q(et)o(w)o(een)i(the)h(start)h(of)g(eac)o(h)e(blo)q(c) o(k)h(\(in)o(teger\))164 816 y Fi(IN)i(len)n(blk)26 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(in)j(eac)o(h)g(blo)q(c)o(k)f(\(in)o(teger\))164 917 y Fi(IN)j(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)164 1079 y Fi(MPI)p 279 1079 17 2 v 20 w(ADD)p 427 1079 V 21 w(INDEX\()h (bu\013er,)g(start,)g(list)p 1077 1079 V 21 w(of)p 1143 1079 V 20 w(indices\))237 1200 y Fl(App)q(end)e(an)h(indexed)e(bu\013er)i(comp)q (onen)o(t)e(to)i(bu\013er.)k(The)c(parameters)e(are:)164 1301 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 1403 y Fi(start)24 b Fl(initial)15 b(p)q(osition)i(for)f(indexing)g(\(c)o(hoice)f (t)o(yp)q(e\))164 1505 y Fi(list)p 241 1505 V 21 w(of)p 307 1505 V 20 w(indices)26 b Fl(list)15 b(of)i(relativ)o(e)d(indices)i(of)g(en)o (tries)f(\(arra)o(y)h(of)h(in)o(tegers\))237 1606 y(Consider,)f(for)h (example,)c(the)j(follo)o(wing)g(fragmen)o(t)f(of)i(F)l(ortran)g(co)q(de)164 1721 y Fg(DOUBLE)23 b(PRECISION)g(A\(10,20\))164 1781 y(INTEGER)g(B,)i (C\(5,10\))164 1841 y(INTEGER)e(BH)164 1901 y(...)164 1961 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 2022 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(B,)h(1,)g(MPI_INT\))164 2082 y(CALL)f(MPI_ADD_VE)o(C)f(\(BH,)h(A\(1,3\),)f(11,)h(4,)h(2,)g (MPI_DOUBLE\))164 2142 y(CALL)f(MPI_ADD_IN)o(DEX)o(\(BH)o(,)f(C\(3,7\),)g (\(4,2,1\),)f(MPI_INT\))237 2256 y Fl(Then)15 b(the)g(bu\013er)g(asso)q (ciated)g(with)g(the)f(handle)h Fg(BH)f Fl(consists)h(of)g(the)g(sequence)f (of)164 2316 y(v)m(ariables)237 2377 y Fg(B,)25 b(A\(1,3\),)e(A\(2,3\),)g (A\(5,3\),)g(A\(6,3\),)g(A\(9,3\),)g(A\(10,3\),)g(A\(3,4\),)g(A\(4,4\),)164 2437 y(A\(7,4\),)g(A\(8,4\),)g(A\(1,5\),)g(C\(2,8\),)g(C\(5,7\),)g(C\(4,7\))p Fl(.)949 2599 y(16)p eop %%Page: 17 17 bop 237 307 a Fl(A)11 b(message)f(created)g(from)g(this)h(bu\013er)g(will)f (consist)h(of)g(a)g(sequence)f(of)h(one)g(in)o(teger,)164 367 y(follo)o(w)o(ed)k(b)o(y)h(elev)o(en)e(double)j(precision)e(reals,)h(follo)o (w)o(ed)f(b)o(y)h(three)f(in)o(tegers.)237 428 y(A)d(bu\013er)g(handle)g(can) g(b)q(e)g(used)g(for)g(comm)o(unic)o(ation,)e(ev)o(en)h(if)g(it)h(is)g(not)g (asso)q(ciated)164 488 y(with)20 b(an)o(y)f(v)m(ariables)h(\(i.e.,)f(ev)o(en) g(if)g(it)g(w)o(as)i(set)e(b)o(y)h(an)o(y)g Fg(MPI)p 1337 488 16 2 v 17 w(ADD)p 1432 488 V 17 w(xxx)f Fl(call\).)31 b(Suc)o(h)164 548 y(handle)21 b(is)g(asso)q(ciated)h(with)e(an)i(empt)o(y)d(bu\013er,)j (and)f(a)h(message)e(created)g(from)g(it)164 608 y(con)o(tains)c(no)h(data.) 164 778 y Fk(Discussion:)237 834 y Fj(The)h(main)g(mo)q(di\014cations)h (w.r.t.)25 b(the)18 b(prop)q(osal)f(of)g(Gropp)h(and)f(Lusk)h(is)g(measuring) 164 890 y(length)h(in)g(elemen)o(ts,)g(rather)f(than)g(b)o(ytes.)29 b(Seem)19 b(more)e(natural,)i(since)g(t)o(yp)q(e)g(is)f(kno)o(wn.)164 947 y(Also,)j(ob)s(ject)e(creation)h(uses)f(a)h(generic)g(function,)i(rather) d(than)g(a)g(function)i(sp)q(eci\014c)g(to)164 1003 y(bu\013er)c (descriptors.)28 b(As)17 b(result,)h(I)g(ga)o(v)o(e)f(up)h(on)f(the)h(size)g (parameter.)26 b(This)18 b(ma)o(y)f(not)g(b)q(e)164 1060 y(suc)o(h)k(a)g (loss:)32 b(it)21 b(is)g(not)g(clear)h(that)e(sp)q(ecifying)j(a)d(maxim)o(um) h(length)h(at)e(bu\013er)h(ob)s(ject)164 1116 y(creation)h(is)g(useful,)h (since)g(indices)g(of)e(arbitrary)g(size)i(ma)o(y)d(need)j(to)d(b)q(e)j (stored)d(in)j(the)164 1173 y(ob)s(ject.)164 1483 y Fi(1.6.1)55 b(Data)19 b(Con)n(v)n(ersion)164 1576 y Fl(The)d(t)o(yp)q(es)g(and)h(the)f (lo)q(cations)h(of)g(the)f(en)o(tries)f(used)h(to)h(create)e(a)i(message)f (is)g(solely)164 1636 y(determined)f(from)i(the)g(information)g(in)g(the)h (bu\013er)g(descriptor,)f(using)h(the)g(storage)164 1696 y(asso)q(ciation)g (rules)e(sp)q(eci\014ed)g(b)o(y)g(the)g(host)i(language)f(and)h(its)e(implem) o(en)n(tation;)e(the)164 1756 y(t)o(yp)q(e)22 b(and)h(the)g(lo)q(cations)g (of)g(these)g(en)o(tries)e(do)i(not)h(dep)q(end)e(on)i(the)e(declaration)164 1816 y(for)g(the)g(corresp)q(onding)g(v)m(ariables)g(in)g(the)f(calling)g (program.)38 b(It)21 b(is)h(not)g(required)164 1876 y(that)h(the)f(data)h(t)o (yp)q(es)f(sp)q(eci\014ed)f(in)h(a)h(bu\013er)g(descriptor)e(matc)o(h)g(the)h (data)h(t)o(yp)q(es)164 1937 y(of)h(the)f(corresp)q(onding)i(elemen)o(ts)20 b(in)j(the)h(host)g(program.)43 b(Ho)o(w)o(ev)o(er,)23 b(in)g(case)h(of)164 1997 y(mismatc)o(hes,)c(the)i(corresp)q(ondence)g(b)q(et)o(w)o(een)f(en)o (tries)g(in)h(the)g(host)g(program)g(and)164 2057 y(en)o(tries)10 b(in)g(a)h(message)f(created)h(with)f(the)h(bu\013er)g(descriptor)g(ma)o(y)e (b)q(e)i(implem)o(en)o(tati)o(on)164 2117 y(dep)q(enden)o(t.)34 b(No)20 b(data)h(con)o(v)o(ersion)f(o)q(ccurs)h(when)g(data)g(is)f(mo)o(v)o (ed)f(from)g(a)i(sender)164 2177 y(bu\013er)c(in)o(to)e(a)i(message.)237 2238 y(Consider)g(the)f(follo)o(wing)g(fragmen)o(t)f(of)h(F)l(ortran)h(co)q (de)164 2352 y Fg(REAL)24 b(A\(100\))164 2412 y(INTEGER)f(BH)164 2472 y(...)949 2599 y Fl(17)p eop %%Page: 18 18 bop 164 307 a Fg(CALL)24 b(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f (MPI_PERSIST)o(EN)o(T\))164 367 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(1\),)g (1,)h(MPI_REAL\))164 428 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(2\),)g(1,)h (MPI_INT\))164 488 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(3\),)g(1,)h (MPI_LOGICA)o(L\))164 548 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(4\),)g(1,)h (MPI_COMPLE)o(X\))164 608 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(6\),)g(1,)h (MPI_DOUBLE)o(\))164 668 y(CALL)f(MPI_ADD_BL)o(OCK)o(\(BH)o(,)f(A\(8\),)g(4,) i(MPI_CHAR\))237 782 y Fl(A)17 b(message)g(created)g(from)f(this)h(bu\013er)h (will)f(consist)g(of)h(a)g(sequence)e(con)o(taining)164 843 y(one)k(real,)f(one)h(in)o(teger,)f(one)h(logical,)f(one)h(complex,)e(one)i (double,)g(and)g(four)g(c)o(har-)164 903 y(acters.)31 b(No)19 b(data)i(con)o(v)o(ersion)d(o)q(ccurs)i(when)g(v)m(alues)f(are)h(copied)f (from)f(the)h(sender)164 963 y(bu\013er)e(to)g(the)f(message.)22 b(Th)o(us,)16 b(the)h(\014rst)g(en)o(try)e(in)i(the)f(message)g(is)g(a)h (real)g(n)o(um)o(b)q(er)164 1023 y(equal)g(to)h Fg(A\(1\))p Fl(;)f(the)h(second)g(en)o(try)f(is)g(an)i(in)o(teger)d(n)o(um)o(b)q(er)g (that)j(happ)q(ens)g(to)f(ha)o(v)o(e)164 1083 y(the)13 b(same)f(binary)g (represen)o(tation)h(as)g Fg(A\(2\))p Fl(;)g(the)f(third)h(en)o(try)f(is)h(a) g(logical)g(v)m(alue)f(that)164 1144 y(happ)q(ens)k(to)f(ha)o(v)o(e)g(the)f (same)g(binary)h(represen)o(tation)g(as)g Fg(A\(3\))p Fl(;)f(the)h(fourth)g (en)o(try)f(in)164 1204 y(a)g(complex)e(n)o(um)o(b)q(er)g(with)h(a)i(binary)e (represen)o(tation)g(iden)o(tical)f(to)j Fg(A\(4\),)23 b(A\(5\))p Fl(;)13 b(the)164 1264 y(\014fth)k(en)o(try)f(is)h(a)g(double)g(precision)f (v)m(alue)h(with)f(a)i(binary)e(represen)o(tation)h(iden)o(tical)164 1324 y(to)j Fg(A\(6\),)j(A\(7\))p Fl(;)c(and)h(if)f(w)o(ords)h(ha)o(v)o(e)f (four)g(b)o(ytes)g(then)g(the)g(last)h(four)g(en)o(tries)e(are)164 1384 y(b)o(ytes)f(that)g(mak)o(e)f(the)h(binary)g(represen)o(tation)g(of)h Fg(A\(8\))p Fl(,)d(in)i(the)h(b)o(yte)e(order)h(of)h(the)164 1445 y(executing)d(mac)o(hine.)237 1505 y(The)f(corresp)q(ondence)g(b)q(et)o (w)o(een)g(the)g(\014rst)g(sev)o(en)f(en)o(tries)g(of)i(the)e(arra)o(y)i Fg(A)e Fl(and)i(the)164 1565 y(\014rst)g(\014v)o(e)g(en)o(tries)f(of)h(the)g (message)g(created)f(from)g(this)h(bu\013er)h(is)f(determined)d(b)o(y)j(the) 164 1625 y(rules)k(of)h(F)l(ortran)g(77)g(on)g(storage)h(asso)q(ciation:)29 b(Eac)o(h)19 b(v)m(ariable)g(of)h(t)o(yp)q(e)f Fg(INTEGER)p Fl(,)164 1685 y Fg(REAL)p Fl(,)11 b(or)j Fg(LOGICAL)c Fl(o)q(ccup)o(y)j(one)g Fh(numeric)i(stor)n(age)g(unit)p Fl(;)f(a)f(v)m(ariable)g(of)h(t)o(yp)q(e)e Fg(DOUBLE)164 1746 y Fl(or)21 b Fg(COMPLEX)c Fl(o)q(ccup)o(y)j(t)o(w)o(o)g(n) o(umeric)e(storage)j(units.)33 b(Th)o(us,)21 b(the)f(same)f(corresp)q(on-)164 1806 y(dence)c(will)f(hold)i(for)g(an)o(y)g(implem)o(e)o(n)o(tation.)i(Ho)o (w)o(ev)o(er,)c(di\013eren)o(t)h(implem)o(en)n(tations)164 1866 y(ma)o(y)20 b(ha)o(v)o(e)g(di\013eren)o(t)g(binary)i(enco)q(dings)f(of)h (in)o(teger,)f(real)g(and)h(logical)e(v)m(alues,)i(so)164 1926 y(that)17 b(the)f(actual)g(v)m(alues)g(transferred)g(b)o(y)g(the)g(message)g (ma)o(y)e(di\013er.)237 1986 y(The)21 b(corresp)q(ondence)h(b)q(et)o(w)o(een) e(the)h(en)o(try)f Fg(A\(8\))g Fl(of)h(the)g(arra)o(y)l(,)h(and)g(the)f(last) 164 2046 y(four)j(c)o(haracter)g(en)o(tries)e(in)i(the)f(message)g(is)h (implem)o(en)o(tation)d(dep)q(enden)o(t,)k(since)164 2107 y(the)19 b(F)l(ortran)i(language)f(do)q(es)h(not)f(sp)q(ecify)f(a)h(corresp)q(ondence) g(b)q(et)o(w)o(een)e(c)o(haracter)164 2167 y(storage)j(units)g(and)g(n)o (umeric)d(storage)j(units)f(\(an)h(arra)o(y)g(of)g(c)o(haracters)f(cannot)h (b)q(e)164 2227 y(equiv)m(alenced)16 b(with)i(an)g(arra)o(y)g(of)g(in)o (tegers\).)26 b(Di\013eren)o(t)17 b(results)h(ma)o(y)e(o)q(ccur)i(in)g(big-) 164 2287 y(endians)e(or)h(small-endians)e(mac)o(hines,)f(or)i(in)g(32)h(bit)f (or)h(64)g(bit)f(mac)o(hines.)237 2347 y(The)i(same)e(holds,)i(symmetri)o (call)o(y)l(,)c(when)k(a)g(message)e(is)i(receiv)o(ed.)k(En)o(tries)17 b(are)164 2408 y(mo)o(v)o(ed)i(from)g(the)i(message)f(in)o(to)h(the)f(receiv) o(er)f(memory)f(according)j(to)g(the)g(infor-)164 2468 y(mation)h(pro)o (vided)g(b)o(y)g(the)g(bu\013er)i(descriptor,)f(with)g(no)g(regard)g(to)g (the)g(w)o(a)o(y)f(the)949 2599 y(18)p eop %%Page: 19 19 bop 164 307 a Fl(corresp)q(onding)17 b(v)m(ariables)f(are)h(declared)e(in)h (the)g(receiving)f(program.)237 367 y(When)c(data)h(is)f(mo)o(v)o(ed)e(in)h (a)i(homogeneous)f(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)i(no)q(des)i(ha)o (ving)164 428 y(the)21 b(same)f(arc)o(hitecture,)g(then)h(no)g(data)h(con)o (v)o(ersion)e(o)q(ccur)h(at)h(an)o(y)f(p)q(oin)o(t)g(during)164 488 y(data)16 b(transfer.)21 b(Assume,)13 b(in)i(the)g(previous)g(example,)d (that)k(an)f(iden)o(tically)e(declared)164 548 y(bu\013er)j(descriptor)g(is)f (used)h(to)g(receiv)o(e)e(data)i(in)o(to)g(an)g(iden)o(tically)e(declared)h (arra)o(y)h(at)164 608 y(the)g(receiving)e(pro)q(cess.)21 b(Then,)16 b(when)g(these)f(t)o(w)o(o)h(no)q(des)g(comm)o(unicate,)c(the)k(v)m(alues)164 668 y(in)g(the)h(\014rst)g(eigh)o(t)f(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(of)h(the)g(sender)f(will)g(b)q(e)h(copied)f(in)o(to)g(the)h (\014rst)164 729 y(eigh)o(t)e(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(at)h(the)f(receiv)o(er)e(\(assuming)i(that)h(reals)f(o)q(ccup)o(y)h (the)f(same)164 789 y(storage)21 b(as)g(four)f(b)o(ytes\).)33 b(In)20 b(particular,)g(in)g(a)g(homogeneous)h(en)o(vironmen)o(t,)c(it)j(is) 164 849 y(p)q(ossible)c(to)h(comm)o(unic)o(ate)d(using)i(only)g(c)o(haracter) g(t)o(yp)q(ed)g(messages.)237 909 y(When)22 b(data)g(is)f(mo)o(v)o(ed)e(in)o (to)i(a)h(heterogeneous)g(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)h(no)q(des)164 969 y(ha)o(ving)14 b(distinct)f(arc)o(hitectures,)g(data)i(con)o(v)o(ersion)e (ma)o(y)g(o)q(ccur)h(during)g(the)g(transfer:)164 1029 y(Eac)o(h)21 b(en)o(try)e(is)i(con)o(v)o(erted)e(from)h(the)g(data)h(represen)o(tation)f (used)h(on)g(the)g(sending)164 1090 y(no)q(de)c(to)f(the)g(data)i(represen)o (tation)d(used)i(in)e(the)h(receiving)f(no)q(de.)237 1150 y(Consider)i(the)f (follo)o(wing)g(fragmen)o(t)f(of)h(F)l(ortran)h(co)q(de.)164 1264 y Fg(REAL)24 b(X,)h(Y)164 1324 y(CHARACTER)d(\(4\))j(Z)164 1384 y(INTEGER)e(BH)164 1445 y(...)164 1505 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f (MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 1565 y(CALL)i(MPI_ADD_BL)o(OCK)e (\(BH,)i(X,)h(1,)g(MPI_REAL\))164 1625 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i (Y,)h(4,)g(MPI_CHAR\))164 1685 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(Z,)h(4,)g (MPI_CHAR\))237 1800 y Fl(Assume)11 b(that)i(the)g(same)e(arra)o(ys)i Fg(X,)25 b(Y,)g(Z)12 b Fl(and)h(bu\013er)g(handle)g Fg(BH)f Fl(are)g(created)h(at)164 1860 y(t)o(w)o(o)g(pro)q(cesses)h(A)e(and)i(B;)e (this)h(handle)g(is)g(used)g(b)o(y)g(A)f(to)i(create)e(and)i(send)f(a)h (message)164 1920 y(for)e(B,)f(b)o(y)h(B)f(to)i(receiv)o(e)c(this)j(message.) 19 b(F)l(urther)12 b(assume)f(that)i(b)q(oth)g(A)e(and)i(B)e(run)h(on)164 1980 y(distinct)g(no)q(des,)j(with)e(p)q(ossibly)g(di\013eren)o(t)g(32)h(bit) f(arc)o(hitectures.)18 b(Then)c Fg(X)f Fl(at)g(pro)q(cess)164 2040 y(B)19 b(is)g(assigned)h(the)g(v)m(alue)f(of)g Fg(X)h Fl(at)f(pro)q(cess)h(A)f(\(up)h(to)g(rounding)g(errors)g(that)f(ma)o(y)164 2100 y(o)q(ccur)12 b(during)g(con)o(v)o(ersion\);)f Fg(Z)h Fl(on)g(pro)q(cess)g(B)f(is)h(assigned)g(the)f(c)o(haracter)g(string)h(v)m (alue)164 2161 y(of)18 b Fg(Z)f Fl(on)h(pro)q(cess)g(A;)f(if)f(b)q(oth)j(no)q (des)f(use)g(ASCI)q(I)f(enco)q(ding,)g(then)h(no)g(con)o(v)o(ersion)e(is)164 2221 y(required)e(for)h(the)g(c)o(haracters.)21 b(On)15 b(the)g(other)h (hand,)f(v)m(ariable)g Fg(Y)g Fl(on)h(pro)q(cess)g(B)e(ma)o(y)164 2281 y(b)q(e)19 b(allo)q(cated)g(a)g(v)m(alue)f(that)h(di\013ers)g(from)f (the)g(v)m(alue)h(of)g(v)m(ariable)f Fg(Y)g Fl(on)h(pro)q(cess)h(A.)164 2341 y(This)12 b(ma)o(y)f(o)q(ccur)h(if)g(the)g(t)o(w)o(o)g(no)q(des)h(use)f (a)h(di\013eren)o(t)e(b)o(yte)g(sequence)g(\(little-endian)g(vs)164 2401 y(big-endian\),)i(or)g(use)f(a)h(di\013eren)o(t)e(binary)i(represen)o (tation)e(for)i(reals)f(\(IEEE)h(vs)f(HEX\).)164 2462 y(Th)o(us,)18 b(in)f(order)h(to)g(ensure)g(correct)f(execution)g(in)g(a)h(heterogeneous)g (en)o(vironmen)o(t,)949 2599 y(19)p eop %%Page: 20 20 bop 164 307 a Fl(it)16 b(is)g(imp)q(ortan)o(t)g(that)g(the)h(t)o(yp)q(es)f (of)g(v)m(alues)h(in)f(a)h(message)e(matc)o(h)g(the)h(t)o(yp)q(es)g(of)h(the) 164 367 y(corresp)q(onding)g(v)m(alues)f(in)g(the)g(sending)h(and)g(in)f(the) g(receiving)e(program.)164 512 y Fm(1.7)70 b(Receiv)n(e)20 b(Criteria)164 604 y Fl(The)15 b(selection)f(of)h(a)g(message)f(b)o(y)h(a)g (receiv)o(e)e(op)q(eration)i(is)g(done)g(uniquely)f(according)164 664 y(to)23 b(the)g(v)m(alue)g(of)g(the)g(message)f(en)o(v)o(elop)q(e.)40 b(The)22 b(receiv)o(e)f(op)q(eration)j(sp)q(eci\014es)e(an)164 725 y Fi(en)n(v)n(elop)r(e)k(pattern)p Fl(;)g(a)d(message)f(can)h(b)q(e)h (receiv)o(ed)c(b)o(y)j(that)g(receiv)o(e)e(op)q(eration)164 785 y(only)c(if)f(its)h(en)o(v)o(elop)q(e)f(matc)o(hes)f(that)j(pattern.)24 b(A)17 b(pattern)g(sp)q(eci\014es)g(v)m(alues)g(for)g(the)164 845 y Fg(source)p Fl(,)h Fg(tag)h Fl(and)h Fg(context)d Fl(\014elds)j(of)g (the)g(message)f(en)o(v)o(elop)q(e.)30 b(In)20 b(addition,)g(the)164 905 y(v)m(alue)15 b(for)h(the)f Fg(dest)e Fl(\014eld)i(is)g(set,)g(implicitl) o(y)l(,)d(to)k(b)q(e)f(equal)g(to)h(the)f(receiving)e(pro)q(cess)164 965 y(id.)38 b(The)22 b(receiv)o(er)e(ma)o(y)g(sp)q(ecify)i(a)g Fi(DONTCARE)g Fl(v)m(alue)g(for)g Fg(source)p Fl(,)f(or)h Fg(tag)p Fl(,)164 1025 y(indicating)e(that)i(an)o(y)e(source)h(and/or)h(tag)g(are)f (acceptable.)34 b(It)21 b(cannot)g(sp)q(ecify)g(a)164 1086 y(DONTCARE)d(v)m(alue)h(for)f Fg(context)e Fl(or)j Fg(dest)p Fl(.)26 b(Th)o(us,)19 b(a)g(message)f(can)h(b)q(e)f(receiv)o(ed)164 1146 y(b)o(y)k(a)g(receiv)o(e)e(op)q(eration)j(only)e(if)h(it)f(is)h (addressed)h(to)f(the)g(receiving)e(task,)k(has)f(a)164 1206 y(matc)o(hing)18 b(con)o(text,)h(has)h(matc)o(hing)e(source)h(unless)h (source=DONTCARE)f(in)g(the)164 1266 y(pattern,)d(and)h(has)g(a)f(matc)o (hing)f(tag)i(unless)f(tag=DONTCARE)h(in)f(the)g(pattern.)237 1326 y(The)k(length)f(of)h(the)g(receiv)o(ed)d(message)i(m)o(ust)f(b)q(e)i (less)g(or)g(equal)f(the)g(length)h(of)164 1387 y(the)d(receiv)o(e)f (bu\013er.)26 b(I.e.,)16 b(all)h(incoming)f(data)i(m)o(ust)f(\014t,)g (without)h(truncation,)g(in)o(to)164 1447 y(the)f(receiv)o(e)f(bu\013er.)25 b(It)17 b(is)h(erroneous)g(to)g(receiv)o(e)d(a)j(message)f(whic)o(h)g(length) g(exceed)164 1507 y(the)e(receiv)o(e)d(bu\013er,)j(and)g(the)g(outcome)e(of)j (program)e(where)h(this)f(o)q(ccurs)h(is)g(undeter-)164 1567 y(mined.)164 1712 y Fm(1.8)70 b(Comm)n(unicati)o(on)21 b(Mo)r(de)164 1804 y Fl(A)16 b(sending)g(op)q(eration)h(can)g(o)q(ccur)f(in)g(one)g(of)h(t) o(w)o(o)f(mo)q(des:)164 1906 y Fi(REGULAR)24 b Fl(The)e(send)g(ma)o(y)f (start)i(whether)f(or)g(not)h(a)g(matc)o(hing)d(receiv)o(e)g(has)286 1966 y(b)q(een)c(p)q(osted.)164 2068 y Fi(READ)n(Y)25 b Fl(The)16 b(send)h(ma)o(y)d(start)j(only)f(if)g(a)g(matc)o(hing)f(receiv)o(e)f(has)j(b) q(een)f(p)q(osted.)237 2169 y(A)21 b Fi(ready)k(send)c Fl(can)h(start)g(only) g(if)f(a)h(matc)o(hing)e(receiv)o(e)g(is)h(already)h(p)q(osted;)164 2229 y(otherwise)17 b(the)g(op)q(eration)i(is)e(erroneous)h(and)g(its)f (outcome)g(is)g(unde\014ned.)25 b(In)17 b(some)164 2290 y(systems,)12 b(this)g(will)g(allo)o(w)h(to)g(optimize)d(comm)o(unic)o(ation)h(and)i(a)o(v) o(oid)f(a)h(hand-shaking)164 2350 y(op)q(eration)k(that)g(is)f(otherwise)g (required.)949 2599 y(20)p eop %%Page: 21 21 bop 164 307 a Fk(Discussion:)28 b Fj(I)19 b(deleted)i(the)e(symmetric)g (ready)g(receiv)o(e.)32 b(Will)21 b(reviv)o(e)f(it)f(if)g(there)h(is)f(a)164 367 y(requiremen)o(t)d(for)e(it.)164 630 y Fm(1.9)70 b(Comm)n(unicati)o(on)21 b(Handles)164 722 y Fl(An)13 b(opaque)i(comm)o(uni)o(cation)c(ob)s(ject)i (iden)o(ti\014es)g(v)m(arious)h(prop)q(erties)g(of)g(a)g(comm)o(uni-)164 782 y(cation)j(op)q(eration,)g(suc)o(h)g(as)g(the)g(bu\013er)g(descriptor)g (that)g(is)g(asso)q(ciated)g(with)g(it,)f(its)164 842 y(con)o(text,)e(the)g (tag)h(and)g(destination)f(parameters)g(to)h(b)q(e)f(used)h(for)g(a)g(send,)f (or)h(the)f(tag)164 903 y(and)f(source)g(parameters)f(to)h(b)q(e)g(used)g (for)h(a)f(receiv)o(e.)18 b(In)12 b(addition,)h(this)g(ob)s(ject)f(stores)164 963 y(information)17 b(ab)q(out)h(the)g(status)g(of)g(the)g(last)f(comm)o (unication)e(op)q(eration)j(that)g(w)o(as)164 1023 y(p)q(erformed)f(with)g (this)h(ob)s(ject.)26 b(This)18 b(ob)s(ject)f(is)h(accessed)f(using)i(a)f (comm)o(unicati)o(on)164 1083 y(handle.)237 1143 y(One)i(can)h(consider)f (comm)o(uni)o(cation)e(op)q(erations)j(to)g(consist)g(of)f(the)g(follo)o (wing)164 1204 y(sub)q(op)q(erations:)164 1299 y Fi(INIT\(op)r(eration,)d (params,)i(handle\))25 b Fl(Pro)q(cess)15 b(pro)o(vides)f(all)f(relev)m(an)o (t)g(parame-)286 1359 y(ters)18 b(for)g(its)g(participation)g(in)g(the)f (comm)o(unication)e(op)q(eration)k(\(t)o(yp)q(e)f(of)g(op-)286 1419 y(eration,)g(data)h(bu\013er,)f(tag,)h(participan)o(ts,)f(etc.\).)25 b(A)18 b(handle)g(is)g(created)f(that)286 1479 y(iden)o(ti\014es)e(the)h(op)q (eration.)164 1576 y Fi(ST)-5 b(AR)g(T\(handle\))26 b Fl(The)16 b(comm)o(unic)o(ation)e(op)q(eration)j(is)f(started)164 1673 y Fi(COMPLETE\(handle\))25 b Fl(The)16 b(comm)o(unication)d(op)q(eration)k (is)f(completed.)164 1770 y Fi(FREE\(handle\))24 b Fl(The)17 b(handle,)e(and)i(asso)q(ciated)g(resources)g(are)f(freed.)164 1866 y(Correct)g(in)o(v)o(o)q(cation)g(of)g(these)g(sub)q(op)q(erations)i(is) f(a)f(sequence)f(of)i(the)f(form)520 1959 y Fi(INIT)i Fl(\()p Fi(ST)-5 b(AR)g(T)20 b(COMPLETE)p Fl(\))1224 1939 y Fb(\003)1263 1959 y Fi(FREE)p Fd(:)164 2053 y Fl(I.e.,)12 b(a)h(handle)g(needs)g(b)q(e)g (created)g(b)q(efore)g(comm)o(unic)o(ation)e(o)q(ccurs;)j(it)f(can)g(b)q(e)g (reused)164 2113 y(only)g(after)g(the)g(previous)g(use)g(has)h(completed;)d (and)j(it)f(needs)g(to)g(b)q(e)h(freed)e(ev)o(en)o(tually)164 2173 y(\(of)i(course,)f(one)g(can)h(assume)e(that)i(all)f(handles)g(are)h (freed)e(at)i(program)f(termination,)164 2233 y(b)o(y)j(default\).)237 2293 y(The)k(ab)q(o)o(v)o(e)f(scenario)h(p)q(ertains)g(to)g Fh(p)n(ersistent)g Fl(handles.)31 b(One)20 b(can)f(also)i(create)164 2354 y Fh(ephemer)n(al)15 b Fl(handles.)21 b(Suc)o(h)15 b(handle)f(p)q (ersists)h(only)g(un)o(til)f(the)g(comm)o(unication)e(op)q(er-)164 2414 y(ation)h(is)g(completed,)e(at)i(whic)o(h)f(p)q(oin)o(t)h(it)f(is)h (destro)o(y)o(ed.)19 b(Th)o(us,)13 b(correct)f(in)o(v)o(o)q(cation)h(of)164 2474 y(sub)q(op)q(erations)18 b(with)e(an)h(ephemeral)d(handle)i(is)g Fi(INIT)i(ST)-5 b(AR)g(T)20 b(COMPLETE)p Fl(.)949 2599 y(21)p eop %%Page: 22 22 bop 237 307 a Fl(A)21 b(user)g(ma)o(y)e(directly)h(in)o(v)o(ok)o(es)f(these)i (sub)q(op)q(erations.)37 b(This)22 b(w)o(ould)f(allo)o(w)g(to)164 367 y(amortize)16 b(the)i(o)o(v)o(erhead)f(of)h(setting)g(up)g(a)g(comm)o (unication)d(o)o(v)o(er)i(man)o(y)f(successiv)o(e)164 428 y(uses)23 b(of)h(the)e(same)g(handle,)i(and)g(allo)o(ws)f(to)g(o)o(v)o(erlap)g(comm)o (uni)o(cation)d(and)k(com-)164 488 y(putation.)29 b(Simpler)16 b(comm)o(unication)g(op)q(erations)k(com)o(bine)c(sev)o(eral)i(of)h(these)f (sub-)164 548 y(op)q(erations)24 b(in)o(to)e(one)g(op)q(eration,)j(th)o(us)d (simplifying)e(the)i(use)g(of)h(comm)o(unicati)o(on)164 608 y(primitiv)o(e)o(s.)f(Th)o(us,)c(one)f(only)h(needs)f(to)h(sp)q(ecify)e (precisely)g(the)h(seman)o(tics)f(of)h(these)164 668 y(sub)q(op)q(erations)i (in)e(order)g(to)h(sp)q(ecify)e(the)h(seman)o(tics)f(of)h(MPI)g(p)q(oin)o(t)g (to)h(p)q(oin)o(t)f(com-)164 729 y(m)o(unication)d(op)q(erations.)237 789 y(W)l(e)h(sa)o(y)f(that)h(a)g(comm)o(unication)d(op)q(eration)j(\(send)g (or)g(receiv)o(e\))e(is)h Fi(p)r(osted)g Fl(once)164 849 y(a)23 b Fi(start)g Fl(sub)q(op)q(eration)i(w)o(as)f(in)o(v)o(ok)o(ed;)g(the)f(op)q (eration)h(is)f Fi(completed)f Fl(once)h(the)164 909 y Fi(complete)17 b Fl(sub)q(op)q(eration)j(completes.)k(A)17 b(send)h(and)h(a)f(receiv)o(e)e (op)q(eration)j Fi(matc)n(h)164 969 y Fl(if)d(the)h(receiv)o(e)d(pattern)j (sp)q(eci\014ed)g(b)o(y)f(the)h(receiv)o(e)d(matc)o(hes)i(the)g(message)g(en) o(v)o(elop)q(e)164 1029 y(created)g(b)o(y)f(the)h(send.)164 1159 y Fi(1.9.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)164 1252 y Fl(An)g(ob)s(ject)g(for)g(a)h(send)f(op)q(eration)h(is)f(created)g(b)o (y)g(a)g(call)g(to)g Fi(MPI)p 1452 1252 17 2 v 21 w(INIT)p 1598 1252 V 20 w(SEND)p Fl(.)164 1312 y(A)g(call)g(to)g Fi(MPI)p 487 1312 V 21 w(INIT)p 633 1312 V 20 w(RECV)h Fl(is)f(similarly)d(used)k(for) g(creating)f(an)h(ob)s(ject)f(for)h(a)164 1372 y(receiv)o(e)8 b(op)q(eration.)21 b(The)11 b(creation)f(of)h(a)g(comm)o(unication)d(ob)s (ject)i(is)h(a)g(lo)q(cal)g(op)q(eration)164 1432 y(that)17 b(need)e(not)i(in)o(v)o(olv)o(e)d(comm)o(unicati)o(on)g(with)i(a)h(remote)e (pro)q(cess.)164 1553 y Fi(MPI)p 279 1553 V 20 w(INIT)p 424 1553 V 20 w(SEND)g(\(handle,)g(bu\013er)p 972 1553 V 20 w(handle,)h(dest,)f (tag,)g(con)n(text,)f(mo)r(de,)164 1613 y(p)r(ersistence\))237 1733 y Fl(Creates)j(a)f(send)h(comm)o(uni)o(cation)d(ob)s(ject.)20 b(P)o(arameters)15 b(are)164 1835 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i(not)f(b)q(e)f(asso)q(ciated)i(with) 286 1895 y(an)o(y)16 b(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1997 y Fi(IN)i(bu\013er)p 394 1997 V 20 w(handle)25 b Fl(handle)16 b(to)h(send)f(bu\013er)h(descriptor)164 2099 y Fi(IN)h(dest)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i(destination)f(\(in)o(teger\))164 2200 y Fi(IN)i(tag)25 b Fl(user)16 b(tag)h(for)g(messages)e(sen)o(t)h(with)g (this)g(handle)h(\(in)o(teger\))164 2302 y Fi(IN)h(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages)e(sen)o(t)h(with)g(this)h (handle)164 2404 y Fi(IN)h(mo)r(de)24 b Fl(send)11 b(mo)q(de)f(\(state)h(t)o (yp)q(e,)g(with)f(t)o(w)o(o)h(v)m(alues:)19 b Fg(MPI)p 1324 2404 16 2 v 17 w(REGULAR)8 b Fl(and)k Fg(MPI)p 1699 2404 V 17 w(READY)p Fl(\))949 2599 y(22)p eop %%Page: 23 23 bop 164 307 a Fi(IN)18 b(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g (\(state)h(t)o(yp)q(e,)g(with)g(t)o(w)o(o)g(v)m(alues:)18 b Fg(MPI)p 1623 307 16 2 v 18 w(PERSISTEN)o(T)286 367 y Fl(and)f Fg(MPI)p 462 367 V 17 w(EPHEMERAL)p Fl(\))164 529 y Fi(MPI)p 279 529 17 2 v 20 w(INIT)p 424 529 V 20 w(RECV)c(\(handle,)h(bu\013er)p 975 529 V 19 w(handle,)g(source,)f(tag,)h(con)n(text,)e(p)r(er-)164 589 y(sistence\))237 710 y Fl(Create)k(a)h(receiv)o(e)d(handle.)21 b(P)o(arameters)15 b(are)164 812 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i(not)f(b)q(e)f(asso)q(ciated)i(with) 286 872 y(an)o(y)16 b(ob)s(ject)g(b)q(efore)g(the)g(call.)164 973 y Fi(IN)i(bu\013er)p 394 973 V 20 w(handle)25 b Fl(handle)16 b(to)h(receiv)o(e)d(bu\013er)i(descriptor.)164 1075 y Fi(IN)i(source)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i(source,)f(or)g(DONTCARE)g(\(in)o (teger\).)164 1177 y Fi(IN)i(tag)25 b Fl(user)d(tag)h(for)f(messages)g (receiv)o(ed)e(with)i(this)g(handle,)h(or)f(DONTCARE)286 1237 y(\(in)o(teger\).)164 1339 y Fi(IN)c(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages)e(receiv)o(ed)f(with)j(this)f(handle.)164 1440 y Fi(IN)i(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g(\(state)h (t)o(yp)q(e,)g(with)g(t)o(w)o(o)g(v)m(alues:)18 b Fg(MPI)p 1623 1440 16 2 v 18 w(PERSISTEN)o(T)286 1501 y Fl(and)f Fg(MPI)p 462 1501 V 17 w(EPHEMERAL)p Fl(\))237 1602 y(See)f(Section)g(1.5.2)g(for)h(a) f(discussion)h(of)f(source,)g(tag)h(and)g(con)o(text.)164 1772 y Fk(Discussion:)48 b Fj(I)18 b(ha)o(v)o(e)f(not)f(included)k(prop)q(osals)e (for)e(partially)i(sp)q(eci\014ed)i(message)d(han-)164 1828 y(dles,)f(that)e(some)h(p)q(eoples)i(seem)e(to)g(desire.)237 1885 y(I)h(ha)o(v)o(e)e(merged)i(all)g(handle)g(setup)g(in)o(to)f(one)g (call.)164 2195 y Fi(1.9.2)55 b(Comm)n(unication)21 b(Start)164 2348 y(MPI)p 279 2348 17 2 v 20 w(ST)-5 b(AR)g(T\(handle\))164 2449 y(IN)18 b(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)949 2599 y(23)p eop %%Page: 24 24 bop 237 307 a Fl(The)17 b Fg(MPI)p 419 307 16 2 v 17 w(START)f Fl(function)g(starts)i(the)f(execution)e(of)j(a)f(comm)o(unic)o(ation)e(op)q (era-)164 367 y(tion)k(\(send)h(or)f(receiv)o(e\).)28 b(A)19 b(sender)g(should)h(not)g(up)q(date)g(the)f(send)h(bu\013er)f(after)h(a)164 428 y(send)e(op)q(eration)g(has)h(started)f(and)g(un)o(til)f(it)g(is)g (completed.)24 b(A)17 b(receiv)o(er)e(should)j(not)164 488 y(access)f(the)f(receiv)o(e)f(bu\013er)i(after)f(a)h(receiv)o(e)e(op)q (eration)i(w)o(as)h(started)f(and)g(un)o(til)f(it)g(is)164 548 y(completed.)21 b(A)16 b(program)h(that)g(do)q(es)h(not)f(satisfy)g(this) g(condition)f(is)h(erroneous)g(and)164 608 y(its)f(outcome)f(is)h (undetermined.)164 738 y Fi(1.9.3)55 b(Comm)n(unication)21 b(Completion)164 891 y(MPI)p 279 891 17 2 v 20 w(W)-6 b(AIT)19 b(\()f(handle,)h(return)p 864 891 V 20 w(status)p 1030 891 V 20 w(handle\))164 992 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)164 1094 y Fi(OUT)19 b(return)p 466 1094 V 19 w(handle)26 b Fl(handle)16 b(that)g(is)g(asso)q(ciated)i(with)e(return)g(status)h(ob)s (ject.)237 1196 y(A)23 b(call)f(to)h Fi(MPI)p 574 1196 V 21 w(W)-6 b(AIT)23 b Fl(returns)g(when)g(the)g(send)g(op)q(eration)h(iden)o (ti\014ed)d(b)o(y)164 1256 y Fg(handle)16 b Fl(is)i(complete.)24 b(The)18 b(completion)e(of)i(a)g(send)h(op)q(eration)f(indicates)g(that)g (the)164 1316 y(sender)h(is)g(no)o(w)h(free)e(to)i(up)q(date)f(the)g(lo)q (cations)h(in)f(the)g(send)g(bu\013er,)h(or)g(an)o(y)f(other)164 1376 y(lo)q(cation)e(that)f(can)h(b)q(e)f(referenced)f(b)o(y)h(the)g(send)g (op)q(eration.)23 b(Ho)o(w)o(ev)o(er,)14 b(it)h(do)q(es)i(not)164 1436 y(indicate)11 b(that)i(the)f(message)g(has)h(b)q(een)f(receiv)o(ed;)f (rather)h(it)g(ma)o(y)f(ha)o(v)o(e)h(b)q(een)g(bu\013ered)164 1497 y(b)o(y)k(the)g(comm)o(uni)o(cation)e(subsystem.)237 1557 y(The)k(completion)e(of)h(a)i(receiv)o(e)c(op)q(eration)j(indicates)f(that)h (the)g(receiv)o(er)d(is)j(no)o(w)164 1617 y(free)e(to)i(access)f(the)g(lo)q (cations)h(in)f(the)g(receiv)o(e)e(bu\013er,)i(whic)o(h)g(con)o(tain)g(the)g (receiv)o(ed)164 1677 y(message,)f(or)i(an)o(y)g(other)f(lo)q(cation)h(that)g (can)f(b)q(e)h(referenced)e(b)o(y)h(the)g(receiv)o(e)e(op)q(era-)164 1737 y(tion.)21 b(It)16 b(do)q(es)h(not)g(indicate)e(that)i(the)f(matc)o (hing)e(send)j(op)q(eration)g(has)g(completed.)237 1798 y(The)g(call)e (returns)i(a)g(handle)f(to)h(an)g(opaque)g(ob)s(ject)f(that)h(con)o(tains)g (information)164 1858 y(on)g(the)f(completed)e(op)q(eration)j({)g(the)f Fi(return)i(status)e Fl(ob)s(ject.)164 1978 y Fi(MPI)p 279 1978 V 20 w(ST)-5 b(A)g(TUS)20 b(\(handle,)f(\015ag,)g(return)p 1029 1978 V 19 w(handle\))164 2080 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation) 14 b(handle)164 2182 y Fi(OUT)19 b(\015ag)25 b Fl(logical)164 2283 y Fi(OUT)19 b(return)p 466 2283 V 19 w(status)p 631 2283 V 21 w(handle)25 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)h(with)e(return)h (status)286 2343 y(ob)s(ject.)949 2599 y(24)p eop %%Page: 25 25 bop 237 307 a Fl(A)18 b(call)g(to)h Fi(MPI)p 561 307 17 2 v 20 w(ST)-5 b(A)g(TUS)20 b Fl(returns)f Fg(flag=true)c Fl(if)j(the)g(op)q (eration)h(iden)o(ti\014ed)164 367 y(b)o(y)d Fg(handle)f Fl(is)i(complete,)e (In)h(suc)o(h)h(case,)g(the)g(return)f(handle)h(p)q(oin)o(ts)h(to)f(an)h (opaque)164 428 y(ob)s(ject)h(that)h(con)o(tains)g(information)e(on)j(the)e (completed)e(information.)31 b(It)19 b(returns)164 488 y Fg(flag=false)o Fl(,)12 b(otherwise.)21 b(In)14 b(suc)o(h)h(case,)f(the)h(v)m(alue)g(of)g (the)g(return)f(handle)h(is)g(unde-)164 548 y(\014ned.)237 608 y(Implem)o(en)o(tation)e(notes:)237 668 y(A)g(call)f(to)h Fg(MPI)p 510 668 16 2 v 17 w(WAIT)f Fl(blo)q(c)o(ks)g(only)h(the)f(executing) g(thread.)20 b(If)13 b(the)f(executing)g(pro-)164 729 y(cess)h(is)f(m)o (ultithreaded,)f(then)i(other)g(threads)g(within)g(the)f(pro)q(cess)i(can)f (b)q(e)g(sc)o(heduled)164 789 y(for)j(execution.)237 849 y(The)h(use)h(of)f (a)h(blo)q(c)o(king)f(receiv)o(e)e(op)q(eration)j(\()p Fg(MPI)p 1197 849 V 17 w(WAIT)p Fl(\))e(allo)o(ws)h(the)g(op)q(erating)164 909 y(system)k(to)h(desc)o(hedule)f(the)h(blo)q(c)o(k)o(ed)f(thread)i(and)f (sc)o(hedule)f(another)i(thread)g(for)164 969 y(execution,)c(if)h(suc)o(h)g (is)g(a)o(v)m(ailable.)32 b(The)20 b(use)g(of)g(a)h(non)o(blo)q(c)o(king)e (receiv)o(e)f(op)q(eration)164 1029 y(\()p Fg(MPI)p 264 1029 V 17 w(STATUS)p Fl(\))11 b(allo)o(ws)i(the)f(user)h(to)g(sc)o(hedule)f (alternativ)o(e)g(activities)f(within)h(a)h(single)164 1090 y(thread)j(of)h(execution.)237 1150 y(The)h(in)o(tended)e(implem)o(en)o (tation)f(of)j Fg(MPI)p 1027 1150 V 17 w(STATUS)d Fl(is)j(for)g(that)g(op)q (eration)g(to)g(re-)164 1210 y(turn)e(as)g(so)q(on)h(as)g(p)q(ossible.)k(Ho)o (w)o(ev)o(er,)14 b(if)h(rep)q(eatedly)g(called)g(for)h(an)g(op)q(eration)g (that)164 1270 y(is)g(enabled,)f(it)h(m)o(ust)f(ev)o(en)o(tually)f(succeed.) 237 1330 y(The)g(return)f(status)h(ob)s(ject)f(for)g(a)h(send)g(op)q(eration) g(carries)f(no)h(information.)19 b(The)164 1391 y(return)13 b(status)h(ob)s(ject)f(for)h(a)f(receiv)o(e)e(op)q(eration)j(carries)f (information)f(on)i(the)f(source,)164 1451 y(tag)h(and)h(length)e(of)h(the)g (receiv)o(ed)d(message.)20 b(These)14 b(\014elds)f(are)h(required)e(b)q (ecause)i(the)164 1511 y(receiv)o(e)h(op)q(eration)j(ma)o(y)e(ha)o(v)o(e)h (sp)q(eci\014ed)g Fg(DONTCARE)e Fl(in)i(either)f(source)i(or)g(tag)g (\014eld,)164 1571 y(and)f(the)f(message)f(ma)o(y)g(ha)o(v)o(e)g(b)q(een)i (shorter)f(than)h(the)f(receiv)o(e)e(bu\013er.)164 1692 y Fi(MPI)p 279 1692 17 2 v 20 w(RETURN)p 546 1692 V 20 w(ST)-5 b(A)g(T\()19 b(handle,)g(len,)g(source,)f(tag\))164 1793 y(IN)g(handle)26 b Fl(handle)16 b(to)g(return)g(status)h(ob)s(ject)164 1895 y Fi(OUT)i(len)24 b Fl(di\013erence)15 b(b)q(et)o(w)o(een)h(length)g(of)g (receiv)o(e)e(bu\013er)i(and)h(length)f(of)g(receiv)o(ed)286 1955 y(message,)24 b(in)f(b)o(ytes.)42 b(Th)o(us,)25 b(the)e(v)m(alue)g (returned)g(is)g(zero)g(if)g(the)g(receiv)o(ed)286 2015 y(message)11 b(matc)o(hes)g(the)g(the)h(receiv)o(e)e(bu\013er,)j(p)q(ositiv)o(e)e(if)h(it) f(is)h(shorter)g(\(in)o(teger\).)164 2117 y Fi(OUT)19 b(source)24 b Fl(rank)16 b(of)h(message)e(sender)h(in)g(message)g(con)o(text)f(\(in)o (teger\).)164 2219 y Fi(OUT)k(tag)24 b Fl(tag)17 b(of)g(receiv)o(ed)d (message)h(\(in)o(teger\).)164 2430 y Fk(Discussion:)949 2599 y Fl(25)p eop %%Page: 26 26 bop 237 307 a Fj(I)18 b(put)g(the)g(di\013erence)h(b)q(et)o(w)o(een)f (message)f(bu\013er)h(and)g(message)f(length)h(as)g(the)g(v)m(alue)164 364 y(returned,)e(rather)f(than)g(length)h(of)g(receiv)o(ed)g(message,)f(so)h (that)e(it)i(migh)o(t)g(b)q(e)g(easy)f(to)g(test)164 420 y(for)g(exact)f (matc)o(h.)237 480 y(The)f(use)g(of)e(a)i(return)f(status)f(ob)s(ject,)h (rather)g(than)h(a)f(list)h(of)f(parameters)f(ma)o(y)h(simplify)164 540 y(the)k(use)g(of)g(MPI)f(routines,)i(if)f(the)g(v)m(alues)h(stored)e(in)i (the)f(ob)s(ject)f(are)h(seldom)g(c)o(hec)o(k)o(ed.)23 b(A)164 601 y(prede\014ned)17 b(return)e(status)f(ob)s(ject)h(should)h(b)q(e)g(pro)o (vided,)g(to)e(ease)h(programming.)164 851 y Fi(1.9.4)55 b(Multiple)20 b(Completions)164 943 y Fl(It)f(is)g(con)o(v)o(enien)o(t)e(to)i(b)q(e)g(able) g(to)h(w)o(ait)e(for)i(the)f(completion)e(of)i(an)o(y)g(or)h(all)e(the)h(op-) 164 1003 y(erations)f(in)g(a)g(set,)g(rather)g(than)g(ha)o(ving)g(to)g(w)o (ait)g(for)g(sp)q(eci\014c)g(message.)25 b(A)17 b(call)h(to)164 1064 y Fg(MPI)p 245 1064 16 2 v 17 w(WAITANY)g Fl(or)j Fg(MPI)p 604 1064 V 17 w(STATUSANY)c Fl(can)k(b)q(e)f(used)h(to)f(w)o(ait)h(for)f(the) g(completion)f(of)164 1124 y(one)e(out)g(of)f(sev)o(eral)g(op)q(erations;)h (a)g(call)f(to)g Fg(MPI)p 1079 1124 V 18 w(WAITALL)e Fl(can)i(b)q(e)h(used)g (to)f(w)o(ait)h(for)164 1184 y(all)f(p)q(ending)g(op)q(erations)i(in)e(a)g (list.)164 1304 y Fi(MPI)p 279 1304 17 2 v 20 w(W)-6 b(AIT)h(ANY)20 b(\()e(list)p 710 1304 V 21 w(of)p 776 1304 V 21 w(handles,)h(index,)f (return)p 1338 1304 V 20 w(status)p 1504 1304 V 20 w(handle\))237 1425 y Fl(Blo)q(c)o(ks)k(un)o(til)g(one)g(of)h(the)g(op)q(erations)h(asso)q (ciated)f(with)g(the)f(comm)o(unicati)o(on)164 1485 y(handles)16 b(in)f(the)g(arra)o(y)h(has)g(completed.)j(Returns)d(the)f(index)g(of)g(that) h(handle)g(in)f(the)164 1545 y(arra)o(y)l(,)h(and)g(returns)g(the)g(status)g (of)h(that)f(op)q(eration)h(in)e(the)h(ob)s(ject)f(asso)q(ciated)i(with)164 1605 y(the)f(return)p 384 1605 15 2 v 17 w(status)p 528 1605 V 19 w(handle.)21 b(The)16 b(parameters)f(are:)164 1720 y Fi(IN)j(list)p 324 1720 17 2 v 22 w(of)p 391 1720 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 1821 y Fi(OUT)19 b(index)24 b Fl(index)16 b(of)g(handle)g(for)h(op)q(eration)g (that)g(completed)d(\(in)o(teger\).)164 1923 y Fi(OUT)19 b(return)p 466 1923 V 19 w(status)p 631 1923 V 21 w(handle)25 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)h(with)e(return)h(status)286 1983 y(ob)s(ject.)i(Set)16 b(to)g(return)g(status)h(of)g(op)q(eration)g(that)g (completed.)237 2097 y(The)11 b(successful)f(execution)g(of)h Fg(MPI)p 892 2097 16 2 v 17 w(WAITANY\(lis)o(t)p 1220 2097 V 15 w(of)p 1287 2097 V 18 w(handles,)23 b(index,)g(return)p 1871 2097 V 17 w(status)p 2044 2097 V 16 w(handle\))164 2157 y Fl(is)11 b(equiv)m(alen)o(t)e(to)i(the)g(successful)f(execution)g(of)h Fg(MPI)p 1127 2157 V 17 w(WAIT\(handl)o(e[i)o(],)22 b(return)p 1710 2157 V 16 w(status)p 1882 2157 V 17 w(handle\))p Fl(,)164 2218 y(where)15 b Fg(i)g Fl(is)h(the)f(v)m(alue)g(returned)g(b)o(y)g Fg(index)f Fl(and)i Fg(handle[i])d Fl(is)i(the)g Fg(i)p Fl(-th)h(handle)f(in) 164 2278 y(the)h(list.)237 2338 y(If)f(more)f(then)h(one)g(op)q(eration)i(is) e(enabled)g(and)g(can)h(terminate,)d(one)i(is)g(arbitrar-)164 2398 y(ily)20 b(c)o(hosen)i(\(sub)s(ject)f(to)g(the)h(restrictions)f(on)h(op) q(eration)g(termination)e(order,)i(see)164 2458 y(Section)16 b(1.13\).)949 2599 y(26)p eop %%Page: 27 27 bop 164 367 a Fg(MPI)p 245 367 16 2 v 17 w(WAITANY)23 b(\()j(list)p 623 367 V 17 w(of)p 692 367 V 17 w(handles,)d(index,)g(return)p 1275 367 V 17 w(status)p 1448 367 V 16 w(handle\))164 428 y Fl(is)241 529 y Fg({MPI_WAIT_)o(RE)o(CV)f(\(handle[1],)g(\\return\\_s)o(tat)o (us)o(\\_h)o(and)o(le\))o(;)g(index)i(=)h(0})g(||)g(...)164 589 y(||)241 650 y({MPI_WAIT_)o(RE)o(CV)d(\(handle[n],)g(\\return\\_s)o(tat)o (us)o(\\_h)o(and)o(le\))o(;)g(index)i(=)h(n-1})237 751 y Fl(\(\\)p Fe(jj)p Fl(")18 b(indicates)f(c)o(hoice;)f(one)i(of)f(the)g(alternativ)o(es)g (is)g(c)o(hosen,)g(nondeterministi-)164 812 y(cally)l(.\))164 932 y Fi(MPI)p 279 932 17 2 v 20 w(ST)-5 b(A)g(TUSANY)21 b(\()d(list)p 777 932 V 22 w(of)p 844 932 V 20 w(handles,)h(index,)f(return)p 1405 932 V 20 w(status)p 1571 932 V 20 w(handle\))237 1052 y Fl(Causes)c(either)e(one)h(or)h(none)f(of)g(the)g(op)q(erations)h(asso)q (ciated)g(with)f(the)g(comm)o(uni-)164 1112 y(cation)g(handles)f(to)h (return.)20 b(In)12 b(the)g(former)g(case,)g(it)g(has)i(the)e(same)f(return)i (seman)o(tics)164 1173 y(as)j(a)g(call)f(to)h Fg(MPI)p 492 1173 16 2 v 17 w(WAIT)p 613 1173 V 17 w(ANY)p Fl(.)e(In)i(the)f(later)g (case,)g(it)g(returns)h(a)g(v)m(alue)f(of)h(-1)g(in)g Fg(index)164 1233 y Fl(and)h Fg(return)p 418 1233 V 16 w(status)p 590 1233 V 16 w(handle)d Fl(is)i(unde\014ned.)22 b(The)16 b(parameters)f(are:)164 1347 y Fi(IN)j(list)p 324 1347 17 2 v 22 w(of)p 391 1347 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s (jects.)164 1449 y Fi(OUT)19 b(index)24 b Fl(index)e(of)h(handle)g(for)g(op)q (eration)h(that)g(completed,)d(or)j(-1)f(if)g(none)286 1509 y(completed)14 b(\(in)o(teger\).)164 1611 y Fi(OUT)19 b(return)p 466 1611 V 19 w(status)p 631 1611 V 21 w(handle)25 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)h(with)e(return)h(status)286 1671 y(ob)s(ject.)h(Set)c(to)h(return)f(status)g(of)h(op)q(eration)g(that)f (completed,)e(if)h(an)o(y;)h(unde-)286 1731 y(\014ned)g(when)g Fg(index)24 b(=)i(-1)p Fl(.)237 1845 y Fi(MPI)p 352 1845 V 21 w(W)-6 b(AIT)h(ALL\(list)p 728 1845 V 20 w(of)p 793 1845 V 21 w(handles,)19 b(list)p 1106 1845 V 21 w(of)p 1172 1845 V 20 w(return)p 1348 1845 V 20 w(status)p 1514 1845 V 20 w(handles\))237 1966 y Fl(Blo)q(c)o(ks)14 b(un)o(til)f(all)i(comm)o(uni)o(cation)d(op)q (erations)k(asso)q(ciated)g(with)e(handles)h(in)f(the)164 2026 y(list)i(complete,)e(and)j(return)g(the)f(status)h(of)g(all)f(these)h(op)q (erations.)23 b(The)17 b(parameters)164 2086 y(are:)164 2200 y Fi(IN)h(list)p 324 2200 V 22 w(of)p 391 2200 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 2302 y Fi(OUT)19 b(list)p 384 2302 V 21 w(of)p 450 2302 V 20 w(return)p 626 2302 V 20 w(status)p 792 2302 V 20 w(handles)25 b Fl(Must)f(ha)o(v)o(e)f(the)g(same)g(length)g(as)h(the)286 2362 y(\014rst)c(list.)30 b(Eac)o(h)19 b(return)h(status)g(ob)s(ject)f(is)g (set)h(to)f(the)h(return)f(status)h(of)g(the)286 2422 y(corresp)q(onding)d (op)q(eration)g(in)f(the)g(\014rst)h(list.)949 2599 y(27)p eop %%Page: 28 28 bop 164 307 a Fm(1.10)70 b(Blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Blo)q(c)o(king)d(send)h(and)h(receiv)o(e)d(op)q(erations)j(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(er-)164 460 y(ations)k(in)o(to)f (one)h(call.)32 b(The)21 b(op)q(eration)g(returns)f(only)g(when)h(the)f(comm) o(unicati)o(on)164 520 y(completes)g(and)i(no)g(comm)o(unic)o(ation)e(ob)s (ject)h(p)q(ersists)h(after)f(the)h(call)f(completed.)164 580 y(Ho)o(w)o(ev)o(er,)14 b(the)i(bu\013er)h(descriptor)e(ob)s(ject)h(needs)g(b) q(e)h(created)e(ahead)i(of)g(the)f(call.)237 640 y(W)l(e)g(use)g(the)g(follo) o(wing)g(naming)g(con)o(v)o(en)o(tion)f(for)h(suc)o(h)g(op)q(erations:)777 695 y Fa(")824 737 y Fe(\000)822 797 y Fi(R)885 695 y Fa(#)8 b(")966 737 y Fi(SEND)962 797 y(RECV)1145 695 y Fa(#)237 898 y Fl(The)16 b(\014rst)h(letter)e(\(v)o(oid)h(or)g Fi(R)p Fl(\))g(indicates)g (the)g(start)h(mo)q(de)e(\(regular)h(or)h(ready\).)164 1018 y Fi(MPI)p 279 1018 17 2 v 20 w(SEND)i(\(bu\013er)p 639 1018 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1078 y Fl(is)164 1170 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1231 y(MPI_START\()o(han)o (dl)o(e\))164 1291 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1443 y Fi(MPI)p 279 1443 V 20 w(RSEND)c(\(bu\013er)p 681 1443 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1503 y Fl(is)164 1596 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1656 y(MPI_START\()o(han)o(dl)o (e\))164 1716 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1868 y Fi(MPI)p 279 1868 V 20 w(RECV\(bu\013er)p 626 1868 V 20 w(handle,)14 b(source,)f(tag,)g(con)n(text,)g(return)p 1494 1868 V 20 w(status)p 1660 1868 V 20 w(handle\))164 1929 y Fl(is)164 2021 y Fg(MPI_INIT_R)o(ECV)o (\(h)o(and)o(le,)22 b(buffer_han)o(dl)o(e,)g(source,)h(tag,)i(context,)d (EPHEMERAL\))164 2081 y(MPI_START\()o(han)o(dl)o(e\))164 2141 y(MPI_WAIT\(h)o(and)o(le)o(,re)o(tur)o(n_s)o(ta)o(tus)o(_ha)o(nd)o(le\))237 2233 y Fi(Implemen)n(tation)d(note:)237 2293 y Fl(While)k(these)g(functions)h (can)g(b)q(e)g(impleme)o(n)o(ted)d(via)i(calls)g(to)i(functions)e(that)164 2354 y(implem)o(en)o(t)c(sub)q(op)q(erations,)25 b(as)e(describ)q(ed)f(in)g (this)g(subsection,)h(an)g(e\016cien)o(t)d(im-)164 2414 y(plemen)o(tation)d (ma)o(y)h(optimize)f(a)o(w)o(a)o(y)i(these)g(m)o(ultiple)d(calls,)j(pro)o (vided)f(it)h(do)q(es)h(not)164 2474 y(c)o(hange)c(the)g(b)q(eha)o(vior)g(of) h(correct)e(programs.)949 2599 y(28)p eop %%Page: 29 29 bop 164 307 a Fm(1.11)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Non)o(blo)q(c)o(king)g(send)h(and)g(receiv)o(e)d(op)q(erations)k (com)o(bine)d(the)h(\014rst)h(t)o(w)o(o)g(sub)q(op)q(era-)164 460 y(tions)d(\()p Fg(INIT)e Fl(and)i Fg(START)p Fl(\))e(in)o(to)h(one)h (call.)30 b(They)19 b(use)h(ephemeral)d(comm)o(unicati)o(on)164 520 y(ob)s(jects,)e(so)h(that)f(the)g(op)q(eration)i(is)e(completed,)d(and)k (the)f(asso)q(ciated)i(resources)e(are)164 580 y(freed,)g(b)o(y)h(using)h (one)f(of)h(the)f(functions)g Fg(MPI)p 1014 580 16 2 v 17 w(WAIT,)24 b(MPI)p 1263 580 V 17 w(STATUS,)f(MPI)p 1563 580 V 18 w(WAITANY,)164 640 y(MPI)p 245 640 V 17 w(STATUSANY)p Fl(,)11 b(or)j Fg(MPI)p 656 640 V 17 w(WAITALL)p Fl(.)d(Here,)i(to)q(o,)i(a)g(bu\013er)f(ob)s(ject)f (has)i(to)f(b)q(e)g(created)164 700 y(ahead)j(of)f(the)g(comm)o(unication)d (initiation)j(op)q(eration.)237 761 y(W)l(e)h(use)g(the)f(same)g(naming)g (con)o(v)o(en)o(tion)g(as)i(for)f(blo)q(c)o(king)f(op)q(erations:)24 b(a)17 b(pre\014x)164 821 y(of)f Fi(R)g Fl(indicates)g(the)f Fg(READY)g Fl(mo)q(de.)20 b(In)c(addition,)g(a)g(pre\014x)g(of)g Fi(I)g Fl(is)g(used)g(to)h(indicate)164 881 y Fh(imme)n(diate)f Fl(\(i.e.,)e(non)o(blo)q(c)o(king\))i(execution.)164 1001 y Fi(MPI)p 279 1001 17 2 v 20 w(ISEND)j(\(handle,)g(bu\013er)p 856 1001 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1062 y Fl(is)164 1163 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o (e,)g(dest,)i(tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1224 y(MPI_START\()o(han)o(dl)o(e\))164 1385 y Fi(MPI)p 279 1385 V 20 w(IRSEND)18 b(\(handle,)h(bu\013er)p 897 1385 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1446 y Fl(is)164 1547 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1608 y(MPI_START\()o(han)o(dl)o (e\))164 1769 y Fi(MPI)p 279 1769 V 20 w(IRECV\(handle,)15 b(bu\013er)p 839 1769 V 19 w(handle,)f(source,)f(tag,)g(con)n(text,)g(return) p 1706 1769 V 19 w(status)p 1871 1769 V 21 w(handle\))164 1830 y Fl(is)164 1931 y Fg(MPI_INIT_R)o(ECV)o(\(h)o(and)o(le,)22 b(buffer_han)o(dl)o(e,)g(source,)h(tag,)i(context,)d(EPHEMERAL\))164 1991 y(MPI_START\()o(han)o(dl)o(e\))164 2136 y Fm(1.12)70 b(Blo)r(c)n(k)22 b(Sending)h(Op)r(erations)164 2228 y Fl(The)h(most)g(frequen)o(t)f(t)o(yp)q (e)g(of)i(bu\013er)f(used)g(is)g(a)h(con)o(tiguous)g(bu\013er)f(of)h(n)o (umeric)164 2288 y(storage)18 b(units,)f(i.e.,)f(a)h(con)o(tiguous)h (bu\013er)g(of)f(w)o(ords)h(that)g(ma)o(y)d(con)o(tain)i(either)f(IN-)164 2349 y(TEGER,)23 b(REAL)g(or)h(LOGICAL)f(v)m(alues)g(\(in)g(F)o(OR)l(TRAN\).) e(In)h(a)i(homogeneous)164 2409 y(en)o(vironmen)o(t)14 b(suc)o(h)j(messages)g (can)g(b)q(e)g(used)h(to)f(send)h(arbitrary)f(sequences)f(of)h(con-)164 2469 y(tiguous)g(items,)d(where)i(eac)o(h)f(item)g(o)q(ccupies)h(an)g(in)o (teger)g(n)o(um)o(b)q(er)e(of)j(w)o(ords.)949 2599 y(29)p eop %%Page: 30 30 bop 237 307 a Fl(W)l(e)17 b(sp)q(ecialize)f(the)i(functions)f(in)g(the)h(t)o (w)o(o)f(previous)g(subsections)h(to)g(this)f(case,)164 367 y(th)o(us)f(a)o(v)o(oiding)f(the)g(need)g(for)h(the)g(creation)f(of)h(a)g (bu\013er)g(descriptor)f(ob)s(ject.)21 b(W)l(e)15 b(use)164 428 y(the)i(same)f(naming)g(sc)o(heme)f(used)i(in)g(the)f(previous)h (subsections,)g(and)h(app)q(end)g(a)f Fi(B)164 488 y Fl(in)f(the)g(function)g (name,)f(for)h Fg(BLOCK)p Fl(.)164 608 y Fi(MPI)p 279 608 17 2 v 20 w(SENDB)j(\(start,)f(len,)g(dest,)g(tag,)g(con)n(text\))164 668 y Fl(is)164 770 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 830 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o (dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 890 y(MPI_SEND)f(\(buffer_h)o(and)o (le,)f(len,)i(dest,)g(tag,)g(context\))164 1052 y Fi(MPI)p 279 1052 V 20 w(RSENDB)18 b(\(bu\013er)p 720 1052 V 20 w(handle,)h(dest,)f (tag,)g(con)n(text\))164 1112 y Fl(is)164 1214 y Fg(MPI_CREATE)o(\(bu)o(ff)o (er_)o(han)o(dle)o(,)k(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1274 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\)) 164 1335 y(MPI_RSEND)e(\(buffer_han)o(dle)o(,)g(len,)j(dest,)e(tag,)i (context\))237 1436 y Fi(MPI)p 352 1436 V 21 w(RECVB\(bu\013er)p 740 1436 V 19 w(handle,)14 b(source,)f(tag,)g(con)n(text,)g(return)p 1607 1436 V 20 w(status)p 1773 1436 V 20 w(handle\))164 1496 y Fl(is)164 1598 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1658 y(MPI_ADD\(bu)o(ffe)o(r_)o (han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 1719 y(MPI_RECV\(b)o(uff)o (er)o(_ha)o(ndl)o(e,)e(source,)h(tag,)h(context,)f(return_sta)o(tus)o(_ha)o (nd)o(le\))164 1880 y Fi(MPI)p 279 1880 V 20 w(ISENDB)c(\(handle,)f(bu\013er) p 895 1880 V 20 w(handle,)h(dest,)f(tag,)g(con)n(text\))164 1941 y Fl(is)164 2042 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 2103 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 2163 y(MPI_ISEND\()o(han)o(dl)o (e,)e(buffer_hand)o(le,)g(dest,)i(tag,)g(context\))164 2325 y Fi(MPI)p 279 2325 V 20 w(IRSENDB)18 b(\(handle,)h(bu\013er)p 937 2325 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 2385 y Fl(is)949 2599 y(30)p eop %%Page: 31 31 bop 164 307 a Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 367 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 428 y(MPI_IRSEND)o(\(ha)o(nd)o (le,)e(buffer_han)o(dle)o(,)g(dest,)i(tag,)g(context\))164 587 y Fi(MPI)p 279 587 17 2 v 20 w(IRECVB\(handle,)14 b(bu\013er)p 878 587 V 20 w(handle,)g(source,)f(tag,)g(con)n(text,)g(return)p 1746 587 V 19 w(status)p 1911 587 V 21 w(handle\))164 647 y Fl(is)164 746 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 807 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 867 y(MPI_IRECV\()o(han)o(dl)o(e,) e(buffer_hand)o(le,)g(source,)h(tag,)h(context\))164 1075 y Fk(Discussion:)237 1132 y Fj(I)18 b(use)f(w)o(ord)g(coun)o(t,)g(rather)f (than)h(b)o(yte)g(coun)o(t.)26 b(I)18 b(b)q(eliev)o(e)h(w)o(ord)d(messages)h (are)g(m)o(uc)o(h)164 1188 y(more)e(prev)m(alen)o(t)g(than)g(b)o(yte)g (messages,)f(and)h(it's)g(a)g(blessing)h(not)f(to)f(ha)o(v)o(e)h(to)f(m)o (ultiply)j(b)o(y)164 1245 y(4)h(for)g(eac)o(h)h(message.)30 b(Byte)19 b(messages)f(are)g(still)j(a)o(v)m(ailable)f(the)f(hard)f(w)o(a)o (y)l(.)30 b(Also,)20 b(w)o(ord)164 1301 y(messages)15 b(\014t)g(w)o(ell)h (with)f(the)h(F)l(ortran)e(n)o(umeric)i(storage)e(unit)i(concept.)237 1357 y(If)21 b(w)o(e)f(are)g(to)g(add)h(more)f(functions,)j(m)o(y)d(next)g (addition)i(w)o(ould)f(b)q(e)g(a)f(v)o(ector)g(send)164 1414 y(op)q(eration.)164 1739 y Fm(1.13)70 b(Correctness)164 1951 y Fk(Discussion:)27 b Fj(The)18 b(material)h(in)g(this)g(section)g(has)g(not) f(y)o(et)g(b)q(een)h(discussed)h(b)o(y)f(MPIF.)164 2011 y(Some)d(or)g(all)h (of)f(it)h(is)f(lik)o(ely)i(to)e(mo)o(v)o(e)g(to)f(Section)i Fk(??)p Fj(.)23 b(It)16 b(is)h(incorp)q(orated)g(here)f(for)g(com-)164 2072 y(pleteness.)164 2322 y Fi(1.13.1)55 b(Order)164 2414 y Fl(MPI)15 b(preserv)o(es)g(the)h(order)g(of)g(messages)f(b)q(et)o(w)o(een)g (an)o(y)h(\014xed)f(pair)h(of)h(pro)q(cesses.)k(In)164 2474 y(other)16 b(w)o(ords,)g(if)f(pro)q(cess)h(A)f(executes)f(t)o(w)o(o)i (successiv)o(e)e(send)i Fg(start)d Fl(sub)q(op)q(erations,)949 2599 y(31)p eop %%Page: 32 32 bop 164 307 a Fl(pro)q(cess)13 b(B)e(executes)g(t)o(w)o(o)h(successiv)o(e)e (receiv)o(e)g Fg(start)g Fl(op)q(erations,)k(and)e(b)q(oth)h(receiv)o(es)164 367 y(matc)o(h)19 b(either)h(sends,)i(then)f(the)f(\014rst)h(receiv)o(e)e (will)h(receiv)o(e)e(the)j(message)f(sen)o(t)g(b)o(y)164 428 y(the)g(\014rst)g(send,)h(and)g(the)f(second)g(receiv)o(e)e(will)h(receiv)o (e)f(the)i(message)g(sen)o(t)f(b)o(y)h(the)164 488 y(second)h(send.)36 b(Th)o(us,)22 b(if)e(a)i(t)o(w)o(o)f(messages)f(from)g(the)h(same)f(source)h (can)g(satisfy)h(a)164 548 y(p)q(ending)d(receiv)o(e,)c(the)j(\014rst)h (message)e(sen)o(t)h(is)g(accepted;)g(if)g(a)g(message)g(can)g(satisfy)164 608 y(t)o(w)o(o)e(p)q(ending)h(receiv)o(es,)c(the)j(\014rst)h(receiv)o(e)d(p) q(osted)j(is)f(satis\014ed.)237 668 y(The)h(last)g(paragraph)i(assumes)e (that)g(the)g(send)g Fg(start)f Fl(op)q(erations)i(are)f(ordered)164 729 y(b)o(y)k(the)g(program)h(order)f(at)h(pro)q(cess)g(A,)f(and)h(the)f (receiv)o(e)f Fg(start)f Fl(op)q(erations)k(are)164 789 y(ordered)f(b)o(y)f (the)h(program)f(order)h(at)g(pro)q(cess)h(B.)e(If)g(a)h(pro)q(cess)h(is)e(m) o(ultithreaded)164 849 y(and)f(the)f(op)q(erations)h(are)f(executed)f(b)o(y)g (distinct)h(threads,)g(then)g(the)g(seman)o(tics)f(of)164 909 y(the)d(threaded)g(system)e(ma)o(y)h(not)h(de\014ne)g(an)g(order)h(b)q(et)o (w)o(een)e(the)g(t)o(w)o(o)h(op)q(erations,)h(in)164 969 y(whic)o(h)g(case)g (the)g(condition)g(is)g(v)o(oid.)164 1099 y Fi(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)164 1192 y Fl(W)l(e)22 b(can)h(mo)q(del)f(the)g (execution)g(of)h(MPI)f(programs)h(as)h(an)f(in)o(teraction)f(b)q(et)o(w)o (een)164 1252 y(executing)h(pro)q(cesses)h(that)g(execute)e(eac)o(h)h(their)g (o)o(wn)h(program,)h(and)f(the)g Fi(com-)164 1312 y(m)n(unication)c (subsystem)p Fl(.)g(The)c(comm)o(unic)o(ation)d(subsystem)i(ma)o(y)f(ha)o(v)o (e)h(v)m(arious)164 1372 y(constrain)o(ts)h(on)h(the)f(amoun)o(t)g(of)g (resources)g(it)g(can)h(use.)k(E.g.:)237 1432 y(Bounds)e(on)f(the)g(n)o(um)o (b)q(er)e(and)i(total)h(sizes)e(of)h(activ)o(e)f(comm)o(unic)o(ation)f(ob)s (jects.)164 1492 y(Suc)o(h)g(b)q(ound)h(can)g(b)q(e)f(global,)g(p)q(er)h(no)q (de,)f(or)h(p)q(er)f(pair)g(of)h(comm)o(unic)o(ating)d(no)q(des.)237 1553 y(Bounds)g(on)g(the)f(n)o(um)o(b)q(er)f(and)i(total)g(sizes)f(of)h (messages)f(bu\013ered)g(in)h(the)f(system.)164 1613 y(Suc)o(h)19 b(b)q(ound)h(can,)g(again,)g(b)q(e)g(global,)g(p)q(er)f(no)q(de,)h(or)g(p)q (er)f(pair)h(of)f(comm)o(unicati)o(ng)164 1673 y(no)q(de.)j(In)16 b(addition,)f(a)i(message)e(ma)o(y)g(b)q(e)h(bu\013ered)g(at)g(the)g(sender,) g(at)g(the)g(receiv)o(er,)164 1733 y(at)h(b)q(oth,)f(or)h(p)q(erhaps)g(at)g (another)g(place)e(altogether.)237 1793 y(Th)o(us,)g(it)g(will)f(b)q(e)h (di\016cult)e(to)j(set)f(rules)f(on)i(resource)e(managemen)o(t)f(of)i(the)g (com-)164 1854 y(m)o(unication)g(subsystem.)21 b(Ho)o(w)o(ev)o(er,)15 b(it)h(is)g(generally)g(exp)q(ected)f(that)i(impleme)o(n)o(ters)164 1914 y(will)e(pro)o(vide)f(information)h(on)h(the)g(mec)o(hanism)c(used)k (for)g(resource)g(allo)q(cation,)f(and)164 1974 y(that)22 b(query)g(and)g (set)g(functions)g(will)f(allo)o(w)h(to)g(query)g(and)g(p)q(ossibly)g(con)o (trol)g(the)164 2034 y(amoun)o(t)15 b(of)i(a)o(v)m(ailable)f(resources.)237 2094 y(W)l(e)d(pro)o(vide)f(in)g(this)h(section)g(a)g(set)g(of)g(minim)o(al)d (requiremen)o(ts)g(on)j(the)g(comm)o(uni-)164 2155 y(cation)h(subsystem.)19 b(Programs)14 b(that)g(execute)e(on)j(an)o(y)e(subsystem)g(that)h(ful\014ls)f (these)164 2215 y(minim)o(al)18 b(requiremen)o(ts)g(are)j Fi(safe)g Fl(and)h(will)d(p)q(ort)j(to)f(an)o(y)g(MPI)f(impleme)o(n)o(tation.)164 2275 y Fi(Unsafe)h Fl(programs)h(ma)o(y)d(execute)h(on)i(some)e(MPI)h(implem) o(e)o(n)o(tations,)f(dep)q(ending)164 2335 y(on)h(the)f(amoun)o(t)f(of)i(a)o (v)m(ailable)e(resources)h(and)h(the)f(implem)o(en)o(tation)d(used)k(for)f (the)164 2395 y(MPI)14 b(comm)o(unication)e(subsystem.)19 b(Finally)14 b Fi(erroneous)g Fl(programs)h(nev)o(er)e(execute)164 2456 y(correctly)l(.)19 b(\(While)14 b(it)g(is)g(desirable)g(to)g(detect)g (erroneous)h(programs,)g(it)f(is)g(not)h(p)q(ossi-)949 2599 y(32)p eop %%Page: 33 33 bop 164 307 a Fl(ble)13 b(to)g(do)h(so)g(at)g(compile)d(time,)h(and)i(often)f (prohibitiv)o(e)f(to)i(do)f(so)i(a)e(run)h(time.)k(Th)o(us,)164 367 y(the)e(do)q(cumen)o(t)f(do)q(es)j(not)f(sp)q(ecify)e(a)i(b)q(eha)o(vior) g(for)g(erroneous)g(programs,)f(although)164 428 y(the)g(desired)g(b)q(eha)o (vior)g(is)g(to)g(return)g(a)h(useful)f(error)g(message.\))224 542 y(1.)24 b(If)15 b(a)h(pro)q(cess)h(executes)d(an)j Fg(INIT)d Fl(op)q(eration,)i(then)g(the)f(op)q(eration)i(ev)o(en)o(tually)286 602 y(succeeds,)c(or)g(a)g Fh(r)n(esour)n(c)n(e)h(exc)n(eption)g Fl(o)q(ccurs.)21 b(The)13 b(standard)h(do)q(es)g(not)g(sp)q(ecify)286 662 y(when)23 b(a)h(resource)g(exception)e(is)h(allo)o(w)o(ed)g(to)h(o)q (ccur.)43 b(It)23 b(is)g(exp)q(ected)f(that)286 722 y(an)g(op)q(erational)h (de\014nition)e(will)g(b)q(e)h(made)f(a)o(v)m(ailable,)h(in)f(the)h(form)f (of)h(test)286 782 y(programs)14 b(that)f(ha)o(v)o(e)g(to)h(execute)e(with)h (no)h(resource)g(exceptions.)19 b(It)13 b(is)g(highly)286 843 y(desirable)e(to)i(ha)o(v)o(e)e(generous)i(b)q(ounds)h(on)e(the)g(n)o(um)o(b) q(er)f(of)h(concurren)o(tly)f(activ)o(e)286 903 y(comm)o(unic)o(ation)20 b(ob)s(jects)j(eac)o(h)f(pro)q(cess)h(ma)o(y)e(ha)o(v)o(e,)i(so)h(that,)g(in) e(practice,)286 963 y Fg(INIT)15 b Fl(op)q(erations)i(will)e(alw)o(a)o(ys)h (b)q(e)h(guaran)o(teed)f(to)h(succeed.)224 1065 y(2.)24 b(Eac)o(h)d(pro)q (cess)g(can)g(initiate)f(a)h(comm)o(uni)o(cation)d(op)q(eration)k(for)f(eac)o (h)f(activ)o(e)286 1125 y(comm)o(unic)o(ation)e(ob)s(ject.)32 b(I.e.)f(correct)20 b Fg(START)e Fl(op)q(erations)j(alw)o(a)o(ys)f(succeed) 286 1185 y(\(ev)o(en)o(tually\).)224 1287 y(3.)k(A)10 b(send)h(op)q(eration)h (is)f Fi(enabled)g Fl(if)f(the)h(sending)g(pro)q(cess)g(has)h(issued)e(a)i Fg(COMPLETE)286 1347 y Fl(op)q(eration)20 b(and)g(the)g(receiving)e(pro)q (cess)i(has)g(issued)g(a)f Fg(START)f Fl(op)q(eration)j(for)286 1407 y(a)e(matc)o(hing)f(receiv)o(e.)28 b(Symmetri)o(call)o(y)l(,)16 b(a)k(receiv)o(e)d(op)q(eration)j(is)f Fi(enabled)g Fl(if)286 1467 y(the)h(receiving)e(pro)q(cess)j(has)f(issued)g(a)g Fg(COMPLETE)d Fl(op)q(eration)k(and)g(the)f(send-)286 1528 y(ing)j(pro)q(cess)h(has)g (issued)f(a)g Fg(START)f Fl(op)q(eration)i(for)f(a)h(matc)o(hing)d(send.)42 b(An)286 1588 y(enabled)19 b(op)q(eration)i(ma)o(y)d(b)q(ecome)g Fi(disabled)j Fl(either)e(b)q(ecause)h(it)f(completes)286 1648 y(successfully)c(or,)h(in)g(the)g(case)g(of)h(a)g(receiv)o(e,)c(b)q(ecause)k (the)f(matc)o(hing)f(message)286 1708 y(is)h(successfully)f(receiv)o(ed)f(b)o (y)i(another)g(receiv)o(e)e(op)q(eration.)286 1789 y Fi(An)30 b(enabled)h(op)r(eration)f(either)f(completes)g(successfully)i(or)f(b)r(e-) 286 1849 y(comes)18 b(p)r(ermanen)n(tly)g(disabled.)224 1951 y Fl(4.)24 b(A)16 b Fg(FREE)f Fl(op)q(eration)i(alw)o(a)o(ys)f(succeeds)f (\(ev)o(en)o(tually\).)237 2065 y(The)f(four)h(conditions)f(guaran)o(tee)h (progress)g(in)e(the)h(comm)o(unication)d(subsystem.)164 2125 y(The)19 b(third)f(condition)h(guaran)o(tee)g(\(w)o(eak\))f(fairness)h(among) f(comp)q(eting)g(comm)o(uni-)164 2186 y(cation)e(requests.)237 2246 y(Examples)f(\(in)o(v)o(olving)g(t)o(w)o(o)h(pro)q(cessors)h(with)f (ranks)h(0)g(and)f(1\))237 2306 y(The)g(follo)o(wing)g(program)g(is)g(safe,)g (and)h(should)g(alw)o(a)o(ys)f(succeed.)164 2468 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))949 2599 y Fl(33)p eop %%Page: 34 34 bop 164 307 a Fg(IF)25 b(\(rank.EQ.0)o(\))d(THEN)241 367 y(CALL)i (MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))241 428 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 488 y(ELSE)101 b(!)25 b(rank.EQ.1)241 548 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o (f,)e(len,)i(0,)h(tag,)f(context\))241 608 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o (f,)e(len,)i(0,)h(tag,)f(context\))164 668 y(END)h(IF)237 765 y Fl(The)16 b(follo)o(wing)g(program)g(is)g(erroneous,)h(and)g(should)f(alw)o (a)o(ys)g(fail.)164 934 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 994 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 1054 y(CALL)i(MPI_RECVB\()o(rec)o(vbu) o(f,)e(len,)i(1,)h(tag,)f(context\))241 1114 y(CALL)g(MPI_SENDB\()o(sen)o (dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 1174 y(ELSE)101 b(!)25 b(rank.EQ.1)241 1235 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i (0,)h(tag,)f(context\))241 1295 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,) i(0,)h(tag,)f(context\))164 1355 y(END)h(IF)237 1463 y Fl(The)17 b(receiv)o(e)e(op)q(eration)j(of)f(the)f(\014rst)i(pro)q(cess)f(m)o(ust)f (complete)e(b)q(efore)k(its)e(send,)164 1523 y(and)23 b(can)g(complete)e (only)h(if)g(the)h(matc)o(hing)e(receiv)o(e)f(of)j(the)g(second)f(pro)q (cessor)i(is)164 1584 y(executed;)17 b(the)h(receiv)o(e)d(op)q(eration)k(of)f (the)g(second)g(pro)q(cess)h(m)o(ust)d(complete)g(b)q(efore)164 1644 y(its)g(send)g(and)h(can)f(complete)e(only)i(if)f(the)h(matc)o(hing)f (receiv)o(e)f(of)i(the)g(\014rst)g(pro)q(cess)h(is)164 1704 y(executed.)j(This)c(program)g(will)f(deadlo)q(c)o(k.)237 1764 y(The)i(follo)o(wing)h(program)f(is)g(unsafe,)g(and)h(ma)o(y)e(succeed)h(or)g (fail,)g(dep)q(ending)g(on)164 1824 y(implem)o(en)o(tati)o(on.)164 1993 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 2053 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 2113 y(CALL)i(MPI_SENDB\()o(sen)o(dbu)o (f,)e(len,)i(1,)h(tag,)f(context\))241 2173 y(CALL)g(MPI_RECVB\()o(rec)o(vbu) o(f,)e(len,)i(1,)h(tag,)f(context\))164 2233 y(ELSE)101 b(!)25 b(rank.EQ.1)241 2293 y(CALL)f(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i(0,)h (tag,)f(context\))241 2354 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i(0,) h(tag,)f(context\))164 2414 y(END)h(IF)949 2599 y Fl(34)p eop %%Page: 35 35 bop 237 307 a Fl(The)19 b(message)f(sen)o(t)g(b)o(y)g(eac)o(h)h(pro)q(cess)g (has)g(to)g(b)q(e)g(copied)f(out)i(b)q(efore)e(the)h(send)164 367 y(op)q(eration)d(returns)g(and)g(the)f(receiv)o(e)e(op)q(eration)j (starts.)22 b(F)l(or)16 b(the)f(program)g(to)h(com-)164 428 y(plete,)i(it)h(is)g(necessary)g(that)h(at)f(least)g(one)h(of)f(the)g(t)o(w)o (o)g(messages)g(sen)o(t)g(is)g(bu\013ered)164 488 y(out)e(of)g(either)f(pro)q (cesses')h(address)g(space.)23 b(Th)o(us,)17 b(this)g(program)f(can)h (succeed)f(only)164 548 y(if)h(the)g(comm)o(uni)o(cation)e(system)g(has)j (su\016cien)o(t)e(bu\013er)i(space)f(to)h(bu\013er)f Fg(len)f Fl(w)o(ords)164 608 y(of)h(data.)237 668 y(If)d(additional)h(requiremen)o(ts) d(will)i(b)q(ecome)f(part)i(of)g(the)g(standard)h(\(e.g.,)e(b)q(ounds)164 729 y(on)23 b(the)f(minimal)d(n)o(um)o(b)q(er)i(of)i(concurren)o(tly)e(activ) o(e)h(handles)g(that)h(need)f(b)q(e)h(sup-)164 789 y(p)q(orted,)16 b(then)g(further)g(programs)h(b)q(ecome)d(safe.)949 2599 y(35)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Sun May 9 16:24:06 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA03565; Sun, 9 May 93 16:24:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12973; Sun, 9 May 93 16:23:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 9 May 1993 16:23:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12965; Sun, 9 May 93 16:23:56 -0400 Message-Id: <9305092023.AA12965@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 4843; Sun, 09 May 93 16:24:31 EDT Date: Sun, 9 May 93 16:24:31 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: postscript file Reply-To: SNIR@watson.ibm.com Contains updated version of draft I sent a few weeks ago. Put in corrections people pointed to me, and tried to write more clearly the section on data conversion. No major changes wrt last draft. From owner-mpi-pt2pt@CS.UTK.EDU Wed Jun 9 15:04:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA19292; Wed, 9 Jun 93 15:04:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21784; Wed, 9 Jun 93 15:03:53 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 9 Jun 1993 15:03:51 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from malloco.ing.puc.cl by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21776; Wed, 9 Jun 93 15:03:37 -0400 Received: by malloco.ing.puc.cl (4.1/5.17) id AA01217; Wed, 9 Jun 93 15:03:29 CST From: ricardo@ing.puc.cl (Ricardo Schmutzer von Oldershausen) Message-Id: <9306091903.AA01217@malloco.ing.puc.cl> Subject: Re: Buffer descriptor generalization To: mpi-pt2pt@cs.utk.edu Date: Wed, 9 Jun 93 15:03:28 AST X-Mailer: ELM [version 2.3 PL11] Hello ! I am very enthusiastic about James Cownie's proposal because now, it is not necessary any more to describe a complex structure each time you create a new buffer. Now, you just describe the type of the structure with an unbound buffer _once_, and you call MPI_APPEND_CONTIGUOUS each time you want to create a new buffer of that type (after MPI_CREATE_BUFFER). This simplifies the implementation and usage of user-types. This will also allow to implement a stub generator ('mpigen' ?) which generates the code to describe the unbound buffer from a source language type declaration automatically (reducing programming errors; similar as 'rpcgen' generates XDR functions for RPC). Unfortunetly, as the buffer building functions MPI_APPEND_* have been designed to build a buffer when the data to be sent is already known (amount and type), is seems to be impossible to describe a type with an unbound buffer, if the type contains an array of variable length or if it contains a union (I am writing this rather late, so you may have discussed this already ; my apologies in that case for placing this letter in the forum ...). Let me explain this with two examples: 1) Arrays of variable length: Let's suppose you want to send the following structure (inside a more complex one; simplified for the example): #define MAX_ARRAY_LENGTH 1000 struct var_array { int array_length; double array[MAX_ARRAY_LENGTH]; }; which describes an array of variable length with a maximum size MAX_ARRAY_LENGTH, but where we actually use only the first array_length elements. Therefore we also want to send only the first array_length elements. Describing the buffer inmediately before sending, there is no problem, because we do have a value for array_length: struct variable_array foo1; MPI_BUFFER b; MPI_CREATE_BUFFER(&b, MPI_EPHEMERAL); MPI_APPEND_CONTIGUOUS(b, &foo1.array, foo1.array_length, MPI_DOUBLE); MPI_COMMIT(b); But how do we describe this type (struct var_array) with an unbound buffer ? One solution could be: MPI_BUFFER b; MPI_CREATE_BUFFER(&b, MPI_PERSISTENCE); MPI_APPEND_CONTIGUOUS(b, (struct var_array *)0, MAX_ARRAY_LENGTH, MPI_DOUBLE); MPI_COMMIT(b); We cant't use array_length, because the value is still not known. This has the problem, that MPI will send all 1000 (MAX_LENGTH_ARRAY) elements, even if only the first 10 are used. A possible solution is to implement a slight modified version of MPI_APPEND_CONTIGUOUS (or better yet: of MPI_APPEND_VEC): MPI_APPEND_ARRAY(buffer, start, length, max_count, stride, ?, datatype) where a parameter receives a pointer to the array length (I do not remember if an unbound buffer must have MPI_NULL in his start parameter. In that case, instead of a pointer the new parameter would have to be an offset in bytes from the start parameter): MPI_BUFFER b; MPI_CREATE_BUFFER(&b, MPI_PERSISTENCE); MPI_APPEND_ARRAY(b, (struct var_array *)0->array, &((struct var_array *)0)->array_length, MAX_ARRAY_LENGTH, 1, 1, MPI_DOUBLE); MPI_COMMIT(b); Note, that the count parameter is preserved to allow the proper alignment if this array is inside a more complex structure (see the declaration of var_array). This function would allow us to describe types with arrays of variable length. Discussion: Is there also a need for a MPI_APPEND_HARRAY function ? 2) Unions Here we have an analogous problem, only that instead of not knowing the length of the array in advance, we do not know the type of the data in advance: struct foo_type { int type; union { int i; double d; char c; } foo_union; }; Describing the buffer inmediately before sending, there is no problem, because we do know the data type: struct foo_type foo2; MPI_BUFFER b; MPI_CREATE_BUFFER(&b, MPI_EPHEMERAL); MPI_APPEND_CONTIGUOUS(b, &foo2.type, 1, MPI_INT); switch (foo2.type) { case 0: MPI_APPEND_CONTIGUOUS(b, &foo2.foo_union.i, 1, MPI_INT); case 1: MPI_APPEND_CONTIGUOUS(b, &foo2.foo_union.d, 1, MPI_DOUBLE); case 2: MPI_APPEND_CONTIGUOUS(b, &foo2.foo_union.c, 1, MPI_CHAR); }; MPI_COMMIT(b); But we are unable to describe this type with an unbound buffer, because the type is not known in advance. A solution could be to define a function: MPI_APPEND_UNION(buffer, start, type, size, count, array_of_types) where buffer is the MPI_BUFFER, start is the location of the union, array_of_types is an array of MPI_BUFFERs, count is the length of array_of_types type is a pointer to an integer index into array_of_types (if the start parameter in an unbound buffer must be MPI_NULL, than type should be an offset in bytes from start) and size is the length of the union in bytes. So, the unbound buffer may be described as follows: MPI_BUFFER b; MPI_BUFFER types[3] = { MPI_INT, MPI_DOUBLE, MPI_CHAR }; MPI_CREATE_BUFFER(&b, MPI_PERSISTENCE); MPI_APPEND_UNION(b, ((struct foo_type *)0)->foo_union, &((struct foo_type *)0)->type, MPI_DOUBLE_SIZE, 3, types); MPI_COMMIT(b); I think, the size paramenter is needed to simplify alignment inside a more complex structure, specially if the types in the array_of_types are user-types. In that case, we would need a function MPI_SIZEOF(buffer) (or similar), if the size parameter would not exist. =============================================================================== The functions MPI_APPEND_ARRAY and MPI_APPEND_UNION follow the following goals: 1) Allow to describe user-types with unbound buffers, that contain variable length arrays or unions 2) Simplify the description of user-types 3) Allow to implement (at least in theory) a generator of code for unbound buffers from the source code type description. This would minimize development errors and would simplify the interface of MPI for users. Ricardo. -- +----------------------+---------------------------------+--------------+ | Ricardo A. Schmutzer | Internet: ricardo@ing.puc.cl | Conectividad | | PUC, Santiago, Chile | | Distr. Comp. | +----------------------+---------------------------------+--------------+ From owner-mpi-pt2pt@CS.UTK.EDU Wed Jun 9 15:04:38 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA19298; Wed, 9 Jun 93 15:04:38 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21821; Wed, 9 Jun 93 15:04:44 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 9 Jun 1993 15:04:43 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from malloco.ing.puc.cl by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21795; Wed, 9 Jun 93 15:04:34 -0400 Received: by malloco.ing.puc.cl (4.1/5.17) id AA01246; Wed, 9 Jun 93 15:04:28 CST From: ricardo@ing.puc.cl (Ricardo Schmutzer von Oldershausen) Message-Id: <9306091904.AA01246@malloco.ing.puc.cl> Subject: Pointers... MPI_APPEND_POINTER To: mpi-pt2pt@cs.utk.edu Date: Wed, 9 Jun 93 15:04:27 AST X-Mailer: ELM [version 2.3 PL11] Hello ! In spite of the danger of being flamed (how many people are there in this group ...? ;-), I would like to discuss the possibility to include a MPI_APPEND_POINTER function in the MPI standard. I understand, that the main reason for not including it, is because it would be necessary to manage buffer memory allocation under MPI. Therefore a compromise could be to implement a version of MPI_APPEND_POINTER that does _not_ need any memory allocation. The goal of MPI_APPEND_POINTER is basically to simplify the description of user-types in unbound buffers (see MPI_APPEND_ARRAY and MPI_APPEND_UNION in my previous post), so that lists and trees may be sent through MPI without too much hazzle. If MPI developes to a successfull standard (I hope it will !), than sooner or later, someone will want to use MPI to parallize programs written in other languages that use pointers or where pointers are essential (just think in lisp or scheme that are based on list processing). I think that this capability is essential, so that MPI may be embraced by a lot of people. Note that MPI_APPEND_POINTER does not need to be implemented under languages that do not use pointers, as for example Fortran (as noted by Tony Skjellum). How could we send a linked list without MPI_APPEND_POINTER ? struct list { my_datatype data; /* user-type described with an unbound buffer */ struct list *next; /* points to next member of the list, last is NULL */ }; All members of the linked list could be copied to a contiguous buffer (otherwise MPI_APPEND_INDEXED could be used to avoid the copy) and the contiguous buffer is sent. At the receiving end, a buffer large enough to hold all members of the list must be available (through a buffer made with MPI_APPEND_CONTIGUOUS). After the members are received, the pointers of each element is set to point to the next member in the buffer and the pointer of the last member is set to NULL. This isn't very complicated, but sending a list of lists, or a tree can get far more complex. The point here, is that the buffer where the list is received must be allocated in advance. Therefore the amount of elements (or the maximum amount) must be known in advance. A simpler way to send the linked list could be using MPI_APPEND_POINTER and MPI_POINTER_BUFFER defined as follows: MPI_APPEND_POINTER(buffer, start, size, datatype) where: buffer is the MPI_BUFFER, start is the location of the pointer to a datatype, size is the amount of bytes needed by datatype, and datatype is a MPI_BUFFER describing the type where the pointer points to. MPI_POINTER_BUFFER(buffer, start, size) must be called before the buffer is commited and allows to pass a buffer (a piece of memory) starting at start, and of a length of size bytes (start and size are passed by reference, so that they may be modified). When MPI_APPEND_POINTER needs to "allocate memory", it will use the memory passed by MPI_POINTER_BUFFER. After the completion of the reception of the message, start will point after the last byte used and size will reflect the remaining size of the unused memory. So, the above example could be implemented as follows: MPI_BUFFER b, b2; MPI_BUFFER my_datatype; struct list my_list; char *mem; int memsize; MPI_CREATE_BUFFER(&b, MPI_PERSISTENCE); MPI_APPEND_CONTIGUOUS(b, ((list *)0)->data, 1, my_datatype); MPI_APPEND_POINTER(b, ((list *)0)->next, sizeof(struct list), b); MPI_COMMIT(b); MPI_CREATE_BUFFER(&b2, MPI_PERSISTENCE); MPI_APPEND_CONTIGUOUS(b2, my_list, 1, b); memsize = sizeof(struct list) * 100; /* allow to receive 100 elements */ mem = malloc(memsize); /* memory allocated by the user, not MPI */ MPI_POINTER_BUFFER(b2, &mem, &memsize); /* register the memory buffer */ MPI_COMMIT(b2); Note, that this implementation needs no dynamic memory allocation, because the memory used by MPI_APPEND_POINTER is already allocated by the user. In this respect, we do not have any advantage against the version without MPI_APPEND_ POINTER. But at least with MPI_APPEND_POINTER it is much simplier to describe complex structures that use pointers and therefore it is much simplier to send such user-types through MPI. Discussion: Eventually MPI_POINTER_BUFFER could be modified to receive an array of free memory buffers. Note also, that the receive must have been posted before the message is received, so that the buffer is available. A solution to this, could be to pass a "global" buffer to MPI (passing a MPI_NULL pointer to MPI_POINTER_BUFFER). If you ask me, I do not like the restriccions and work-arounds so that MPI does not need to manage the memory allocation by itself, but between not being able to describe structures with pointers under C at all, and a solution similar to the one described here, I'll like the akward solution. Ricardo. -- +----------------------+---------------------------------+--------------+ | Ricardo A. Schmutzer | Internet: ricardo@ing.puc.cl | Conectividad | | PUC, Santiago, Chile | | Distr. Comp. | +----------------------+---------------------------------+--------------+ From owner-mpi-pt2pt@CS.UTK.EDU Thu Jun 10 06:50:13 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA21791; Thu, 10 Jun 93 06:50:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25414; Thu, 10 Jun 93 06:50:30 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 10 Jun 1993 06:50:29 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25394; Thu, 10 Jun 93 06:50:25 -0400 Received: from tycho.co.uk (float.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA20691 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Thu, 10 Jun 1993 11:49:54 +0100 Date: Thu, 10 Jun 1993 11:49:54 +0100 From: James Cownie Message-Id: <199306101049.AA20691@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00289; Thu, 10 Jun 93 11:49:11 BST To: ricardo@ing.puc.cl Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: <9306091904.AA01246@malloco.ing.puc.cl> (ricardo@ing.puc.cl) Subject: Re: Pointers... MPI_APPEND_POINTER Content-Length: 1646 I am nervous about introducing indirections into the buffer description language for the following reasons : - 1) I do not understand how to express the way in which the MPI engine should walk an arbitrarily linked structure. In particular is there a presumption that the structure contains no cycles ? (Or should MPI be keeping track of visited nodes ?) Is there a presumtion about the value of the NULL pointer ? 2) Introducing pointers has very large implementation implications. At present when a buffer is committed the MPI system knows both the full type signature of the message, and the absolute store addresses of each element. As soon as pointers are introduced, both statements cease to be true. (Since the size of the linked structure may be different each time.) I see a strong similarity between sending linked structures and the mark phase of a garbage collector. This is a non-trivial problem to solve when the data structures to be traversed are unknown at the time the traversal code is to be written. Therefore I would not like to adopt these proposals in the current round of MPI. I would however be interested to see further work on such a system, so that we could better judge the costs and benefits in later MPI rounds. (I hope that such additions can be made compatible with the then existing MPI standard). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jun 10 07:49:40 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA21834; Thu, 10 Jun 93 07:49:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28217; Thu, 10 Jun 93 07:46:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 10 Jun 1993 07:46:22 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28209; Thu, 10 Jun 93 07:46:19 -0400 Via: uk.ac.southampton.ecs; Thu, 10 Jun 1993 12:44:10 +0100 Via: brewery.ecs.soton.ac.uk; Thu, 10 Jun 93 12:36:19 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 10 Jun 93 12:45:42 BST Date: Thu, 10 Jun 93 12:45:43 BST Message-Id: <3510.9306101145@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: minor glitch in section 2.5.2 (buffer operations) Hi, Perhaps this has been picked up on already, but I just noticed that the parameter list of MPI_APPEND_VEC appears to be missing a "start" argument, on page 14 of the full draft Marc sent out on May 28. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Thu Jun 10 09:09:21 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA22218; Thu, 10 Jun 93 09:09:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02834; Thu, 10 Jun 93 09:06:38 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 10 Jun 1993 09:06:37 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02821; Thu, 10 Jun 93 09:06:34 -0400 Via: uk.ac.southampton.ecs; Thu, 10 Jun 1993 14:05:53 +0100 Via: brewery.ecs.soton.ac.uk; Thu, 10 Jun 93 13:58:07 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 10 Jun 93 14:07:32 BST Date: Thu, 10 Jun 93 14:07:33 BST Message-Id: <3573.9306101307@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: another glitch - section 2.5.1 Hi, there seems to be another inconsistency in section 2.5, this time in 2.5.1. At the bottom of page 11 it is stated that Vector can be thought of as a special case of indexed in which the indices are generated with constant stride. However, as far as I can see, the definition of Indexed has no "lenblk" parameter, and each index value points to only one element. So, only Vector operations with lenblk==1 are a special case of Indexed. Or am I missing something? Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Fri Jun 11 08:03:06 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA00674; Fri, 11 Jun 93 08:03:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06863; Fri, 11 Jun 93 08:00:00 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 11 Jun 1993 07:59:59 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06855; Fri, 11 Jun 93 07:59:52 -0400 Via: uk.ac.southampton.ecs; Fri, 11 Jun 1993 12:59:45 +0100 Via: brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 12:52:00 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 13:01:27 BST Date: Fri, 11 Jun 93 13:01:28 BST Message-Id: <4584.9306111201@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: return status handles Hi, I've got another picky question. Reading section 2.8.3 (Communication completion) I notice that MPI_WAIT has its return handle as an IN parameter, although the text says that the call "returns a handle". More confusingly, MPI_STATUS has an OUT return_handle. Is this intentional, and if so what's the motivation? Ian From owner-mpi-pt2pt@CS.UTK.EDU Fri Jun 11 11:56:19 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02009; Fri, 11 Jun 93 11:56:19 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22863; Fri, 11 Jun 93 11:51:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 11 Jun 1993 11:51:32 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22853; Fri, 11 Jun 93 11:51:29 -0400 Via: uk.ac.southampton.ecs; Fri, 11 Jun 1993 16:07:53 +0100 Via: brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 15:46:32 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 15:55:57 BST Date: Fri, 11 Jun 93 15:55:59 BST Message-Id: <4718.9306111455@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: a few more things to fix in the draft Hi, as you may gather, I'm going over the draft rather closely at the moment. I notice that 2.9 and 2.10 still use the old form of MPI_INIT_SEND, with MPI_STANDARD, MPI_SECURE etc. parameters, rather than MPI_INIT_SSEND etc. Likewise, 2.11 still uses MPI_APPEND rather than MPI_APPEND_CONTIGUOUS etc. Ian From owner-mpi-pt2pt@CS.UTK.EDU Fri Jun 11 13:18:51 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02284; Fri, 11 Jun 93 13:18:51 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28165; Fri, 11 Jun 93 13:18:57 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 11 Jun 1993 13:18:56 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28154; Fri, 11 Jun 93 13:18:54 -0400 Via: uk.ac.southampton.ecs; Fri, 11 Jun 1993 17:30:26 +0100 Via: brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 17:22:40 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Fri, 11 Jun 93 17:32:06 BST Date: Fri, 11 Jun 93 17:32:08 BST Message-Id: <4808.9306111632@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: yet another glitch - sect 1.11 Hi, in section 1.11, the expanded versions of the MPI_xSENDC routines have a "count" parameter to MPI_SEND, which is not required. Ian PS - Sorry to keep bothing you all with little details, but at least it acts as a reminder to fix them. I'll try to save up any more problems I find, and post them together in one message. From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 05:35:30 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22267; Mon, 14 Jun 93 05:35:30 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19748; Mon, 14 Jun 93 05:31:38 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 05:31:37 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19740; Mon, 14 Jun 93 05:31:34 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA11179 (5.65c/IDA-1.4.4 for ); Mon, 14 Jun 1993 11:30:29 +0200 Received: by f1neuman.gmd.de id AA16869; Mon, 14 Jun 1993 11:31:13 GMT Date: Mon, 14 Jun 1993 11:31:13 GMT From: Rolf.Hempel@gmd.de Message-Id: <9306141131.AA16869@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: send-receive Cc: gmap10@f1neuman.gmd.de On May 28th David Walker started up a discussion on send-receive operations. Except for one reply by Lyndon Clarke there was not much feedback. I think we have to have a discussion on this topic, so I give it another push with this note. One application of the sendrecv operations is to replace the shift operation by sendrecv plus a topology routine which provides each process with the source and destination id to be used in the sendrecv. An important special case is when for a given process either the source or the destination process does not exist. In the case of an end-off shift this happens if data is shifted away from or towards a boundary process, respectively. I think it would be good if in this case the boundary process could nonetheless use sendrecv instead of dealing with the special case explicitely. For this to work properly we would have to introduce an MPI constant like MPI_NOPID which could be used as source or destination in sendrecv. Any comments? As to the number of different sendrecv functions, David wrote: > I also suggest that we have separate routines for contiguous and > noncontigous buffers (as for send and receive routines), and separate > routines for when the input and output buffers are indentical. Thus: > > MPI_SENDRECV > MPI_SENDRECVC > MPI_EXCHANGE > MPI_EXCHANGEC > > so there are 4 routines in all. I agree with this scheme, except that I think that non-blocking versions would make sense, too. As an example take some grid-based computation, the grid being split up among the parallel processes. At regular intervals you have to update grid values in some overlap areas, and therefore you have to exchange messages with all your neighbors. Using the non-blocking version of the sendrecv all these operations can be started at the same time, thus minimizing waiting times due to bad process synchronization. Of course we all fear the exponential explosion of function numbers, but I think we could live with eight sendrecvs. Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 09:55:20 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23344; Mon, 14 Jun 93 09:55:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04746; Mon, 14 Jun 93 09:54:38 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 09:54:37 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04737; Mon, 14 Jun 93 09:54:35 -0400 Received: from b125.super.org by super (4.1/SMI-4.1) id AA15205; Mon, 14 Jun 93 09:54:39 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA00253; Mon, 14 Jun 93 09:54:38 EDT Date: Mon, 14 Jun 93 09:54:38 EDT From: lederman@super.org (Steve Huss-Lederman) Message-Id: <9306141354.AA00253@b125.super.org> To: mpi-pt2pt@cs.utk.edu Cc: gmap10@f1neuman.gmd.de In-Reply-To: Rolf.Hempel@gmd.de's message of Mon, 14 Jun 1993 11:31:13 GMT <9306141131.AA16869@f1neuman.gmd.de> Subject: send-receive I am trying to reconstruct what happened at the last meeting. I recall that both the topology and collective communications groups met together (the first afternoon?) and discussed this issue. I thought that we decided that we should include it (is my memory correct?). I also thought that one of the points of discussion was where such a function belonged. It is somewhat like a pt-2-pt but having topology info was nice. If we include topology info then you don't need the old EOSHIFT too. A send-receive function also allows for a permutation of data on nodes in an easier way. In summary, I thought the final thoughts were to add it to pt-2-pt and give the actually sender/receiver with a special value if it is a no send. Then there would be topology functions to generate the input values when the send-receive was to be based on topology information. I agree that we should discuss this and finalize our thoughts. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 15:26:12 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA25914; Mon, 14 Jun 93 15:26:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27530; Mon, 14 Jun 93 15:25:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 15:25:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27512; Mon, 14 Jun 93 15:25:08 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA21634; Mon, 14 Jun 1993 15:25:23 -0400 Date: Mon, 14 Jun 1993 15:25:23 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9306141925.AA21634@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Envelope IMHO, the word "envelope" should not be used (as a noun) anywhere in the specification of the MPI draft. MPI does not define an envelope data object, so an envelope is just an unecessary abstraction. David From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 16:05:52 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA26641; Mon, 14 Jun 93 16:05:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29986; Mon, 14 Jun 93 16:05:47 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 16:05:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29959; Mon, 14 Jun 93 16:05:42 -0400 Via: uk.ac.southampton.ecs; Mon, 14 Jun 1993 21:05:21 +0100 Via: brewery.ecs.soton.ac.uk; Mon, 14 Jun 93 20:57:31 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Mon, 14 Jun 93 21:07:00 BST Date: Mon, 14 Jun 93 21:07:03 BST Message-Id: <8358.9306142007@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: Envelope > From: walker@gov.ornl.epm.rios2 (David Walker) > IMHO, the word "envelope" should not be used (as a noun) anywhere in > the specification of the MPI draft. MPI does not define an envelope data > object, so an envelope is just an unecessary abstraction. Hmmm, maybe. I think we definitely need some sort of collective name for those fields in the message that are not the data, though. Otherwise the description of matching, in receive criteria (sect. 2.6) would be messy, for example. However, perhaps the connotations of containment and objectness, which the word "envelope" carries, make it an inappropriate one to use in this context. People often speak of a "message header" - maybe we should just have a header, and data, where the header contains source, destination, tag and context fields? Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 16:22:11 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA27770; Mon, 14 Jun 93 16:22:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01174; Mon, 14 Jun 93 16:22:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 16:22:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01166; Mon, 14 Jun 93 16:22:07 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA07439; Mon, 14 Jun 93 16:22:16 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA20647; Mon, 14 Jun 93 16:22:22 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Mon, 14 Jun 93 16:22:12 -0400 Message-Id: <9306142022.AA10623@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA10623; Mon, 14 Jun 93 16:22:12 -0400 To: mpi-pt2pt@cs.utk.edu Subject: Re: Envelope >> From: walker@gov.ornl.epm.rios2 (David Walker) > >> IMHO, the word "envelope" should not be used (as a noun) anywhere in >> the specification of the MPI draft. MPI does not define an envelope data >> object, so an envelope is just an unecessary abstraction. > >Hmmm, maybe. I think we definitely need some sort of collective name for >those fields in the message that are not the data, though. > .... People often speak of a "message header" - maybe we should >just have a header, and data, where the header contains source, destination, >tag and context fields? I second that we rename our "envelopes" the more commonly used terms "message headers", "header info.", and "header field", wherever appropriate. I remember finding the term "envelope" a little confusing myself the first time I heard it used -- which indeed was in connection with MPI. Anne ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] School of Electrical Engineering | (607) 272-2986 [home/answ.mach] Cornell University, 351 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 14 17:45:54 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA28922; Mon, 14 Jun 93 17:45:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06753; Mon, 14 Jun 93 17:45:38 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Jun 1993 17:45:37 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06736; Mon, 14 Jun 93 17:45:35 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA20953; Mon, 14 Jun 93 16:44:54 CDT Date: Mon, 14 Jun 93 16:44:54 CDT From: Tony Skjellum Message-Id: <9306142144.AA20953@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, elster@cs.cornell.edu Subject: Re: Envelope All this stuff about "envelopes." I think it needs to be a noun, because we need to talk about the kind of information that we expect to be available when a message is accepted. Excluding with from discussion is detrimental. So, though we do not structure the envelopes specifically in MPI, they need to exist, and we need to say what's in them (eg, source information, tag information, context, etc). I think we are artificially hamstrung by David's original suggestion. Calling them message headers or envelopes makes little difference to me, in the long-run. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Tue Jun 15 08:20:11 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03649; Tue, 15 Jun 93 08:20:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03485; Tue, 15 Jun 93 08:19:45 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 15 Jun 1993 08:19:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03477; Tue, 15 Jun 93 08:19:43 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13268; Tue, 15 Jun 1993 08:20:00 -0400 Message-Id: <9306151220.AA13268@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Envelope Date: Tue, 15 Jun 93 08:20:00 -0500 From: David W. Walker On reflection perhaps I was being too picky in objecting to "envelope". However, I think its important in a specification document to clearly distinguish something that is a part of the standard, and something that is being defined purely for convenience of exposition. Also I think that envelope isn't quite the right term. After all it swhat's written on the envelope that's important, not the envelope itself. David From owner-mpi-pt2pt@CS.UTK.EDU Tue Jun 15 10:53:56 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA05505; Tue, 15 Jun 93 10:53:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13087; Tue, 15 Jun 93 10:53:46 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 15 Jun 1993 10:53:46 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13079; Tue, 15 Jun 93 10:53:45 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13542; Tue, 15 Jun 1993 10:54:02 -0400 Message-Id: <9306151454.AA13542@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: mpi_statusall Date: Tue, 15 Jun 93 10:54:02 -0500 From: David W. Walker The current version of the draft does not have an mpi_statusall routine, i.e., a routine that returns a flag as true if all the communication operations in a list are complete, and returns it as false otherwise. Was such a routine ever discussed? Are there compelling reasons for including or excluding it? David From owner-mpi-pt2pt@CS.UTK.EDU Tue Jun 15 11:14:50 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA05692; Tue, 15 Jun 93 11:14:50 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14373; Tue, 15 Jun 93 11:13:48 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 15 Jun 1993 11:13:47 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14244; Tue, 15 Jun 93 11:13:28 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA00227 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 15 Jun 1993 16:13:31 +0100 Date: Tue, 15 Jun 1993 16:13:31 +0100 From: James Cownie Message-Id: <199306151513.AA00227@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA01202; Tue, 15 Jun 93 16:12:38 BST To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: <9306151454.AA13542@rios2.epm.ornl.gov> (message from David W. Walker on Tue, 15 Jun 93 10:54:02 -0500) Subject: Re: mpi_statusall Content-Length: 625 > Was such a routine ever discussed? Not that I can remember. > Are there compelling reasons for including or excluding it? Reasons which spring to mind (I'll let you judge their compulsion level) For : symettry with mpi_wait, mpi_waitall implementation efficiency. (One call instead of many). Against: you can do it yourself with a loop. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jun 15 17:53:13 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA09552; Tue, 15 Jun 93 17:53:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09887; Tue, 15 Jun 93 17:53:07 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 15 Jun 1993 17:53:07 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09879; Tue, 15 Jun 93 17:53:05 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA16558; Tue, 15 Jun 93 21:53:09 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01208; Tue, 15 Jun 93 15:51:31 MDT Date: Tue, 15 Jun 93 15:51:31 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9306152151.AA01208@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: send-receive Cc: Rolf.Hempel@gmd.de > I agree with this scheme, except that I think that non-blocking versions > would make sense, too. As an example take some grid-based computation, > the grid being split up among the parallel processes. At regular > intervals you have to update grid values in some overlap areas, and > therefore you have to exchange messages with all your neighbors. > Using the non-blocking version of the sendrecv all these operations > can be started at the same time, thus minimizing waiting times due > to bad process synchronization. > > Rolf Hempel I agree. We have applications that will benefit from non-blocking versions of send-recv. Tom Henderson From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 21 10:05:47 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19070; Mon, 21 Jun 93 10:05:47 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04519; Mon, 21 Jun 93 10:04:51 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 21 Jun 1993 10:04:50 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04511; Mon, 21 Jun 93 10:04:48 -0400 Message-Id: <9306211404.AA04511@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0401; Sun, 20 Jun 93 17:33:43 EDT Date: Sun, 20 Jun 93 17:29:07 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: postscript file pt2pt chapter draft sent in separate message Reply-To: SNIR@watson.ibm.com Corrected the MPI_CANCEL section to fit with the vote at last meeting (still - there some open questions there that we need to discuss) Included the proposal of Jim Cownie for using buffer descriptors as datatype. This needs to be seriously discussed. Added send-receive and exchange, which we also need to discuss. From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 21 10:05:52 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19076; Mon, 21 Jun 93 10:05:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04520; Mon, 21 Jun 93 10:04:51 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 21 Jun 1993 10:04:48 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04378; Mon, 21 Jun 93 10:04:23 -0400 Message-Id: <9306211404.AA04378@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0397; Sun, 20 Jun 93 17:28:41 EDT Date: Sun, 20 Jun 93 17:28:40 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V5.DVI.* %%Pages: 37 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 4 df<12041206A2EAC660EAF7E0EA3F80EA1F 00EA3F80EAF7E0EAC660EA0600A212040B0D7E8D11>3 D E /Fb 2 63 df<127012F8A3127005 057D840C>58 D<12E01278121EEA0780EA01E0EA0078131EEB0780EB01E0EB0078141EEC0780A2 EC1E001478EB01E0EB0780011EC7FC1378EA01E0EA0780001EC8FC127812E019187D9520>62 D E /Fc 3 107 df15 D<12C012F0123C120FEA03C0EA00F0133C130FEB03C0EB00F0143C140FEC0380EC 0F00143C14F0EB03C0010FC7FC133C13F0EA03C0000FC8FC123C12F012C0C9FCA7B61280A21922 7D9920>21 D<12C0B3B3A502297B9E0C>106 D E /Fd 51 122 df40 D<12E07E12787E7E7E7E13801203A213C01201A712031380A2120713005A121E5A5A5A5A0A 1D7D9914>I<1238127C127EA2123E121EA2127C12F81260070A7A8414>44 D<127012F8A312700505798414>46 D48 D<12035AA25A123FB4FC 12F712471207ACEAFFF0A30C177C9614>I51 D<127012F8A312701200A6127012F8A312700510798F14>58 D61 D65 DIIIIII73 D76 D<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A213 03A538FE0FE0A31317809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136E A4EAFE3EA311177F9614>III82 DII<38FF8FF8A3381C01C0AE EA1E03000E1380EA0F073807FF006C5AEA00F81517819614>I<38FE0FE0A338380380A5001813 0013E3EA1DF7A213B7A5EA0DB6A21316EA0F1EA31317809614>87 D<38FE3F80A3383C1E00EA1C 1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>89 DII93 D97 D<12FCA3121CA413FCEA1FFE7FEB0780EA1E03 001C13C01301A4EA1E03EB0780EA1F0FEBFF005BEA1CF81217809614>II<137EA3130E A4EA07CEEA1FFE123FEA7C3EEA781EEAF00E12E0A4EAF01EEA703EEA787E383FFFC0EA1FEFEA07 8F12177F9614>II<133C13FE1201EA03DE138C1380A2EAFFFEA3EA0380 AAEA7FFCA30F177F9614>II<12FCA3121CA413F8EA1FFE7F130FEA1E07121CA838FF9FE0A313178096 14>I<1203EA0780A2EA0300C7FCA4EAFF80A31203AAEAFFFEA30F187D9714>I<12FCA3121CA438 1DFF80A3381C78005BEA1DE0EA1FC07FA2EA1EF0EA1C7013787F38FF3F80A311177F9614>107 DI110 DII<38FF1F80EB7FC013FF38 07E080EBC0005BA290C7FCA6EAFFFCA31210808F14>114 DI<1207A5 EAFFFCA3EA0700A6130EA3139E13FCEA03F8EA00E00F157F9414>II<38FF7F80A3381C1C00A36C5AA3EA0630EA0770A36C5AA311107F 8F14>I<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C5AA35BA3EA7B80127F90C7 FC127E123C11187F8F14>121 D E /Fe 7 86 df67 DI<12FEA21238AA12FEA2070E7F8D0B>73 D79 D83 D<387FFF80A2EA60E138C0E0C0A200001300A7EA07FCA2120E7F8D16>I<38FE0F C0A238380300A9EA1C06EA0FFCEA03F8120E7E8D18>I E /Ff 14 86 df<1230127812F8127812 7005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA01C0A6EA0380A61207EAFF F8A20D1C7C9B15>49 D<14301470A214F0801301A2EB0378A21306147CEB0C3CA21318A21330A2 497EA2EBFFFEA23801801EA23803001F801206120F397FC0FFF012FF1C1D7F9C1F>65 D<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E1300001E14C05AA2 5A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E06CB45AD8007EC7FC 1C1E7C9C1E>67 D<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B13 C113C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>69 D<390FFF9FFE143F3900F003C0A53901E00780A590B5FC481400EBC00FA53807801EA6000F133E 39FFF3FFE015C01F1C7E9B1F>72 D<380FFF80A23800F000A5485AA6485AA6485AA6120FEAFFF8 A2111C7F9B0F>I77 D<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214 811387A2EB83C1390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13 C01F1C7E9B1F>II<380FFFFC14FF3900F00F80140315C01401A23801E003 A315801407EC0F003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<38 0FFFF814FE3900F00F80140315C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880 A2143E3807803CA51518000F1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 D<001FB512F05A383C07810038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6137C 381FFFE0A21C1C7C9B1E>84 D<39FFF8FFE0A2390F001E00140CA4001E5BA6485BA6485BA30038 5BA2383C0180D81C03C7FCEA0E0EEA07FCEA01F01B1D7A9B1F>I E /Fg 17 122 df97 D99 DI< EA01E0EA07F8EA0E18121C123812701370EA7FE0EAFF80EAE000A41310EA6038EA70F0EA3FC0EA 1F000D127B9113>II104 D<13C0EA01E013C01380C7FCA6120E123FEA33801263EAC700A21207120EA35A13 40EA38C0A3EA3980EA3F00121E0B1C7D9B0D>I108 D<391E1F07C0393F3F8FE0396761D8703863C1F038C7 81E0A2380701C0A2390E0380E0A3EC81C2391C0701C6A2EC038CA239380E01F839180600F01F12 7D9122>IIII114 DI<13C01201A3EA0380A4EAFFE0A2EA 0700A2120EA45AA31320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I118 D<380F0180EA1F8312331263EB870012C3EA0707A2EA0E0EA4EA0C1C121C120CEA0E3CEA0FF812 03EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7FC111A7D9114>121 D E /Fh 42 124 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80FC01387ED 83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<1238127C12FEA3127C 123807077C8610>46 D<13181378EA01F812FFA21201B3A7387FFFE0A213207C9F1C>49 DII<14E013011303A21307130F131FA2 1337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2380007E0A7EBFF FEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018C7FCA4EA19FC EA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8386007E0003013 C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00F05A1478123C12 3E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF8383C1FFCEA7807EB 01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC00001130017207E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F 140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A >65 DII II72 D77 DII< 3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400B4FC13E06CB4 FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA36C131EA26C13 3C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F007814070070 14030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127>I97 DIII<13FE3807FF80 380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E003E13186C13 30380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B51280380F87E7EA1F 03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC121CA2381FFFE0 14F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB45A000113C019 217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I<13 1C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78FCEA7FF0EA1FC0 102E83A311>I108 D<3AFF87F00FE090399FFC3FF8 3A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I< 38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FF C0380F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E038 07FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F 80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A512 07EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01 FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015B A26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<39FFF07FC0A2390FC01C00 6C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F8380381FC 48C67E000E137E39FF81FFE0A21B167F951E>120 D<39FFF01FE0A2390FC00600A2EBE00E0007 130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B 1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>I123 D E /Fi 64 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4B5FCA2EA0E07 AC387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07AC387F9FE0A2 131A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E138190380380C01500A4B6 12E0A2380E0380AC397F8FE3FCA21E1A809920>I34 D<126012F012F812781218A31230127012E01240050B7D 990B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0AA1260A21270 A212301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E121C120C7E12 07A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A267E9B0F>I< 126012F0A212701230A31260A212C01240040B7D830B>44 DI<126012 F0A2126004047D830B>I<130CA2131C1318A213381330A213701360A213E013C0A212011380A2 120313005A1206A2120E120CA2121C1218A212381230A212701260A212E05AA20E257E9B13>I< EA0780EA1FE0EA3870EA3030EA7038EA6018A2EAE01CACEA6018A2EA7038EA3030EA3870EA1FE0 EA07800E1A7E9813>I<1260EA7FFEA213FCEA600CEAC0181330A2EA006013C0A21201A2EA0380 A41207A8EA03000F1A7E9813>55 D57 D<126012F0A212601200A8126012F0A2126004107D8F0B>I<126012F0A212601200A812 6012F0A212701230A31260A212C0124004177D8F0B>I63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47EA2130000061378A3487F121E39 FF81FFC0A21A1B7F9A1D>65 DIIIIII<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39FFF3FFC0A21A1A7F 991D>III<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3EB 7818A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>77 DI<137F3801FFC03807C1 F0380F0078001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C13 1C001E133C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II82 DI<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FCAF3803FFF0A2191A 7F991C>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC5DD807801306EB819E A2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE13F0EBFC03017C5BA2 EB7801A201385BEB3000241B7F9927>87 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2 EB01C0A6EB0380A2381F0F00EA1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C 1EEA700EA212E0A61270131EEA3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA 03BCEA073CEA0E181300A5EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA 3877EA7038A5EA3870EA3FE0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781E EA1FF8EA07E010197F9013>I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A 7F9915>I<1218123CA21218C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00 C01300A6EA07E0A21200B0126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F 80A2EB3C0013305B5B121DEA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12 FCA2121CB3A4EAFF80A2091A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301 AA39FF9FE7F8A21D107F8F20>II< EA07E0EA1FF8EA381CEA700EEA6006EAE007A6EA700EA2EA3C3CEA1FF8EA07E010107F8F13>I< EAFCF8EAFFFEEA1F0F381C07801303EB01C0A6EB03801307381F0F0013FEEA1CF890C7FCA5B47E A212177F8F15>IIII<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B 177F960F>II<38FF3F80A2381C0E 00130CA26C5AA21338EA0730A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00 381C3E0C13361337000E5B13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1F C0A2380E1E00EA071CEA03B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A213 10808F14>I<38FF3F80A2381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA21261 00F3C7FC12C712FE127811177F8F14>III E /Fj 18 118 df<127812FCA412781200A5127812FCA4127806117D900C>58 D63 D68 D73 D77 D97 D99 D101 D<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C 3CEA3FF8EA37E00070C7FCA2EA3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF8038 07FC0013197F9016>103 D<121E123FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>105 D108 D<39FF1F81F890387FE7FE391FE3EE3E903881 F81FA2EB01F0AA3AFFE7FE7FE0A223117F9026>IIII115 D<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03E00D187F9711>I<38FF1FE0 A2EA1F03AB1307130F380FFFFCEA03F316117F9019>I E /Fk 52 122 df<90381FE1F890B512 FC3903F03F3E3807C07EEB807C000F141C1500A5B612C0A2390F807C00AE397FE1FFC0A21F1D80 9C1C>11 D13 D<13301360EA01C0EA038013005A120E121E121C123CA212381278A312F85AA97E12 78A31238123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13>40 D<12C0126012387E120C120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C01203 A2138012071300120E120C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B1203 12071206A2120C121812301220080F7D860D>44 DI63 D65 DI<90381FE0209038FFF8 E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500A7007E1460A27E15C06C 7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>IIII<90380FF0 0890387FFE383901FC07783907E001F8380FC0004848137848C71238A2481418127E12FE1500A5 EC7FFFA2007EEB01F8127F7EA2EA1F80EA0FC0EA07F03801FC0739007FFE7890380FF818201C7D 9B26>I<39FFFC3FFFA2390FC003F0AA90B5FCA2EBC003AC39FFFC3FFFA2201C7E9B25>II76 DI<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0EB0FE0EB07F013 03EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC013701530201C7E 9B25>III82 D<3807F860381FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7FFE6C7E148000 0F13C06C13E0EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00EAC3FC141C7D 9B1B>I<007FB512E0A238781F81007013800060146000E0147000C01430A400001400B03807FF FEA21C1C7E9B21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001C03900FC0780 90387FFE00EB0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D137000031460 6D13E000015C7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0FD8A2EB07F0A3 6D5AA26D5AA2211C7F9B24>I<3BFFFC7FFE0FFCA23B0FC007E000C081D9E003130100071680EC 07F8D803F0EC0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000150CEC387F90397E30 3F18A290397F601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC00036D5CA22E1C7F9B 31>I<397FFE1FFEA23907F001803803F8036C6C48C7FC000013066D5AEB7F1C6D5A14B0EB1FE0 130FA26D7E6D7E1307497EEB0CFEEB187EEB387F496C7EEB601F01C07F00016D7E496C7EEA0300 3AFFF03FFF80A2211C7F9B24>I<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C 5B3800FE015DD97F03C7FCEB3F871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I< EA0FF8EA1FFE383E1F80130714C0121C1200EA03FF121FEA3F87EA7E0712FCA3130FEA7E1F383F FBF8EA0FE115127F9117>97 DIIII<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EA FFF8A2EA0F80AEEA7FF8A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4 003C1300EA3E1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078 A3007813F0383E03E0381FFFC03803FE00151B7F9118>II<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEA FFE0A20B1E7F9D0E>I107 D I<39FF1FC0FE90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F91 28>I<38FF1FC0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7F E0EB1F8090C7FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEA FFFCA2EA1F00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F 03EBFEFFEA03F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803 E300A2EA01F6A213FE6C5AA21378A2133016127F9119>I<39FF8FF8FEA2391F03E030A2018313 70000FEBF0601386D807C613C0EBCEF8EBEC790003EB7D80EBF83D0001EB3F00A2497E0000131E EBE00EA21F127F9122>I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA01FC5B1200137C13 FEEA01DFEA039F38070F80380607C0380C03E038FF07FCA216127F9119>I<38FFC1FCA2381F00 60EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5AA21378A21330A25B1270EA F8E0EAC0C0EAE380007FC7FC123E161A7F9119>I E /Fl 80 126 df<126012F0AF12601200A4 126012F0A212600419779816>33 D<13C01201A3EA07F0EA1FFC48B4FCEA7DCF38F1C78012E1A3 38F1C300EA79C0127FEA1FF0EA07FCEA01FE13CFEBC780EAF1C3A3EAE1C712F13879DF00EA3FFE 6C5AEA07E0EA01C0A2120011207E9C16>36 D38 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E12 70A312781238123C7E7E7E13C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780 120313C01201A313E01200A7120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C 16>III<1238127C127EA2123E 120E121E123C127C12F81260070B798416>II<127012F8A31270050578 8416>IIIIII<137C13FC13DC1201EA039CA2EA071C120F120E121E123C1238127812F0B512E0 A338001C00A53801FFC0A313197F9816>II<13F8EA03FEEA 0FFFEA1F0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2EAF80F38F00780A2EAE00312F0A2 1270EA7807EB0F006C5AEA1FFEEA0FF8EA03E011197E9816>I<12E0B51280A338E00F00131EC6 5A13381378137013F05B12015B12035BA3120790C7FCA7111A7E9916>III<127012F8A312701200A8127012F8A312700512789116> I<1238127CA312381200A81238127CA3123C121C123C123812F812F012600618799116>II I<12E07E12FC123E6C7EEA07E06C7EEA00FC133EEB1F801307131FEB3E0013FCEA03F0485AEA1F 80003EC7FC12FC12F05A11157E9616>I<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2 EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816>65 DI<3801F180EA07FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E3870 03801278A2EA3C07381F0F00EA0FFE6C5AEA01F011197E9816>I I<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E 13197F9816>III<387F1FC038FFBFE038 7F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013197F9816>III76 D<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313 197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377 A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>III82 DI<387FFF E0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F0 381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FE0FE0EAFF1FEA FE0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813B8A3EA01B013F0A26C5A13197F 9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803A313E3EA39F3A213B300191300 A61313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F131F380E1E00131CEA073C1338 EA03B813F012015B120012017F120313B81207131CA2EA0E0EA2487E387F1FC000FF13E0007F13 C013197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C 5AA8EA03F8487E6C5A13197F9816>I<387FFF80B5FCA238E007005B131E131CEA003C5B137013 F0485A5B1203485A90C7FC5A381E0380121C123C12781270B5FCA311197E9816>II93 D95 D97 D<127E12FE127E120EA4 133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF000E1300 EA067C1419809816>II<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F487E1270 EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA3FFE487E 6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5AEA1FFC48 5AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C07C0383F FF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF148013871303A2120E A9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5BEA0FF8A2 13BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870 AA38FE7CF8A31512809116>IIIII<38FF0F80EB3FE013FFEA07F1EBE0C0 EBC0005BA290C7FCA7EAFFFCA313127F9116>II<12035AA4 EA7FFFB5FCA20007C7FCA75BEB0380A2130713873803FF005BEA00F811177F9616>I<387E1F80 EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C13F83803F3F01512809116>I<387F1FC000FF13 E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4EA03B8A3EA01F0A26C5A13127F9116>I<38FF1F E013BF131F38380380A413E33819F300A213B3EA1DB7A4EA0F1EA313127F9116>I<387F1FC013 3F131F380F1C00EA073CEA03B813F012016C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF 3FE0387F1FC013127F9116>I<387F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA 039CA21398EA01B8A2EA00F0A35BA3485A1279127BEA7F806CC7FC123C131B7F9116>I123 D<12E0B3AE0320779C16>I<12FCB4FC13C012031201A97F6CB4FCEB7F80A2EBFF00EA 01E05BA9120312FF90C7FC12FC11207E9C16>I E /Fm 43 122 df<903901FF81FF011F01EF13 C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712F8 A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7FA928>11 D45 D<121C127FEAFF80A5EA7F00121C09097B8813>I48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA6 22>III<14075C5C5C5C5CA25B5B497E130F130E131C 1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D27 7EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80 FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806C EBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C01580A2150038 70001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A1378 1C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F90393FF007 FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503A3484891C7 FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007 F0010FB512C0010391C7FC9038003FF829297CA832>II73 D77 DIII82 D<007FB712C0A3903980 7FC03FD87C00140700781503A20070150100F016E0A2481500A5C71500B3A490B612E0A32B287E A730>84 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC38 07FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E 9A21>97 DIIII<9038FF81F00003EBE7FC390FC1FE7C 391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE00018 13800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A14 00A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14> 105 DIII<3BFF C07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF0013 8001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0 390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF8090 38FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE 0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E38 0FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800 127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0 137C143C7EA26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203 A21207381FFFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B> I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A 26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7 FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3AFFFE03FF80A3 3A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB 1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA 3FE0EA0FC021277F9A24>121 D E /Fn 75 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C 17>II<9038 3F03F09038FFCFF83903C0FC1C390701F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A 7FC7FCFF80A2211D809C23>I<127012F8A71270AC12201200A4127012F8A31270051E7D9D0C> 33 DI< 127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA03001206 120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C120C 120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380A2 1201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<127012 F012F8A212781218A31230A2127012601240050D7D840C>44 DI<1270 12F8A3127005057D840C>I<1303A213071306A2130E130C131C1318A213381330A213701360A2 13E013C0A21201138012031300A25A1206A2120E120CA2121C1218A21238123012701260A212E0 5AA210297E9E15>II<12035A123FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>III<131CA2133C137CA213DC1201139C1203 EA071C1206120E120C121812381230126012E0B512C0A238001C00A63801FFC0A2121C7F9B15> II<13F0EA03FCEA070CEA0E0EEA1C 1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E12F0130FA51270A2130E1238131C EA1C38EA0FF0EA03E0101D7E9B15>I<1260387FFF80A21400EA6003EAC0065BA2C65A5BA25BA2 5BA21201A2485AA41207A76CC7FC111D7E9B15>III<127012F8A312701200A8127012F8A3127005127D910C>I<127012F8A312701200A8 127012F012F8A212781218A31230A2127012601240051A7D910C>I<007FB512C0B612E0C9FCA8 B612E06C14C01B0C7E8F20>61 D63 D<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3 001F131F39FFC0FFF0A21C1D7F9C1F>65 DI<90381F80 80EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070 EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>IIII<90381F8080 EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA5ECFFF0 A20070EB07801278A27E121C121E7E3807800F3803F0393800FFF090381FC0001C1E7E9C21>I< 39FFF3FFC0A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>II76 DIIII82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA 1FFEEA07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F812 1E7E9C17>I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B038 03FFFCA21B1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0 EB7FC0011FC7FC1A1D7E9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA2018014700007 9038678060A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B 13F6017E137EA3013C133CA3011C133801181318281D7F9B2B>87 D<397FF0FFC0A23907C03E00 00031338EBE03000011370EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E013 71EB61F0EBC0F800011378EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I92 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E013 0014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E 00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0 A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>II<12FC A2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9F E0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83 B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFB EA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA 1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F 9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA31312 7F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713CCA2136038 03E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00 F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2 381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC12 7E123C131A7F9116>III E /Fo 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815FE489039001F FF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F01 3FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB 01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF00101 07903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC0113C0 EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5BA212FF AA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB3F8090 3A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fp 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D67 D97 D<903801FFC0011F13F8017F 13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03 FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB01 F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0 5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13 FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07 FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F 13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80 91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D A529>114 D116 D E /Fq 29 120 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D48 DIII57 D69 D<903807F00890383FFC1890 38FC0E383801E0033903C001F83807800048C71278121E15385AA2007C14181278A212F81500A6 EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801F0013900FC079890383FFE089038 07F80020247DA226>71 D<3803FFF0A238000F00B3A6127012F8A3EAF01EEA601CEA3878EA1FF0 EA07C014237EA119>74 D76 DI<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801 38EF038038C7FF00EA81FC14247DA21B>83 D<3BFFF03FFC07FEA23B0F0007C001F00203EB00E0 1760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F0183C06A3903978 383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D486C5AA32F237F A132>87 D97 D99 D<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA38001278127012F0A712701278EA38 01EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A5001C1300EA1E0FEA0F1EEA 1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070481338A4007013700078 13F0381E03C0380FFF803801FC0015217F9518>103 D<121C121E123E121E121CC7FCA8120E12 FEA2121E120EAFEAFFC0A20A227FA10E>105 D<120E12FEA2121E120EAAEB0FFCA2EB07E0EB03 80EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA217 237FA21A>107 D<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF 809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E 1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II<380E1F8038FE7FE038FFC1F0380F0078120E14 3CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FCA8EAFFE0A2171F7F941B> I114 DI<000E137038FE07F0A2EA1E00000E1370AC14F0 1301380703783803FE7FEA01F818157F941B>117 D<39FF8FF9FFA2391E01C07CD81C03133800 0EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D81980141DA2EBF00F000014 00A2497EEB600620157F9423>119 D E /Fr 22 118 df45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>II<1207EA0F80121FA2120FEA0700C7 FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8A20D307EAF12>I<260781FEEB3FC03BFF87FF 80FFF0903A8E07C1C0F83B0F9803E3007C2707B001E6133C9026E000FC7F495BA3495BB3486C48 6C133F3CFFFC1FFF83FFF0A2341F7E9E38>109 D<380781FE39FF87FF8090388E07C0390F9803 E03807B0019038E000F05BA35BB3486C487E3AFFFC1FFF80A2211F7E9E25>II<38 0780FC39FF87FF8090389E07C0390FB803E03907E000F04913F8157C5B153EA3151FA9153EA315 7C6D137815F89038E001F09038B803E090389E0FC0903887FF00EB81FC0180C7FCAB487EEAFFFC A2202D7E9E25>I<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EBC000A35BB3487EEAFFFE A2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A1430A37E6C1300127EEA 3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C7EA27E14186C13386C13 7038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A21207120F121FB512E0A2 3803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>II E /Fs 5 85 df<1630167016F0A21501A21503A21507150FA2151B821531A2156115E1 15C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2ECC000A249B5FC5B91C7FC1306 A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE90383FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E83147E188002FE140FA24A15C0 A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA2011F157E17FE4A5CA2013F4A 5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495AED07E00001EC3F80B600FE C8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7121EA24A140CA2147EA214 FEA25CA20101151CEEC0184A1400A201031301A202F05B15030107130F91B5FC93C7FCECE00F01 0F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA2137EA213FEA25BA21201B512 FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092380003F0EE01F84AEB00FC A2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804AEB3F00167E0107EB03F891 B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA2013F1301A21400A249495AA2 137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C130F121800385CA20030011F 1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FEA25CA21301A25CA21303A2 5CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D3174B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 966 a Fs(D)26 b(R)g(A)f(F)h(T)225 1057 y Fr(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)652 1148 y(P)n(oin)n(t)i(to)g(P)n(oin)n(t)g(c)n(hapter)511 1331 y Fq(Marc)16 b(Snir,)g(William)d(Gropp)18 b(and)e(Ewing)h(Lusk)828 1452 y(June)g(20,)f(1993)p eop %%Page: 1 2 bop 75 356 a Fp(Chapter)32 b(1)75 564 y Fo(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)884 780 y Fn(Marc)15 b(Snir)684 830 y(William)10 b(Gropp)j(and)h(Ewing)f(Lusk)75 967 y Fm(1.1)70 b(In)n(tro)r(duction)75 1058 y Fn(This)15 b(c)o(hapter)h(is)f(a)f(draft)h(of)g(the)g(curren)o(t)i (prop)q(osal)d(for)h(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(uni)o(cation.)k(The) e(MPI)f(pro)q(ce-)75 1108 y(dures)i(are)f(sp)q(eci\014ed)i(using)d(a)h (language)f(indep)q(enden)o(t)i(notation.)24 b(The)16 b(parameters)g(of)f (pro)q(cedure)j(calls)e(are)75 1157 y(mark)o(ed)11 b(as)h Fl(IN)f Fn(or)h Fl(OUT)p Fn(.)f(The)i(call)e(do)q(es)i(not)f(up)q(date)g(a)g (parameter)g(mark)o(ed)e Fl(IN)i Fn(and)g(ma)o(y)e(up)q(date)j(a)e(parameter) 75 1207 y(mark)o(ed)18 b Fl(OUT)p Fn(.)g(If)g(a)h(parameter)f(is)h(a)g (handle)f(to)h(an)g(opaque)g(ob)r(ject,)h(and)f(the)h(ob)r(ject)f(is)g(up)q (dated)h(b)o(y)e(the)75 1257 y(pro)q(cedure)e(call,)d(then)i(the)g(parameter) f(is)g(mark)o(ed)f Fl(OUT)p Fn(.)g(The)h(F)m(ortran)g(77)g(and)g(C)g (bindings)g(are)g(describ)q(ed)i(in)75 1307 y(Chapter)f Fk(??)p Fn(.)158 1357 y(Discussions)f(that)g(are)h(not)e(part)i(of)e(the)h(standard)h (are)f(indicated)g(b)o(y)f(small)f(t)o(yp)q(e.)158 1485 y Fj(Discussion:)38 b Fi(The)13 b(follo)o(wing)j(sections)f(of)f(the)g(in)o(tro)q(duction)i(con)o (tain)f(general)g(information)h(on)e(the)g(design)h(of)75 1531 y(MPI)h(pro)q(cedures.)27 b(The)16 b(material)i(should)g(b)q(e)e(mo)o(v)o(ed) g(to)g(a)g(general)h(in)o(tro)q(duction)i(for)d(the)g(en)o(tire)h(do)q(cumen) o(t.)27 b(The)75 1576 y(actual)14 b(binding)i(of)d(these)g(ob)r(jects)g(in)h (F)m(ortran)g(and)f(C)g(is)g(describ)q(ed)i(in)f(Chapter)f Fj(??)p Fi(.)75 1796 y Fm(1.2)70 b(Data)23 b(T)n(yp)r(es)75 1896 y Fh(1.2.1)55 b(Handle)75 1972 y Fn(MPI)17 b(pro)q(cedures)j(use)e(at)f (v)n(arious)g(places)h Fg(hand)r(les)p Fn(.)28 b(Handles)18 b(are)g(used)g(to)f(access)i(opaque)e(ob)r(jects.)30 b(Suc)o(h)75 2022 y(ob)r(ject)11 b(can)g(b)q(e)g(created,)h(up)q(dated)f(and)g(destro)o(y) o(ed)g(only)f(b)o(y)g(calling)f(suitable)h(MPI)h(pro)q(cedures,)i(and)d(pro)o (viding)75 2072 y(the)k(handle)g(as)g(parameter.)k(A)c(handle)g(to)f(an)h (opaque)g(ob)r(ject)g(is)g(signi\014can)o(t)f(only)g(at)h(the)h(pro)q(cess)g (where)g(the)75 2122 y(ob)r(ject)g(w)o(as)e(created,)i(and)f(cannot)g(b)q(e)h (transferred)g(to)f(another)g(pro)q(cess)i(\(ob)r(jects)f(are)g(lo)q(cal\).) 158 2171 y(Opaque)j(ob)r(jects)g(hide)f(from)f(the)h(user)h(the)g(in)o (ternal)f(represen)o(tation)i(used)f(for)e(v)n(arious)h(MPI)g(ob)r(jects,)75 2221 y(th)o(us)c(allo)o(wing)e(to)h(ha)o(v)o(e)h(similar)d(calls)i(in)h(C)f (and)h(F)m(ortran,)f(allo)o(wing)f(to)h(o)o(v)o(ercome)g(problems)g(with)g (the)i(t)o(yping)75 2271 y(rules)f(in)f(these)i(languages,)d(and)h(allo)o (wing)f(for)h(future)g(extension)i(of)d(their)i(functionalit)o(y)m(.)i(The)e (mec)o(hanism)d(for)75 2321 y(opaque)k(ob)r(jects)h(used)g(here)g(lo)q(osely) e(follo)o(ws)f(the)j(POSIX)f(F)m(ortran)g(binding)f(standard.)158 2371 y(An)i(opaque)h(ob)r(ject)g(can)f(b)q(e)h Fg(p)n(ersistent)f Fn(or)g Fg(ephemer)n(al)p Fn(.)22 b(A)15 b(p)q(ersisten)o(t)i(ob)r(ject)f(p)q (ersists)h(un)o(til)e(destro)o(y)o(ed)75 2421 y(b)o(y)g(an)h(explicit)f(op)q (eration.)23 b(An)16 b(ephemeral)f(ob)r(ject)i(is)e(go)q(o)q(d)h(for)f(a)g (single)h(use;)g(th)o(us,)h(an)e(ephemeral)g(ob)r(ject)75 2470 y(asso)q(ciated)j(with)g(a)f(comm)o(unicatio)o(n)e(op)q(eration)i(disapp)q (ears)i(once)f(this)g(op)q(eration)f(is)h(completed)f(\(or)g(once)75 2520 y(this)d(ob)r(ject)h(is)e(not)h(needed)i(an)o(ymore)c(for)i(the)g (completion)e(of)h(the)i(op)q(eration\).)158 2570 y(Opaque)i(ob)r(ject)g(are) g(created)h(b)o(y)e(calls)g(that)g(are)h(sp)q(eci\014c)g(to)g(eac)o(h)f(ob)r (ject)i(t)o(yp)q(e.)25 b(They)17 b(are)g(destro)o(y)o(ed)75 2620 y(b)o(y)d(a)f(call)h(to)f Fl(MPI)p 363 2620 14 2 v 15 w(FREE)p Fn(.)g(Additional)f(MPI)j(functions)f(are)g(a)o(v)n(ailable)e(to)h (access)j(and)e(up)q(date)h(sp)q(eci\014c)g(opaque)75 2670 y(ob)r(jects.)965 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(2)158 45 y Fk(MPI)p 257 45 15 2 v 17 w(FREE\(handle\))75 161 y(IN)16 b(handle)j Fn(handle)14 b(to)g(ob)r(ject)158 241 y(Mark)c(the)i(ob)r(ject)f(asso)q(ciated)g(with)f(the)i(handle)e(for)g (destruction.)18 b(The)11 b(ob)r(ject)h(can)e(b)q(e)i(destro)o(y)o(ed)f(as)g (so)q(on)75 290 y(as)j(there)h(there)g(is)e(no)h(p)q(ending)g(op)q(eration)f (that)h(is)g(using)f(this)h(ob)r(ject,)g(and)f(the)i(handle)e(b)q(ecomes)h (unde\014ned.)75 340 y(F)m(or)19 b(example,)g(if)f(a)h(comm)o(unication)d(ob) r(ject)k(is)g(freed,)h(after)e(the)h(comm)o(unication)c(started)21 b(but)e(b)q(efore)h(it)75 390 y(completed,)14 b(then)h(the)g(p)q(ending)g(op) q(eration)f(is)g(not)h(a\013ected;)h(one)e(can)h(use)h(the)f(handle)f(in)g (calls)g(to)h Fl(MPI)p 1776 390 14 2 v 15 w(WAIT)75 440 y Fn(or)f Fl(MPI)p 195 440 V 15 w(STATUS)f Fn({)i(one,)f(in)g(fact,)g(m)o(ust)f (complete)h(the)h(comm)o(unicatio)o(n)d(with)i(a)g(successful)i(call)e(to)g Fl(MPI)p 1776 440 V 15 w(WAIT)75 490 y Fn(or)f Fl(MPI)p 194 490 V 15 w(STATUS)p Fn(.)f(Ho)o(w)o(ev)o(er,)i(once)g(the)g(op)q(eration)f (completes,)g(it)g(is)g(erroneous)i(to)e(reuse)i(the)f(freed)g(handle)f(for) 75 539 y(a)h(new)g(comm)o(unicatio)o(n.)158 672 y Fj(Implemen)o(tation)k (note:)54 b Fi(Some)18 b(ob)r(jects,)h(suc)o(h)f(as)g(bu\013er)h(descriptor)g (ob)r(jects,)g(can)f(b)q(e)g(used)h(b)o(y)f(sev)o(eral)75 722 y(distinct)h(concurren)o(t)f(comm)o(unication)h(op)q(erations.)31 b(Th)o(us,)18 b(to)f(\014nd)g(out)g(whether)h(suc)o(h)f(ob)r(ject)g(can)h(b)q (e)f(freed,)g(one)75 772 y(needs)d(a)f(reference)g(coun)o(t)h(of)e(p)q (ending)k(op)q(erations.)1931 861 y Fe(TO)11 b(DISCUSS)158 987 y Fj(Discussion:)33 b Fi(W)m(e)12 b(do)g(not)f(ha)o(v)o(e)h(y)o(et)f(a)h (mec)o(hanism)h(to)e(con)o(trol)h(what)g(storage)g(is)g(allo)q(cated)h(for)e (opaque)i(ob)r(jects.)75 1037 y(Some)g(b)q(eliev)o(e)i(it)f(is)f(desirable)j (to)d(allo)o(w)h(the)f(user)g(to)g(con)o(trol)h(this)g(allo)q(cation.)158 1204 y Fk(MPI)p 257 1204 15 2 v 17 w(ASSOCIA)l(TED\(handle,)g(t)o(yp)q(e\))75 1320 y(IN)i(handle)j Fn(handle)14 b(to)g(ob)r(ject)75 1401 y Fk(OUT)i(t)o(yp)q(e)k Fn(state)158 1481 y(Returns)c(the)f(t)o(yp)q(e)g(of)f (the)i(ob)r(ject)f(the)h(handle)e(is)h(curren)o(tly)h(asso)q(ciated)f(with,)f (if)g(suc)o(h)i(exists.)21 b(Returns)75 1531 y(the)14 b(sp)q(ecial)h(t)o(yp)q (e)f Fl(MPI)p 444 1531 14 2 v 15 w(NULL)f Fn(if)g(the)i(handle)e(is)h(not)g (curren)o(tly)h(asso)q(ciated)f(with)g(an)o(y)f(ob)r(ject.)158 1581 y(MPI)18 b(ma)o(y)e(pro)o(vide)h(prede\014ned)j(opaque)d(ob)r(jects)i (and)f(prede\014ned,)i(static)e(handles)g(to)f(these)i(ob)r(jects.)75 1631 y(Suc)o(h)14 b(ob)r(jects)h(ma)o(y)d(not)i(b)q(e)h(destro)o(y)o(ed.)75 1746 y Fh(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 1823 y Fn(An)d(MPI)g(call) e(ma)o(y)g(need)j(a)e(parameter)g(that)h(is)f(an)g Fg(arr)n(ay)h(of)h(hand)r (les)p Fn(.)23 b(The)17 b(arra)o(y-of-handles)d(is)i(a)f(regular)75 1873 y(arra)o(y)e(with)g(en)o(tries)h(of)f(t)o(yp)q(e)h Fl(handle)e Fn(in)h(consecutiv)o(e)i(initial)c(lo)q(cations)i(in)g(the)h(arra)o(y)m(.)j (Whenev)o(er)d(suc)o(h)g(arra)o(y)75 1923 y(is)g(used,)g(an)g(additional)e Fl(len)h Fn(parameter)g(is)h(required)h(to)e(indicate)h(the)h(n)o(um)o(b)q (er)e(of)g(v)n(alid)f(en)o(tries)j(\(unless)g(this)75 1972 y(n)o(um)o(b)q(er)e(can)h(b)q(e)h(deriv)o(ed)f(otherwise\).)158 2101 y Fj(Discussion:)158 2146 y Fi(The)d(reason)g(for)g(not)g(ha)o(ving)h (self-delimiti)q(ng)i(arra)o(ys)d(is)h(that)f(records)g(are)g(not)g (\014rst-class)h(citizens)g(in)g(F)m(ortran)f(and)75 2192 y(that)h(ha)o(ving) i(an)f(arra)o(y)f(where)h(one)f(en)o(try)h(enco)q(des)g(length,)g(whereas)g (the)f(other)h(en)o(tries)g(enco)q(de)g(handles)h(w)o(as)e(deemed)75 2238 y(to)k(b)q(e)g(to)q(o)g(error-prone.)27 b(This)17 b(line)g(of)f(argumen) o(tation)i(really)g(implies)g(that)e(w)o(e)g(do)g(not)g(view)h(arra)o (y-of-handles)h(as)75 2283 y(an)f(opaque)g(ob)r(ject,)h(and)f(that)f(w)o(e)g (exp)q(ect)h(the)g(user)g(to)f(build)j(these)e(ob)r(jects)f(explicitl)q(y)m (,)k(not)c(via)i(MPI)e(calls.)29 b(The)75 2329 y(alternativ)o(e)15 b(is)f(for)e(arra)o(y-of-handles)k(to)c(b)q(e)i(an)f(opaque)h(ob)r(ject,)f (manipulated)j(via)e(MPI)f(calls.)75 2527 y Fh(1.2.3)55 b(State)75 2604 y Fn(MPI)14 b(pro)q(cedures)i(use)e(at)f(v)n(arious)g(places)h(argumen)o (ts)f(with)g Fg(state)g Fn(t)o(yp)q(es.)19 b(The)14 b(v)n(alues)f(of)g(suc)o (h)h(data)g(t)o(yp)q(e)g(are)75 2654 y(all)f(iden)o(ti\014ed)h(b)o(y)g (names,)e(and)i(no)g(op)q(eration)g(is)f(de\014ned)i(on)f(them.)k(F)m(or)13 b(example,)f(the)j Fl(MPI)p 1589 2654 V 15 w(APPEND)e Fn(routine)75 2704 y(has)h(a)g(state)g(t)o(yp)q(e)h(parameter)e(with)h(v)n(alues)f Fl(MPI)p 864 2704 V 16 w(INTEGER)p Fn(,)e Fl(MPI)p 1123 2704 V 16 w(REAL,)h Fn(etc.)p eop %%Page: 3 4 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(3)75 45 y Fh(1.2.4)55 b(Named)18 b(constan)n(ts)75 122 y Fn(MPI)e(pro)q(cedures)j(sometimes)14 b(assign)i(a)g(sp)q(ecial)g (meaning)e(to)i(a)g(sp)q(ecial)g(v)n(alue)g(of)f(a)h(basic)g(t)o(yp)q(e)h (parameter;)75 172 y(e.g.)i Fl(tag)14 b Fn(is)g(an)g(in)o(teger)g(v)n(alued)g (parameter)g(of)g(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n)f(op)q (erations,)i(with)g(a)g(sp)q(ecial)75 221 y(wildcard)e(v)n(alue,)g Fl(MPI)p 429 221 14 2 v 16 w(ANY)p 511 221 V 15 w(TAG)p Fn(.)f(Suc)o(h)j (parameters)f(will)e(ha)o(v)o(e)i(a)f(range)h(of)g(regular)g(v)n(alues,)f (whic)o(h)h(is)f(a)h(prop)q(er)75 271 y(subrange)f(of)f(the)h(range)g(of)e(v) n(alues)i(of)e(the)i(corresp)q(onding)h(basic)e(t)o(yp)q(e;)i(sp)q(ecial)e(v) n(alues)h(\(suc)o(h)g(as)f Fl(MPI)p 1700 271 V 15 w(ANY)p 1781 271 V 15 w(TAG)p Fn(\))75 321 y(will)j(b)q(e)j(outside)f(the)g(regular)g (range.)24 b(The)16 b(range)g(of)g(regular)f(v)n(alues)h(can)g(b)q(e)g (queried,)h(and)e(sometimes)g(set,)75 371 y(using)f(en)o(vironmen)o(t)e (inquiry)i(or)f(en)o(vironmen)o(t)g(setting)h(functions)g(\(Chapter)h Fk(??)p Fn(\).)158 499 y Fj(Discussion:)158 545 y Fi(Implemen)o(ters)i (should)g(detect,)f(whenev)o(er)g(p)q(ossible,)i(illegal)g(uses)e(of)f(\\sp)q (ecial)j(v)n(alues".)25 b(Th)o(us,)16 b(the)g(use)g(of)f(the)75 590 y Fd(MPI)p 137 590 12 2 v 13 w(ANY)p 210 590 V 13 w(TAG)d Fi(v)n(alue)i(to)f(tag)g(a)g(message)g(sen)o(t)h(will)g(b)q(e)g(\015agged)g (as)f(an)g(error.)75 789 y Fh(1.2.5)55 b(Choice)75 866 y Fn(MPI)15 b(functions)h(sometimes)d(use)j(parameters)f(with)g(a)f Fg(choic)n(e)i Fn(\(or)f(union\))g(data)f(t)o(yp)q(e.)23 b(I.e.,)14 b(distinct)h(calls)g(to) 75 916 y(the)i(same)f(routine)h(ma)o(y)e(pass)i(b)o(y)f(reference)j(actual)d (parameters)h(of)f(di\013eren)o(t)i(t)o(yp)q(es.)27 b(The)17 b(mec)o(hanism)d(for)75 966 y(pro)o(viding)f(suc)o(h)h(parameters)g(will)f (di\013er)h(from)e(language)h(to)h(language.)158 1094 y Fj(Discussion:)158 1140 y Fi(The)i(F)m(ortran)g(77)g(standard)h(sp)q(eci\014es)h(that)e(the)g(t) o(yp)q(e)g(of)g(actual)h(argumen)o(ts)g(need)f(to)g(agree)g(with)h(the)f(t)o (yp)q(e)g(of)75 1185 y(dumm)o(y)g(argumen)o(ts;)i(no)f(construct)f(equiv)n (alen)o(t)i(to)e(C)g(p)q(oin)o(ters)h(is)f(a)o(v)n(ailable.)29 b(Th)o(us,)16 b(it)g(w)o(ould)h(seem)f(that)g(there)g(is)75 1231 y(no)f(standard)h(conforming)g(mec)o(hanism)g(to)e(supp)q(ort)i(c)o (hoice)g(parameters.)22 b(Ho)o(w)o(ev)o(er,)14 b(most)h(F)m(ortran)g (compiler)h(either)75 1277 y(don't)g(c)o(hec)o(k)g(t)o(yp)q(e)g(consistency)i (of)e(calls)h(to)f(external)h(routines,)g(or)f(supp)q(ort)h(a)f(sp)q(ecial)h (mec)o(hanism)h(to)d(link)j(foreign)75 1322 y(\(e.g.,)c(C\))g(routines.)23 b(W)m(e)15 b(accept)g(this)h(nonconformit)o(y)g(with)f(F)m(ortran)g(77)g (standard.)23 b(I.e.,)14 b(w)o(e)g(accept)h(that)g(the)g(same)75 1368 y(routine)f(ma)o(y)g(b)q(e)f(passed)h(an)f(actual)h(parameter)g(of)f(a)g (di\013eren)o(t)h(t)o(yp)q(e)g(at)f(distinct)i(calls.)158 1414 y(Generic)h(routines)g(can)f(b)q(e)g(used)g(in)g(F)m(ortran)g(90)g(to)g(pro)o (vide)h(a)e(standard)i(conforming)g(solution.)24 b(This)16 b(solution)75 1459 y(will)f(b)q(e)e(consisten)o(t)h(with)g(our)f(nonstandard) i(conforming)g(F)m(ortran)e(77)g(solution.)75 1679 y Fm(1.3)70 b(Pro)r(cesses)75 1853 y Fj(Discussion:)18 b Fi(This)c(material)g(b)q(elongs) h(to)e(an)g(in)o(tro)q(duction)j(or)d(en)o(vironmen)o(t)i(section)158 1985 y Fn(An)k(MPI)g(program)e(is)i(executed)h(b)o(y)f(sev)o(eral)g (autonomous)e(pro)q(cesses)22 b(that)c(eac)o(h)i(execute)g(their)f(o)o(wn)75 2035 y(co)q(de,)d(in)f(a)h(MIMD)f(st)o(yle.)23 b(The)16 b(co)q(des)h (executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)h(need)f(not)g(b)q(e)g(iden)o (tical.)22 b(The)16 b(pro)q(cesses)75 2085 y(comm)o(unicate)d(via)h(calls)h (to)g(MPI)g(comm)o(unication)d(primitiv)o(es.)19 b(T)o(ypically)m(,)13 b(eac)o(h)j(pro)q(cessor)h(executes)g(in)e(its)75 2135 y(o)o(wn)f(address)i (space,)f(although)f(shared-memory)f(implem)o(en)o(tations)f(of)i(MPI)h(are)g (p)q(ossible.)20 b(This)14 b(do)q(cumen)o(t)75 2185 y(sp)q(eci\014es)i(the)f (b)q(eha)o(vior)f(of)g(a)g(parallel)g(program)e(assuming)h(that)i(only)e(MPI) i(calls)f(are)h(used)g(for)f(comm)o(unica-)75 2234 y(tion.)j(The)d(in)o (teraction)f(of)g(an)g(MPI)h(program)e(with)h(other)h(p)q(ossible)f(means)g (of)g(comm)o(unicatio)o(n)e(\(e.g.,)h(shared)75 2284 y(memory\))g(is)h(not)h (sp)q(eci\014ed.)158 2334 y(MPI)f(do)q(es)g(not)f(sp)q(ecify)h(the)g (execution)g(mo)q(del)e(for)h(eac)o(h)h(pro)q(cess.)20 b(A)12 b(pro)q(cess)i(can)f(b)q(e)g(sequen)o(tial,)f(or)h(can)75 2384 y(b)q(e)h(m)o(ultithreaded,)f(with)g(threads)i(p)q(ossibly)e(executing)i (concurren)o(tly)m(.)j(Care)c(has)g(b)q(een)h(tak)o(en)f(to)f(mak)o(e)g(MPI) 75 2434 y(\\thread-safe",)h(b)o(y)f(a)o(v)o(oiding)f(the)j(use)f(of)g (implicit)d(global)h(states.)158 2483 y(MPI)h(do)q(es)g(not)f(sp)q(ecify)i (either)f(the)g(in)o(teraction)f(of)g(pro)q(cesses)k(with)c(signals,)f(in)h (a)h(Unix)f(en)o(vironmen)o(t,)f(or)75 2533 y(other)h(ev)o(en)o(ts)h(that)f (do)g(not)g(relate)g(to)g(MPI)g(comm)o(unicatio)o(n.)j(I.e.,)c(signals)g(are) h(not)g(signi\014can)o(t)g(from)e(the)i(view)75 2583 y(p)q(oin)o(t)h(of)g (MPI,)g(and)h(implemen)o(ters)e(should)h(attempt)g(to)g(implemen)o(t)e(MPI)j (so)g(that)f(signals)g(are)h(transparen)o(t:)75 2633 y(an)g(MPI)g(call)f (susp)q(ended)j(b)o(y)d(a)h(signal)f(should)h(resume)g(and)f(complete)h (after)g(the)g(signal)f(is)h(handled.)p eop %%Page: 4 5 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(4)158 45 y(MPI)21 b(do)q(es)g(not)f(pro)o(vide)g(mec)o (hanisms)f(to)h(sp)q(ecify)h(the)g(initial)d(allo)q(cation)h(of)g(pro)q (cesses)k(to)e(an)f(MPI)75 95 y(computation)e(and)i(their)g(binding)f(to)h (ph)o(ysical)f(pro)q(cessors.)38 b(It)20 b(is)f(exp)q(ected)j(that)e(v)o (endors)h(will)d(pro)o(vide)75 145 y(mec)o(hanisms)9 b(to)j(do)f(so)g(either) h(at)g(load)e(time)g(or)i(at)f(run)h(time.)j(Suc)o(h)d(mec)o(hanisms)e(will)g (allo)o(w)f(the)j(sp)q(eci\014cation)75 195 y(of)h(the)h(initial)e(n)o(um)o (b)q(er)h(of)g(required)i(pro)q(cesses,)h(the)e(co)q(de)h(to)e(b)q(e)i (executed)g(b)o(y)f(eac)o(h)g(initial)e(pro)q(cess,)j(and)e(the)75 244 y(allo)q(cation)19 b(of)g(pro)q(cesses)k(to)d(pro)q(cessors.)40 b(Also,)21 b(the)g(curren)o(t)h(prop)q(osal)d(do)q(es)i(not)g(pro)o(vide)f (for)g(dynamic)75 294 y(creation)14 b(or)g(deletion)g(of)g(pro)q(cesses)i (during)e(program)f(execution)h(\(the)h(total)f(n)o(um)o(b)q(er)f(of)g(pro)q (cesses)k(is)d(\014xed\),)75 344 y(although)f(it)g(is)h(in)o(tended)h(to)e(b) q(e)i(consisten)o(t)g(with)e(suc)o(h)i(extension.)k(Finally)m(,)11 b(the)k(curren)o(t)g(prop)q(osal)e(do)q(es)i(not)75 394 y(sp)q(ecify)g(a)g (naming)e(sc)o(heme)i(for)f(pro)q(cesses.)24 b(W)m(e)15 b(prop)q(ose)h(to)e (alw)o(a)o(ys)g(iden)o(tify)g(pro)q(cesses)k(according)d(to)g(their)75 444 y(relativ)o(e)f(rank)g(in)f(a)h(group,)f(so)h(that,)f(e\013ectiv)o(ely)m (,)h(pro)q(cesses)j(are)d(iden)o(ti\014ed)g(b)o(y)g(consecutiv)o(e)h(in)o (tegers.)75 560 y Fh(1.3.1)55 b(Error)18 b(Handling)75 719 y Fj(Discussion:)35 b Fi(This)14 b(section)g(should)h(b)q(e)e(mo)o(v)o(ed)h (to)e(a)h(general)i(in)o(tro)q(duction.)158 852 y Fn(MPI)h(pro)o(vides)h(the) f(user)h(with)f(reliable)g(message)g(transmission:)21 b(A)16 b(message)g(sen)o(t)h(is)f(alw)o(a)o(ys)f(receiv)o(ed)75 901 y(correctly)m(,)20 b(and)f(the)g(user)h(do)q(es)f(not)g(need)g(to)g(c)o(hec)o (k)g(for)g(transmission)e(errors,)k(time-outs,)d(or)h(other)g(error)75 951 y(conditions.)f(In)c(other)g(w)o(ords,)g(MPI)g(do)q(es)h(not)f(pro)o (vide)g(mec)o(hanisms)e(for)h(dealing)g(with)h(failures)f(in)h(the)g(com-)75 1001 y(m)o(unication)9 b(system.)17 b(Where)12 b(the)g(MPI)f(implemen)o (tation)d(is)j(built)f(on)h(an)g(unreliable)g(underlying)g(mec)o(hanism,)75 1051 y(then)16 b(it)f(is)g(the)h(job)f(of)g(the)h(implem)o(en)o(ter)e(of)h (the)h(MPI)f(subsystem)h(to)f(insulate)g(the)h(user)h(from)c(this)j(unrelia-) 75 1101 y(bilit)o(y)m(,)11 b(or)j(to)f(re\015ect)j(unreco)o(v)o(erable)f (errors)g(as)e(global)f(program)g(failures.)18 b(Similarly)10 b(MPI)k(itself)f(pro)o(vides)h(no)75 1150 y(mec)o(hanisms)e(for)h(handling)g (no)q(de)h(failures.)158 1200 y(Of)c(course,)i(MPI)e(programs)f(ma)o(y)f (still)i(b)q(e)h(erroneous.)18 b(A)10 b Fk(program)g(error)f Fn(can)h(o)q(ccur)i(when)e(an)g(MPI)h(call)75 1250 y(is)i(called)g(with)g(an) g(incorrect)h(parameter)f(\(non-existing)g(destination)g(in)f(a)h(send)h(op)q (eration,)f(bu\013er)h(to)q(o)f(small)75 1300 y(in)k(a)f(receiv)o(e)j(op)q (eration,)e(etc.\))29 b(This)17 b(t)o(yp)q(e)g(of)g(error)h(w)o(ould)e(o)q (ccur)i(in)f(an)o(y)f(implemen)o(tation.)24 b(In)18 b(addition,)75 1350 y(a)f Fk(resource)i(error)e Fn(ma)o(y)f(o)q(ccur)i(when)h(a)e(program)f (exceeds)k(the)e(amoun)o(t)e(of)i(a)o(v)n(ailable)d(system)j(resources)75 1400 y(\(n)o(um)o(b)q(er)12 b(of)f(p)q(ending)i(messages,)f(system)g (bu\013ers,)i(etc.\).)k(The)12 b(o)q(ccurrence)k(of)11 b(this)i(t)o(yp)q(e)f (of)g(error)h(dep)q(ends)h(on)75 1449 y(the)j(amoun)o(t)d(of)i(a)o(v)n (ailable)d(resources)19 b(in)d(the)g(system)g(and)g(the)h(resource)h(allo)q (cation)c(mec)o(hanism)g(used;)k(this)75 1499 y(ma)o(y)13 b(di\013er)j(from)d (system)i(to)h(system.)21 b(The)16 b(recommended)e(implemen)o(tation)e(will)i (pro)o(vide)h(generous)h(limits)75 1549 y(on)h(some)f(of)h(the)g(resources)j (so)d(as)g(to)g(alleviate)f(the)i(p)q(ortabilit)o(y)e(problem)g(this)h (represen)o(ts.)30 b(One)18 b(can)g(also)75 1599 y(write)c Fk(safe)g Fn(programs,)e(that)i(are)g(not)g(sub)r(ject)h(to)f(resource)i (errors.)158 1649 y(All)e(MPI)h(calls)f(return)i(an)e(error)i(parameter)e (that)h(indicates)g(successful)h(completion)d(of)h(the)i(op)q(eration,)75 1698 y(or)g(the)g(error)h(condition)e(that)g(o)q(ccurred,)j(otherwise.)24 b(Ho)o(w)o(ev)o(er,)17 b(MPI)f(do)q(es)g(not)g(require)g(the)h(user)f(to)g(c) o(hec)o(k)75 1748 y(error)h(parameters.)23 b(It)16 b(is)g(recommended)e(to)i (pro)o(vide)f(an)h(implemen)o(tatio)o(n)d(where)k(the)f(en)o(vironmen)o(t)f (for)g(an)75 1798 y(MPI)f(execution)g(can)g(b)q(e)g(set)h(so)e(that)h(an)o(y) f(error)h(encoun)o(tered)i(in)d(an)h(MPI)f(call)g(will)f(cause)j(a)e(program) f(to)h(fail)75 1848 y(with)h(a)h(suitable)f(error)i(message)e(returned)j(to)d (the)i(user.)21 b(The)15 b(en)o(vironmen)o(t)f(for)g(an)h(MPI)g(execution)g (can)g(b)q(e)75 1898 y(also)d(set)j(so)e(that)g(\(nonfatal\))f(errors)i(do)f (not)g(cause)h(a)f(program)f(to)h(fail,)e(and)i(the)h(user)g(can)f(pro)o (vide)g(his)g(or)g(her)75 1947 y(o)o(wn)g(error)i(reco)o(v)o(ery)g(co)q(de.) 158 1997 y(MPI)g(calls)g(ma)o(y)e(initiate)h(op)q(erations)h(that)g(con)o (tin)o(ue)g(async)o(hronously)g(after)g(the)h(call)e(returned.)23 b(Th)o(us,)75 2047 y(the)d(op)q(eration)e(ma)o(y)f(return)k(with)d(a)h(co)q (de)h(indicating)d(successful)k(completion,)d(y)o(et)h(later)g(cause)h(an)f (error)75 2097 y(exception)d(to)f(b)q(e)g(raised.)22 b(If)15 b(there)h(is)f(a)g(subsequen)o(t)i(call)d(that)h(relates)h(to)f(the)h(same)e (op)q(eration)h(\(e.g.,)f(a)h(call)75 2147 y(that)i(v)o(eri\014es)i(that)e (an)g(async)o(hronous)i(op)q(eration)e(has)h(completed\))f(then)h(the)g (error)g(parameter)f(asso)q(ciated)75 2197 y(with)e(this)g(call)g(will)f(b)q (e)i(used)g(to)f(indicate)h(the)g(nature)g(of)e(the)i(error.)24 b(In)15 b(a)g(few)h(cases,)g(the)g(error)g(ma)o(y)e(o)q(ccur)75 2246 y(after)h(all)f(calls)h(that)g(relate)h(to)e(the)i(op)q(eration)f(ha)o (v)o(e)g(completed,)f(so)h(that)h(no)e(error)i(parameter)f(can)g(b)q(e)h (used)75 2296 y(to)e(indicate)h(the)g(nature)g(of)f(the)h(error)g(\(e.g.,)f (an)g(error)h(in)f(a)g(send)i(with)e(the)h(ready)g(mo)q(de\).)k(Suc)o(h)c (error)g(m)o(ust)75 2346 y(b)q(e)g(treated)g(as)f(fatal,)e(since)j (information)10 b(cannot)15 b(b)q(e)f(returned)i(for)d(the)i(user)g(to)e (reco)o(v)o(er)i(from)e(it.)158 2478 y Fj(Discussion:)38 b Fi(A)13 b(p)q(ossible)j(mec)o(hanism)f(is)g(to)e(ha)o(v)o(e)i(MPI)e(calls)i (generate)g(a)f(signal)h(that)f(can)g(b)q(e)g(ignored)i(when)75 2528 y(an)e(error)f(is)h(encoun)o(tered.)20 b(A)13 b(more)g(mo)q(dular)i(mec) o(hanism)g(that)f(allo)o(ws)g(to)g(handle)h(an)e(exception)j(within)e(its)g (con)o(text)75 2578 y(is)g(preferred.)p eop %%Page: 5 6 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(5)75 45 y Fm(1.4)70 b(Messages)75 136 y Fn(A)14 b(message)g(consists)g(of)g(an)f Fg(envelop)n(e)i Fn(and)f Fg(data)p Fn(.)75 251 y Fh(1.4.1)55 b(Data)75 328 y Fn(The)17 b(data)f(part)h(of)f(a)g(message)g(consists)h(of)f(a)g(sequence)j(of)d(v)n (alues,)g(eac)o(h)h(of)f(a)g(basic)h(datat)o(yp)q(e)g(in)f(the)h(host)75 377 y(language.)29 b(Th)o(us,)19 b(in)e(F)m(ortran)h(77,)f(a)h(message)f (consists)i(of)e(a)h(sequence)i(of)d(v)n(alues)h(that)g(are)g(eac)o(h)g(of)f (t)o(yp)q(e)75 427 y Fl(INTEGER)p Fn(,)12 b Fl(REAL)p Fn(,)h Fl(DOUBLE)20 b(PRECISION)p Fn(,)12 b Fl(COMPLEX)p Fn(,)g Fl(DOUBLE)20 b(PRECISION)g(COMPLEX)p Fn(,)12 b Fl(LOGICAL)p Fn(,)g(or)i(\(length)g(1\))75 477 y Fl(CHARACTER)p Fn(.)i(A)i(message)f(ma)o(y)f(also)h(con)o(tain)h(a)f(v) n(alue)h(of)f(t)o(yp)q(e)h Fl(BYTE)p Fn(,)f(whic)o(h)h(consists)h(of)e(8)h (binary)f(digits.)75 527 y(A)e(b)o(yte)g(is)g(di\013eren)o(t)h(from)d(a)i(c)o (haracter.)23 b(Di\013eren)o(t)15 b(mac)o(hines)f(ma)o(y)f(ha)o(v)o(e)i (di\013eren)o(t)h(represen)o(tations)h(for)d(the)75 577 y(same)e(c)o (haracter,)i(or)f(ma)o(y)e(use)i(more)f(than)h(one)g(b)o(yte)g(to)g(represen) o(t)i(a)e(c)o(haracter.)19 b(On)13 b(the)g(other)h(hand,)e(a)h(b)o(yte)75 626 y(has)h(the)h(same)e(binary)g(v)n(alue)g(on)h(an)o(y)f(mac)o(hine.)k(A)d (message)g(ma)o(y)d(mix)h(v)n(alues)i(of)f(di\013eren)o(t)i(t)o(yp)q(es.)158 755 y Fj(Missing:)158 800 y Fi(Need)g(to)h(agree)g(on)f(the)h(C)f(t)o(yp)q (es)h(\(including)i(handling)g(of)e(signed/unsign)q(ed\),)i(and)e(on)g(F)m (ortran)g(90)f(t)o(yp)q(es)h(\(in-)75 846 y(cluding)f(handling)h(of)d (kinds\).)75 1044 y Fh(1.4.2)55 b(En)n(v)n(elop)r(e)75 1120 y Fn(The)14 b(follo)o(wing)e(information)f(is)i(asso)q(ciated)i(with)e(eac)o (h)i(message:)75 1197 y Fk(source)20 b Fn(The)14 b(rank)g(of)f(the)i(sending) f(pro)q(cess)h(\(in)o(teger\))75 1277 y Fk(destinatio)o(n)j Fn(The)c(rank)g(of)f(the)i(receiving)f(pro)q(cess)i(\(in)o(teger\))75 1357 y Fk(tag)k Fn(User)15 b(de\014ned)g(\(in)o(teger\))75 1437 y Fk(con)o(text)k Fn(comm)o(unication)11 b(con)o(text)j(\(handle\))158 1514 y(The)g(range)g(of)f(v)n(alid)f(v)n(alues)h(for)g(the)h Fk(source)f Fn(and)g Fk(destination)d Fn(\014elds)k(is)f Fl(0)22 b(...)43 b(n-1)p Fn(,)12 b(where)j Fl(n)e Fn(is)g(the)75 1564 y(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)j(in)c(the)i(sp)q(eci\014ed)h(con)o (text.)k Fl(source)j(=)g(destination)10 b Fn(is)i(allo)o(w)o(ed:)k(a)c(pro)q (cess)i(can)e(send)75 1614 y(a)i(message)f(to)h(itself.)158 1664 y(The)g(ranges)h(of)e(v)n(alid)f(v)n(alues)i(for)f Fl(tag)g Fn(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)k(and)f(can)g(b)q(e)g(found)f (b)o(y)h(calling)e(a)75 1714 y(suitable)i(query)g(function,)f(as)h(describ)q (ed)i(in)d(Chapter)i Fk(??)p Fn(.)158 1763 y(The)f Fl(tag)e Fn(\014eld)i(can)f(b)q(e)h(arbitrarily)e(set)j(b)o(y)e(the)h(application,)d (and)i(can)h(b)q(e)g(used)g(to)f(distinguish)g(di\013eren)o(t)75 1813 y(messages.)158 1863 y Fl(Context)f Fn(should)i(b)q(e)h(a)e(con)o(text)i (shared)g(b)o(y)e(b)q(oth)h(source)h(and)f(destination.)158 1913 y(The)g(actual)g(mec)o(hanism)d(used)k(to)f(asso)q(ciate)g(an)g(en)o(v)o (elop)q(e)g(with)g(a)f(message)h(is)g(implem)o(en)o(tation)d(dep)q(en-)75 1963 y(den)o(t;)h(some)f(of)g(the)i(information)8 b(\(e.g.,)j Fk(sender)f Fn(or)i Fk(receiv)o(er)p Fn(\))e(ma)o(y)g(b)q(e)i(implicit,)d (and)i(need)i(not)f(b)q(e)g(explicitly)75 2012 y(carried)j(b)o(y)e(a)h (message.)158 2145 y Fj(Discussion:)42 b Fi(\\Con)o(text")15 b(has)g(no)o(w)g(b)q(ecome)g(\\comm)o(unicator",)i(but)e(this)g(w)o(ord)g(is) g(akw)o(ard)g(to)g(use.)22 b(Sugges-)75 2195 y(tions?)75 2413 y Fm(1.5)70 b(Comm)n(unication)20 b(Bu\013ers)75 2504 y Fn(The)h(basic)f(p)q (oin)o(t)g(to)g(p)q(oin)o(t)g(comm)o(unicatio)o(n)e(op)q(erations)i(are)h Fk(send)e Fn(and)h Fk(receiv)o(e)p Fn(.)36 b(A)20 b Fk(send)f Fn(op)q(eration)75 2554 y(creates)c(a)d(message;)g(the)i(message)e(data)h(is) f(tak)o(en)h(from)e(the)j Fk(send)f(bu\013er)p Fn(.)j(A)d Fk(receiv)o(e)e Fn(op)q(eration)i(consumes)75 2604 y(a)j(message;)h(the)g(message)f(data)g (is)g(put)h(in)o(to)e(the)i Fk(receiv)o(e)h(bu\013er)p Fn(.)23 b(The)17 b(sp)q(eci\014cation)g(of)f(send)h(or)g(receiv)o(e)75 2654 y(bu\013ers)e(uses)h(the)e(same)f(syn)o(tax.)19 b(The)14 b(syn)o(tax)g(is)g(designed)h(to)f(b)q(e)h(extendable)f(so)h(that)f(new)g(t)o (yp)q(es)h(of)f(bu\013ers)75 2704 y(can)g(added)g(b)o(y)g(v)o(endors.)p eop %%Page: 6 7 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(6)75 45 y Fh(1.5.1)55 b(Comm)n(unication)21 b(bu\013er)d(comp)r(onen)n(ts)75 122 y Fn(A)g(bu\013er)g(consists)g(of)f(a)h (sequence)h(of)e Fk(bu\013er)h(comp)q(onen)o(ts)p Fn(.)27 b(Eac)o(h)18 b(comp)q(onen)o(t)e(consists)j(of)e(a)g(sequence)75 172 y(of)h(\(not)h (necessarily)h(distinct\))f(v)n(ariables)f(of)g(the)i(same)e(basic)h(datat)o (yp)q(e.)33 b(The)19 b(datat)o(yp)q(e)g(of)f(suc)o(h)i(a)e(v)n(ari-)75 221 y(able)c(is)g(sp)q(eci\014ed)i(b)o(y)e(a)g(state)h(parameter.)k(The)c(p)q (ossible)g(v)n(alues)f(for)g(F)m(ortran)g(are)h Fl(MPI)p 1505 221 14 2 v 15 w(INTEGER)p Fn(,)d Fl(MPI)p 1764 221 V 15 w(REAL)p Fn(,)75 271 y Fl(MPI)p 144 271 V 15 w(DOUBLE)p Fn(,)c Fl(MPI)p 377 271 V 15 w(COMPLEX)p Fn(,)g Fl(MPI)p 632 271 V 15 w(DOUBLE)p 779 271 V 14 w(COMPLEX)p Fn(,)g Fl(MPI)p 1033 271 V 15 w(LOGICAL)p Fn(,)f Fl(MPI)p 1287 271 V 16 w(CHARACTER)g Fn(and)j Fl(MPI)p 1651 271 V 15 w(BYTE)p Fn(,)e(corre-)75 321 y(sp)q(onding)14 b(to)h(datat)o(yp)q(es)g Fl(INTEGER)p Fn(,)e Fl(REAL)p Fn(,)g Fl(DOUBLE)21 b(PRECISION)p Fn(,)12 b Fl(COMPLEX)p Fn(,)g Fl(DOUBLE)21 b(PRECISION)f(COMPLEX)p Fn(,)75 371 y Fl(LOGICAL)p Fn(,)d Fl(CHARACTER)h Fn(and)h(to)g(un)o(t)o(yp)q(ed)h Fl(BYTE)p Fn(.)e(The)h(p)q(ossible)h(v)n (alues)f(for)g(C)g(are)g Fl(MPI)p 1543 371 V 16 w(CHAR)p Fn(,)e Fl(MPI)p 1742 371 V 16 w(SHORT)p Fn(,)75 421 y Fl(MPI)p 144 421 V 15 w(INT)p Fn(,)c Fl(MPI)p 316 421 V 15 w(LONG)p Fn(,)f Fl(MPI)p 509 421 V 15 w(FLOAT)p Fn(,)h Fl(MPI)p 725 421 V 15 w(DOUBLE)f Fn(and)i Fl(MPI)p 1031 421 V 15 w(BYTE)p Fn(.)158 549 y Fj(Discussion:)158 595 y Fi(Ma)o(y)k(need)g(to)g(tak)o(e)g(care)g(of)f (signed/unsi)q(gned)k(in)d(C.)f(More)h(elab)q(orate)i(de\014nitions)g(b)q (elong)f(to)f(the)g(language)75 640 y(binding.)158 773 y Fn(There)d(are)f (\014v)o(e)g(kinds)g(of)g(bu\013er)g(comp)q(onen)o(ts:)75 879 y Fk(Con)o(tiguous)f(comp)q(onen)o(t)75 955 y Fn(A)h(sequence)i(of)d(con)o (tiguous)h(v)n(alues)g(of)f(the)h(same)f(basic)h(t)o(yp)q(e,)g(sp)q (eci\014ed)i(b)o(y)75 1029 y Fk(start)k Fn(Initial)12 b(elemen)o(t)75 1107 y Fk(coun)o(t)19 b Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fl(count)c Fc(\025)i Fl(0)p Fn(\))75 1185 y Fk(datat)o(yp)q(e)19 b Fn(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1259 y(Th)o(us,)h(if)e Fl(A)i Fn(is)f(an)h(arra)o(y)f(of)g(in)o(tegers,)h(then)h Fl(A\(1\),)21 b(4,)g(MPI)p 1127 1259 V 15 w(INT)13 b Fn(is)h(the)g(bu\013er)h(comp)q(onen)o (t)e(with)g(en)o(tries)75 1309 y Fl(A\(1\),)21 b(A\(2\),)g(A\(3\),)f(A\(4\))p Fn(.)75 1415 y Fk(V)l(ector)15 b(comp)q(onen)o(t)75 1491 y Fn(A)f(sequence)j(of)c(equally)h(spaced)h(and)f(equally)f(sized)i(blo)q(c)o (ks)f(of)g(elemen)o(ts)g(of)g(the)g(same)g(basic)g(t)o(yp)q(e,)g(sp)q (eci\014ed)75 1541 y(b)o(y)75 1615 y Fk(start)20 b Fn(Initial)12 b(elemen)o(t)75 1693 y Fk(coun)o(t)19 b Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fl(count)c Fc(\025)i Fl(0)p Fn(\))75 1771 y Fk(stride)18 b Fn(Num)o(b)q(er)c(of)f(elemen)o(ts)h(b)q(et)o(w)o(een)h (the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1850 y Fk(len)o(blk)j Fn(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fl(lenblk)c Fb(>)i Fl(0)p Fn(\))75 1928 y Fk(datat)o(yp)q(e)19 b Fn(T)o(yp)q(e)c(of)e(elemen)o(ts)158 2001 y(Assume)20 b(that)h(an)f(elemen) o(t)g(of)f(t)o(yp)q(e)i Fl(datatype)e Fn(o)q(ccupies)i Fl(m)f Fn(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 2051 y(mac)o(hine,)11 b(the)i(bu\013er)h(comp)q(onen)o(t)d(describ)q(ed)k(b)o (y)d(these)i(\014v)o(e)f(parameters)f(consists)i(of)e(the)h Fl(count)e Fn(v)n(ariables)h(at)75 2101 y(addresses)k Fl(start,)21 b(start+m,)f(...)43 b(,)21 b(start+m*\(lenblk-1\),)d(start+m*stride,)75 2151 y(start+m*\(stride+1)o(\),)h(...,)i(start+m*\(stride+l)o(enbl)o(k-1\),)d (...)p Fn(.)i(F)m(or)14 b(example,)f(if)h Fl(A)g Fn(is)h(an)f(arra)o(y)h(of) 75 2201 y(in)o(tegers,)f(then)137 2274 y Fc(\017)21 b Fl(A\(1\),)g(5,)g(3,)g (2,)h(MPI)p 575 2274 V 15 w(INT)c Fn(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e (with)h(en)o(tries)h Fl(A\(1\),)h(A\(2\),)f(A\(4\),)h(A\(5\),)179 2324 y(A\(7\))p Fn(.)137 2402 y Fc(\017)g Fl(A\(1\),)g(5,)g(0,)g(2,)h(MPI)p 575 2402 V 15 w(INT)c Fn(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e(with)h(en)o (tries)h Fl(A\(1\),)h(A\(2\),)f(A\(1\),)h(A\(2\),)179 2452 y(A\(1\))p Fn(.)137 2530 y Fc(\017)g Fl(A\(10\),)f(5,)i(-1,)f(2,)g(MPI)p 618 2530 V 15 w(INT)9 b Fn(is)g(the)h(bu\013er)g(comp)q(onen)o(t)f(with)g(en) o(tries)h Fl(A\(10\),)21 b(A\(11\),)f(A\(9\),)h(A\(10\),)179 2580 y(A\(8\))p Fn(.)158 2654 y Fl(Contiguous)13 b Fn(is)h(a)g(sp)q(ecial)h (case)h(of)e Fl(vector)p Fn(.)19 b(A)c(v)o(ector)g(bu\013er)h(comp)q(onen)o (t)d(can)i(b)q(e)h(used)f(to)f(extract)i(an)75 2704 y(arbitrary)e(submatrix)e (of)i(a)f(t)o(w)o(o-dimensional)e(matrix.)p eop %%Page: 7 8 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(7)75 45 y Fk(Indexed)15 b(comp)q(onen)o(t)75 122 y Fn(A)f(sequence)i(of)d(elemen)o(ts)h(of)f(the)i(same)e(basic)h(t)o(yp)q (e,)g(sp)q(eci\014ed)h(b)o(y)75 205 y Fk(start)20 b Fn(initial)12 b(elemen)o(t)75 288 y Fk(coun)o(t)19 b Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts) h(\()p Fl(count)c Fc(\025)i Fl(0)p Fn(\))75 371 y Fk(arra)o(y)p 188 371 15 2 v 17 w(of)p 244 371 V 16 w(indices)19 b Fn(Arra)o(y)14 b(of)f(displacemen)o(ts)h(of)f(the)i(elemen)o(ts)e(in)h(the)h(bu\013er)g (comp)q(onen)o(ts,)e(relativ)o(e)h(to)f(the)179 421 y(initial)f(elemen)o(t.) 75 504 y Fk(datat)o(yp)q(e)19 b Fn(T)o(yp)q(e)c(of)e(elemen)o(ts)158 587 y(F)m(or)f(example,)f(if)h Fl(A)g Fn(is)g(an)g(arra)o(y)g(of)g(in)o (tegers,)h(and)g Fl(I)f Fn(is)g(an)g(arra)o(y)h(with)f(v)n(alues)g Fl(2,)21 b(-1,)g(2,)h(1)12 b Fn(then)h Fl(A\(3\),)75 636 y(4,)21 b(I,)h(MPI)p 275 636 14 2 v 15 w(INT)13 b Fn(is)h(the)g(bu\013er)h(comp)q (onen)o(t)e(with)h(en)o(tries)h Fl(A\(5\),)21 b(A\(2\),)f(A\(5\),)h(A\(4\))p Fn(.)158 686 y Fl(Vector)13 b Fn(can)h(b)q(e)g(though)o(t)g(of)f(as)h(a)g(sp) q(ecial)g(case)h(of)e Fl(indexed)g Fn(in)g(whic)o(h)h(the)g(indices)h(are)f (generated)h(with)75 736 y(a)f(constan)o(t)g(stride.)158 869 y Fj(Discussion:)33 b Fi(The)10 b(ab)q(o)o(v)o(e)h(statemen)o(t)g(is)g(not)g (quite)g(true,)g(since)h(there)f(is)g(no)f Fd(lenblk)e Fi(parameter)j(on)g (the)g(indexed)75 918 y(comp)q(onen)o(t.)18 b(It)13 b(w)o(as)g(suggested)h (that)f(indexed)i(will)f(pro)o(vide)h(a)e(list)h(of)f(\(p)q(osition,)i(blo)q (c)o(k)p 1410 918 12 2 v 15 w(length\))g(pairs.)75 1109 y Fk(Heterogeneous)e (v)o(ector)i(\(h)o(v)o(ector\))f(comp)q(onen)o(t)75 1186 y Fn(Same)e(as)h Fl(vector)p Fn(,)f(except)j(that)e Fl(stride)f Fn(is)h(giv)o(en)g(in)f(m)o(ultiple)f(of)i(b)o(ytes,)h(rather)g(than)f(m)o (ultiple)e(of)i(elemen)o(ts:)75 1277 y Fk(start)20 b Fn(Initial)12 b(elemen)o(t)75 1360 y Fk(coun)o(t)19 b Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fl(count)c Fc(\025)i Fl(0)p Fn(\))i(\(in)o(teger\)) 75 1443 y Fk(stride)k Fn(Num)o(b)q(er)c(of)f Fk(b)o(ytes)g Fn(b)q(et)o(w)o(een)i(the)f(start)h(of)e(eac)o(h)i(blo)q(c)o(k)e(\(in)o (teger\))75 1526 y Fk(len)o(blk)18 b Fn(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f (in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fl(lenblk)c Fb(>)i Fl(0)p Fn(\))i(\(in)o(teger\))75 1609 y Fk(datat)o(yp)q(e)19 b Fn(T)o(yp)q(e)c(of)e (elemen)o(ts)h(\(state\))158 1700 y(Assume)20 b(that)h(an)f(elemen)o(t)g(of)f (t)o(yp)q(e)i Fl(datatype)e Fn(o)q(ccupies)i Fl(m)f Fn(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 1750 y(mac)o(hine,)15 b(the)i(bu\013er)g(comp)q(onen)o(t)f(describ)q(ed)i(b)o(y)e(these)i(\014v)o (e)e(parameters)h(consists)g(of)f(the)g Fl(count)g Fn(v)n(ariables)75 1800 y(at)c(addresses)j Fl(start,)20 b(start+m,)h(...)43 b(,)21 b(start+m*\(lenblk-1\))o(,)e(start+stride,)g(start+stride+m,)75 1850 y(...,)i(start+stride+m*\(l)o(enblk)o(-1\),)d(...)p Fn(.)158 1900 y(Consider)c(a)g(C)g(arra)o(y)f(declared)i(as)75 1991 y Fl(struct)21 b({)140 2041 y(char)g(info[3];)140 2091 y(short)g(number;)140 2140 y(double)g(val;)228 2190 y(})65 b(a[100])158 2281 y Fn(Assume)14 b(that)h(all)e(structure)j(comp)q(onen)o(ts)e(are)h(b)o(yte)g(aligned,)e(a)h Fl(short)f Fn(o)q(ccupies)j(t)o(w)o(o)d(b)o(ytes,)i(a)f Fl(double)75 2331 y Fn(o)q(ccupies)21 b(eigh)o(t)e(b)o(ytes,)i(and)f(no)f(padding)g(is)g (needed.)37 b(Then)20 b(the)g(bu\013er)g(comp)q(onen)o(t)f Fl(&a,)i(100,)g(13,)g(2,)75 2381 y(MPI)p 144 2381 14 2 v 15 w(CHAR)14 b Fn(extracts)i(the)g(\014rst)f(t)o(w)o(o)g(c)o(haracters)h(of)f (eac)o(h)g(structure.)23 b(The)15 b(bu\013er)h(comp)q(onen)o(t)e Fl(&\(a.number\),)75 2431 y(100,)21 b(13,)g(1,)h(MPI)p 406 2431 V 15 w(SHORT)e Fn(extracts)k(the)e Fl(short)f Fn(\014eld)h(of)f(eac)o(h) i(structure;)k(and)22 b(the)h(bu\013er)g(comp)q(onen)o(t)75 2481 y Fl(&\(a.val\),)d(100,)h(13,)g(1)c Fn(extracts)h(the)g Fl(double)e Fn(\014eld)h(of)g(eac)o(h)g(structure.)30 b(Th)o(us,)18 b(heterogeneous)h(v)o(ectors)75 2531 y(can)14 b(b)q(e)h(used)f(to)g(extract)h (v)o(ectors)g(of)e(homogeneous)g(v)n(alues)h(from)e(heterogeneous)k(data.)158 2580 y Fl(Vector)d Fn(is)g(a)h(sp)q(ecial)g(case)h(of)e Fl(heterogeneous)19 b(vector)p Fn(.)p eop %%Page: 8 9 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(8)75 45 y Fk(Heterogeneous)13 b(indexed)h(\(hindexed\))f(comp) q(onen)o(t)75 122 y Fn(Same)g(as)h(indexed,)g(except)h(that)f(displacemen)o (ts)f(are)i(in)e(b)o(ytes,)h(rather)h(than)f(m)o(ultiples)e(of)h(elemen)o(t)g (size.)75 213 y Fk(start)20 b Fn(initial)12 b(elemen)o(t)75 296 y Fk(coun)o(t)19 b Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fl(count)c Fc(\025)i Fl(0)p Fn(\))75 379 y Fk(arra)o(y)p 188 379 15 2 v 17 w(of)p 244 379 V 16 w(indices)19 b Fn(Arra)o(y)12 b(of)g(b)o(yte)g(displacemen)o(ts)g(of)g(the)g(elemen)o(ts)h(in)e(the)i (bu\013er)g(comp)q(onen)o(ts,)f(relativ)o(e)g(to)179 429 y(the)i(initial)e (elemen)o(t.)75 512 y Fk(datat)o(yp)q(e)19 b Fn(T)o(yp)q(e)c(of)e(elemen)o (ts)158 603 y(F)m(or)18 b(example,)f(if)h Fl(a)g Fn(is)g(the)h(C)f(arra)o(y)g (of)f(the)i(previous)g(example,)e(and)i Fl(I)e Fn(is)i(an)f(arra)o(y)g(with)f (v)n(alues)h Fl(0,)75 653 y(13,)j(13,)g(27,)h(26)16 b Fn(then)i Fl(&a,)k(5,)f(I,)g(MPI)p 782 653 14 2 v 15 w(CHAR)c Fn(is)g(the)h(bu\013er)h (comp)q(onen)o(t)d(with)h(en)o(tries)i Fl(a[0].info[0],)75 703 y(a[1].info[0],)g(a[1].info[0],)g(a[1].info[1],)g(a[1].info[0])p Fn(.)158 753 y(Lik)o(e)e Fl(heterogeneous)i(vector)p Fn(,)d Fl(heterogeneous)j(indexed)c Fn(extends)k(the)e Fl(indexed)f Fn(facilit)o(y)f(so)i(as)g(to)75 803 y(allo)o(w)12 b(the)j(extraction)f(of)f (homogeneous)g(data)h(from)e(a)h(heterogeneous)j(la)o(y)o(out.)158 931 y Fj(Discussion:)158 977 y Fi(The)d(use)h(of)e(displacemen)o(ts)k(in)e (indexed)h(bu\013er)f(comp)q(onen)o(ts)h(has)e(the)h(same)f(e\013ect)g(as)h (indirect)h(addressing)g(in)o(to)75 1022 y(an)d(arra)o(y)g(with)g(initial)j (index)e(zero.)j(I.e.,)11 b(if)h Fd(a)f Fi(is)h(an)g(arra)o(y)g(with)g (initial)j(index)e(zero,)e(then)h(a)g(bu\013er)h(comp)q(onen)o(t)g(de\014ned) 75 1068 y(b)o(y)g(parameters)g Fd(a[0],)k(n,)i(i,)g(type)11 b Fi(extract)h(the)h(elemen)o(ts)g Fd(a[i[0]],)j(a[i[1]],)h(...,)h(a[i[n]])p Fi(.)c(This)f(\014ts)f(w)o(ell)h(C)75 1113 y(usage)g(but)f(ill-\014ts)i(F)m (ortran)e(usage)h(\(initial)h(index)g(one\).)i(W)m(e)c(to)q(ok)h(a)f (decision)i(of)d(principle)k(to)d(force)g(\\start)g(from)f(zero")75 1159 y(index)j(arithmetic)f(on)g(F)m(ortran)f(users,)h(rather)f(than)h(ha)o (v)o(e)f(di\013eren)o(t)i(index)g(arithmetic)f(in)g(the)f(C)g(and)h(F)m (ortran)f(binding)75 1205 y(of)h(MPI.)158 1250 y(The)19 b(use)f(of)h(b)q(oth) g(bu\013er)h(comp)q(onen)o(ts)f(with)h(b)o(yte)e(and)i(elemen)o(t)f(size)g (displacemen)o(ts)j(is)d(motiv)n(ated)h(b)o(y)f(the)75 1296 y(observ)n(ation)c(that)f(some)g(applications)j(require)d(the)g(more)f (general)i(b)o(yte)f(displacemen)o(t,)h(but)f(that)g(most)f(applicatio)q(ns) 75 1342 y(w)o(ould)d(use)g(elemen)o(t)g(size)h(displacemen)o(t,)h(whic)o(h)e (is)g(m)o(uc)o(h)g(more)g(natural)g(and)g(con)o(v)o(enien)o(t)i(\(and)e(mac)o (hine)h(indep)q(enden)o(t\).)75 1387 y(Th)o(us,)f(rather)f(than)h(forcing)g (all)g(users)g(to)f(do)g(b)o(yte)h(arithmetics,)h(w)o(e)e(decided)h(to)f(pro) o(vide)i(separately)g(the)e(more)g(adv)n(anced)75 1433 y(function)14 b(to)f(\\exp)q(ert)h(programmers".)75 1632 y Fh(1.5.2)55 b(Bu\013er)18 b(op)r(erations)75 1709 y Fn(A)c(bu\013er)i(is)e(describ)q(ed)i(b)o(y)e(an)g (opaque)g Fk(bu\013er)h(descriptor)c Fn(ob)r(ject)k(accessed)h(via)e(a)g Fk(bu\013er)h(handle)p Fn(.)i(Suc)o(h)75 1758 y(an)f(ob)r(ject)g(is)g (created)h(b)o(y)f Fl(MPI)p 582 1758 V 15 w(CREATE)p 729 1758 V 15 w(BUFFER)p Fn(.)e(It)i(is)f(asso)q(ciated)i(with)f(successiv)o(e)i (bu\013er)f(comp)q(onen)o(ts)e(b)o(y)75 1808 y(calling)g(in)i(succession)h (one)f(of)f(the)i(functions)e Fl(MPI)p 914 1808 V 15 w(APPEND)p 1061 1808 V 15 w(xxx)p Fn(,)g(where)i Fl(xxx)e Fn(iden)o(ti\014es)h(the)h(t)o (yp)q(e)f(of)f(bu\013er)75 1858 y(comp)q(onen)o(t)e(app)q(ended)h(to)f(the)h (bu\013er.)20 b(It)14 b(b)q(ecomes)h(a)o(v)n(ailable)d(for)i(use)h(in)f(comm) o(unicatio)o(n)e(op)q(erations)i(once)75 1908 y(committed)c(b)o(y)h(the)i (function)e Fl(MPI)p 635 1908 V 15 w(COMMIT)p 782 1908 V 15 w(BUFFER)p Fn(.)f(Once)j(committed,)c(the)k(descriptor)g(cannot)f(b)q(e)g(mo) q(di\014ed)75 1958 y(an)o(y)h(more.)158 2008 y(Note)i(that)f(the)h(commit)d (op)q(eration)i(do)q(es)h(not)g(create)g(y)o(et)g(a)f(message,)g(and)g(do)q (es)i(not)e(bind)g(the)h(data)f(to)75 2057 y(b)q(e)h(sen)o(t.)j(It)c(is)g(a)g (commit)d(for)i(the)i(bu\013er)f(descriptor,)h(not)f(a)g(commi)o(t)d(for)j (the)g(bu\013er)h(con)o(ten)o(t.)158 2107 y(A)f(bu\013er)g(descriptor)h(is)e (destro)o(y)o(ed)h(after)g(the)g(completion)e(of)g(the)i(\014rst)h(comm)o (unicati)o(on)c(op)q(eration)i(that)75 2157 y(uses)i(it,)e(if)g(it)g(is)h (ephemeral,)f(or)h(after)g(it)f(is)h(freed)h(b)o(y)e(a)h(call)f(to)h Fl(MPI)p 1170 2157 V 15 w(FREE)f Fn(and)g(an)o(y)h(p)q(ending)f(comm)o (unication)75 2207 y(op)q(eration)h(that)g(uses)i(it)e(has)g(completed,)g(if) f(it)h(is)g(p)q(ersisten)o(t.)21 b(After)15 b(the)g(bu\013er)g(descriptor)g (ob)r(ject)g(has)g(b)q(een)75 2257 y(destro)o(y)o(ed,)g(the)f(handle)g(is)g (unde\014ned.)158 2342 y Fk(MPI)p 257 2342 15 2 v 17 w(CREA)l(TE)p 471 2342 V 19 w(BUFFER\()h(bu\013er,)g(p)q(ersistence)e(\))158 2427 y Fn(Create)i(a)e(new)i(bu\013er)g(descriptor)g(ob)r(ject.)k(The)14 b(parameters)g(are:)75 2510 y Fk(OUT)i(bu\013er)j Fn(bu\013er)c(handle)75 2593 y Fk(IN)h(p)q(ersistence)i Fn(bu\013er)11 b(p)q(ersistence)h(\(state)e (t)o(yp)q(e)g(with)f(p)q(ossible)g(v)n(alues)g Fl(MPI)p 1350 2593 14 2 v 15 w(PERSISTENT)e Fn(or)j Fl(MPI)p 1705 2593 V 15 w(EPHEMERAL)p Fn(\).)p eop %%Page: 9 10 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)733 b Fn(9)158 45 y Fj(Discussion:)158 95 y Fi(F)m(unction)15 b(really)f(is)g Fd(create)p 585 95 12 2 v 12 w(buffer)p 717 95 V 11 w(descriptor)p Fi(.)f(Ma)o(y)h(w)o(an)o(t)f(to)g(c)o(hange)g(the)h (name.)158 263 y Fk(MPI)p 257 263 15 2 v 17 w(APPEND)p 481 263 V 17 w(CONTIGUOUS\()h(bu\013er,)g(start,)g(coun)o(t,)f(datat)o(yp)q(e\)) 158 348 y Fn(App)q(end)h(a)e(con)o(tigous)h(comp)q(onen)o(t)f(to)h(bu\013er.) 19 b(The)14 b(parameters)g(are:)75 429 y Fk(INOUT)i(bu\013er)j Fn(bu\013er)c(handle)75 512 y Fk(IN)h(start)k Fn(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75 594 y Fk(IN)j(coun)o(t)j Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 676 y Fk(IN)i(datat)o(yp)q(e)k Fn(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 793 y Fk(MPI)p 257 793 V 17 w(APPEND)p 481 793 V 17 w(VEC\()j(bu\013er,)d(start,)h(coun)o(t,)g (stride,)f(len)o(blk,)f(datat)o(yp)q(e)i(\))158 878 y Fn(App)q(end)g(a)e(v)o (ector)i(bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 959 y Fk(INOUT)i(bu\013er)j Fn(bu\013er)c(handle)75 1041 y Fk(IN)h(start)k Fn(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o (t)h(\(c)o(hoice\))75 1123 y Fk(IN)j(coun)o(t)j Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 1206 y Fk(IN)i(stride)j Fn(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(b)q(et)o(w)o(een)j(the)e(start)h(of)e (eac)o(h)h(blo)q(c)o(k)g(\(in)o(teger\))75 1288 y Fk(IN)i(len)o(blk)i Fn(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)i(blo)q(c)o(k)e(\(in)o (teger\))75 1370 y Fk(IN)j(datat)o(yp)q(e)k Fn(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 1530 y Fj(Discussion:)158 1575 y Fi(Whic)o(h)h(order)e(is)h(more)f(natural:)18 b Fd(stride,)f(lenblk)11 b Fi(or)i Fd(lenblk,)j(stride)p Fi(?)158 1743 y Fk(MPI)p 257 1743 V 17 w(APPEND)p 481 1743 V 17 w(INDEXED\()g(bu\013er,)f(start,)g(coun)o (t,)f(arra)o(y)p 1294 1743 V 17 w(of)p 1350 1743 V 17 w(indices,)f(datat)o (yp)q(e\))158 1829 y Fn(App)q(end)i(an)e(indexed)i(bu\013er)g(comp)q(onen)o (t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 1910 y Fk(INOUT)i(bu\013er)j Fn(bu\013er)c(handle)75 1992 y Fk(IN)h(start)k Fn(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 2074 y Fk(IN)j(coun)o(t)j Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 2156 y Fk(IN)i(arra)o(y)p 259 2156 V 17 w(of)p 315 2156 V 17 w(indices)i Fn(arra)o(y)c(of)f(displacemen)o(ts)g(of)h(en)o (tries)h(relativ)o(e)e(to)h(start)h(\(arra)o(y)e(of)h(in)o(tegers\))75 2239 y Fk(IN)i(datat)o(yp)q(e)k Fn(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 2320 y(The)g(index)f(v)n(alues)g(used)h(to)f(create)i(this)f (comp)q(onen)o(t)e(are)i(the)g(v)n(alues)f(pro)o(vided)g(at)g(the)h(call;)e (subsequen)o(t)75 2370 y(up)q(dates)j(of)e(the)i(arra)o(y)e(of)h(indices)g (will)e(not)i(a\013ect)h(the)f(bu\013er)h(descriptor)g(comp)q(onen)o(t.)158 2455 y Fk(MPI)p 257 2455 V 17 w(APPEND)p 481 2455 V 17 w(HVEC\()i(bu\013er,)e (start,)g(coun)o(t,)f(stride,)g(len)o(blk,)f(datat)o(yp)q(e)i(\))158 2540 y Fn(App)q(end)g(a)e(heterogeneous)j(v)o(ector)f(bu\013er)g(comp)q(onen) o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 2621 y Fk(INOUT)i(bu\013er)j Fn(bu\013er)c(handle)75 2704 y Fk(IN)h(start)k Fn(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))p eop %%Page: 10 11 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(10)75 45 y Fk(IN)16 b(coun)o(t)j Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 128 y Fk(IN)i(stride)j Fn(Num)o(b)q(er)13 b(of)h(b)o(ytes)g(b)q(et)o(w)o(een)h(the)g(start)f(of)g (eac)o(h)g(blo)q(c)o(k)g(\(in)o(teger\))75 211 y Fk(IN)i(len)o(blk)i Fn(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)i(blo)q(c)o(k)e(\(in)o (teger\))75 294 y Fk(IN)j(datat)o(yp)q(e)k Fn(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 413 y Fk(MPI)p 257 413 15 2 v 17 w(APPEND)p 481 413 V 17 w(HINDEXED\()i(bu\013er,)f(start,)g(coun)o(t,) g(arra)o(y)p 1332 413 V 16 w(of)p 1387 413 V 17 w(indices,)f(datat)o(yp)q (e\))158 498 y Fn(App)q(end)h(an)e(heterogeneous)j(indexed)f(bu\013er)g(comp) q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 581 y Fk(INOUT)i(bu\013er)j Fn(bu\013er)c(handle)75 664 y Fk(IN)h(start)k Fn(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 747 y Fk(IN)j(coun)o(t)j Fn(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 830 y Fk(IN)i(arra)o(y)p 259 830 V 17 w(of)p 315 830 V 17 w(indices)i Fn(arra)o(y)c(of)f(b)o(yte)h(displacemen)o(ts)g(of)f(en) o(tries)i(relativ)o(e)f(to)f(start)i(\(arra)o(y)f(of)f(in)o(tegers\))75 913 y Fk(IN)j(datat)o(yp)q(e)k Fn(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 1031 y Fk(MPI)p 257 1031 V 17 w(COMMIT)p 485 1031 V 19 w(BUFFER\()h(bu\013er)f(\))158 1117 y Fn(Commi)o(t)7 b(a)j(bu\013er)h(descriptor)g(ob)r(ject.)18 b(The)10 b(ob)r(ject)h(cannot)f (b)q(e)g(c)o(hanged)h(an)o(ymore)d(after)i(it)g(is)g(committed.)75 1166 y(A)h(bu\013er)g(descriptor)h(ob)r(ject)g(can)f(b)q(e)g(used)g(for)g (comm)o(uni)o(cation)d(only)i(after)g(it)h(is)f(committed.)15 b(The)c(parameters)75 1216 y(are:)75 1299 y Fk(INOUT)16 b(bu\013er)j Fn(bu\013er)c(descriptor)g(handle)158 1461 y Fj(Discussion:)158 1507 y Fi(The)i(commit)h(function)g(ma)o(y)g(do)f(prepro)q(cessing)j(of)d (the)g(bu\013er)h(descriptor)h(to)e(simplify)i(its)f(use)f(and)h(commit)75 1552 y(resources)12 b(in)g(the)g(comm)o(unication)i(subsystem)e(for)f(its)h (use.)k(The)c(commit)f(function)i(ma)o(y)f(generate)g(a)f(new)g(ob)r(ject)g (and,)75 1598 y(hence,)i(mo)q(dify)h(the)g(ob)r(ject)f(handle.)158 1644 y(The)18 b(curren)o(t)g(prop)q(osal)h(do)q(es)g(not)f(allo)o(w)h(to)e (create)h(a)g(bu\013er)g(descriptor,)i(use)e(it)h(in)f(a)g(comm)o(unication)i (after)75 1689 y(committed,)14 b(then)g(mo)q(dify)g(it)g(partially)i(and)e (use)g(it)g(for)f(another)h(comm)o(unication)i({)d(a)h(new)f(bu\013er)h (descriptor)h(has)f(to)75 1735 y(b)q(e)h(created.)21 b(Of)13 b(course,)i(the)g(same)f(bu\013er)h(descriptor)h(can)f(b)q(e)f(reused)h(man)o (y)g(times)g(in)g(successiv)o(e)h(comm)o(unications)75 1781 y(that)d(send)g(data)g(from)f(the)g(same)h(bu\013ers;)g(e.g.,)f(a)g(new)h (send)g(with)g(a)f(bu\013er)i(descriptor)g(handle)g(can)f(b)q(e)f(initiated)j (once)75 1826 y(the)e(previous)i(send)f(with)f(this)h(handle)h(has)e (completed.)158 1959 y Fn(Example)f(1:)75 2008 y(Consider)i(the)h(follo)o (wing)c(fragmen)o(t)h(of)i(F)m(ortran)f(co)q(de)75 2100 y Fl(DOUBLE)21 b(PRECISION)e(A\(10,20\))75 2150 y(INTEGER)h(B,)i(C\(5,10\),)e(I\(3\))75 2199 y(INTEGER)g(BH)75 2249 y(DATA)h(\(I\(J\),)g(J=1,3\))f(/9,2,1/)75 2299 y(...)75 2349 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 2399 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(B,)h(1,)f(MPI_INTEGER\))75 2448 y(CALL)g(MPI_APPEND_VEC)e(\(BH,)i(A\(1,3\),)f(11,)h(4,)h(2,)f (MPI_DOUBLE\))75 2498 y(CALL)g(MPI_APPEND_INDEXE)o(D\(BH,)d(C\(3,7\),)j(3,)g (I,)g(MPI_INTEGER\))158 2590 y Fn(Then)14 b(the)h(bu\013er)g(asso)q(ciated)f (with)g(the)g(handle)g Fl(BH)g Fn(consists)h(of)e(the)h(sequence)i(of)e(v)n (ariables)158 2639 y Fl(B,)21 b(A\(1,3\),)g(A\(2,3\),)f(A\(5,3\),)g (A\(6,3\),)h(A\(9,3\),)f(A\(10,3\),)g(A\(3,4\),)h(A\(4,4\),)f(A\(7,4\),)g (A\(8,4\),)75 2689 y(A\(1,5\),)g(C\(2,8\),)h(C\(5,7\),)f(C\(4,7\))p Fn(.)p eop %%Page: 11 12 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(11)158 45 y(A)14 b(message)f(created)j(from)c(this)i(bu\013er) h(will)d(consist)j(of)e(a)g(sequence)j(of)e(one)g(in)o(teger,)f(follo)o(w)o (ed)g(b)o(y)g(elev)o(en)75 95 y(double)h(precision)g(reals,)g(follo)o(w)o(ed) e(b)o(y)i(three)h(in)o(tegers.)158 145 y(Example)d(2:)75 195 y(Supp)q(ose)j(one)f(has)g(an)g(arra)o(y)f(of)g(data)h(declared)h(in)e(C)h (as)75 272 y Fl(struct)21 b({)140 322 y(int)h(type;)140 372 y(double)f(position[3];)140 422 y(double)g(velocity[3];)228 472 y(})g(particle[1000])158 549 y Fn(and)d(one)g(w)o(an)o(ts)h(to)f (transfer)h(in)f(one)g(message)g(the)h(n)o(um)o(b)q(er)e(of)h(particles)g(of) g(t)o(yp)q(e)h(1,)f(the)h(n)o(um)o(b)q(er)f(of)75 599 y(particles)g(of)g(t)o (yp)q(e)g(2,)g(and)g(the)g(p)q(ositions)g(of)f(all)g(the)h(particles)h(of)e (these)i(t)o(w)o(o)f(t)o(yp)q(es.)31 b(The)18 b(follo)o(wing)d(co)q(de)75 649 y(fragmen)o(t)d(creates)k(a)e(suitable)f(bu\013er)i(descriptor.)75 727 y Fl(int)21 b(count1)g(=)g(0;)75 777 y(int)g(count2)g(=)g(0;)75 826 y(...)75 876 y(mpi_create_buffer)o(\()e(&handle,)h(MPI_PERSISTENT\);)75 926 y(mpi_append_contig)o(uous\()e(handle,)j(\(void)f(*\)&count1,)g (MPI_INT\);)75 976 y(mpi_append_contig)o(uous\()e(handle,)j(\(void)f (*\)&count2,)g(MPI_INT\);)75 1026 y(for)h(\(i=0;)g(i<1000;)f(i++\))h({)140 1075 y(if)h(\(particle[i].typ)o(e)d(==)i(1\))h({)206 1125 y(count1++;)206 1175 y(mpi_append_conti)o(guous)o(\()d(&handle,)h(\(void)h (*\)&particle[i].po)o(sitio)o(n,)e(3,)75 1225 y(MPI_DOUBLE\);)g(})140 1275 y(elseif)i(\(particle[i].type)d(==)k(2\))f({)206 1325 y(count2++;)206 1374 y(mpi_append_conti)o(guous)o(\()e(&handle,)h(\(void)h (*\)&particle[i].po)o(sitio)o(n,)e(3,)75 1424 y(MPI_DOUBLE\);)g(})140 1474 y(})75 1524 y(mpi_commit_buffer)o(\(hand)o(le\))75 1574 y(...)158 1651 y Fn(The)14 b(bu\013er)h(descriptor)f(handle)g(can)g(no)o(w)f (b)q(e)h(used)g(to)g(send)g(\(rep)q(eatedly\))h(the)f(particle)g(coun)o(ts)g (for)f(t)o(yp)q(e)75 1701 y(1)g(and)f(2,)h(and)g(the)g(p)q(ositions)g(of)f (the)i(particles)f(with)g(these)h(t)o(w)o(o)f(t)o(yp)q(es,)g(as)g(long)f(as)h (the)h(t)o(yp)q(es)g(of)e(the)h(particles)75 1751 y(is)h(not)g(mo)q (di\014ed.)158 1801 y(Supp)q(ose)i(the)f(incoming)d(data)i(is)h(to)f(b)q(e)h (stored)h(at)e(the)h(receiv)o(er)h(in)o(to)e(the)h(t)o(w)o(o)g(coun)o(t)f(v)n (ariables)g Fl(count1)75 1851 y Fn(and)h Fl(count2)f Fn(and)h(in)o(to)f(the)i Fl(position)d Fn(comp)q(onen)o(ts)i(of)f(the)i(initial)d(en)o(tries)j(of)f (the)g(arra)o(y)g(of)f(particles.)23 b(The)75 1900 y(follo)o(wing)10 b(co)q(de)i(generates)i(a)e(suitable)g(bu\013er)h(descriptor)h(for)d(the)i (receiv)o(e)g(op)q(eration)f(\(w)o(e)h(assume)f(4)f(b)o(ytes)i(for)75 1950 y(in)o(teger)h(and)g(8)g(b)o(ytes)g(for)g(double\).)75 2028 y Fl(...)75 2078 y(mpi_create_buffer)o(\()19 b(&handle,)h (MPI_PERSISTENT\);)75 2128 y(mpi_append_contig)o(uous\()e(&handle,)i(\(void)h (*\)&count1,)f(MPI_INT\);)75 2177 y(mpi_append_contig)o(uous\()e(handle,)j (\(void)f(*\)&count2,)g(MPI_INT\);)75 2227 y(mpi_append_hvec\()e(handle,)j (\(void)g(*\)&particle[0].)o(posit)o(ion,)d(1000,)j(52,)g(3,)75 2277 y(MPI_DOUBLE\);)e(mpi_commit_buffer\()f(handle\))75 2327 y(...)158 2405 y Fn(A)d(bu\013er)g(handle)f(can)h(b)q(e)g(used)g(for)f(comm)o (unication,)d(ev)o(en)k(if)f(it)g(is)g(not)g(asso)q(ciated)i(with)e(an)o(y)g (v)n(ariables)75 2455 y(\(i.e.,)e(ev)o(en)i(if)e(it)h(w)o(as)g(not)g(set)h(b) o(y)f(an)o(y)g Fl(MPI)p 761 2455 14 2 v 15 w(APPEND)p 908 2455 V 15 w(xxx)f Fn(call\).)17 b(Suc)o(h)d(a)f(handle)g(is)g(asso)q(ciated)h (with)f(an)g(empt)o(y)75 2504 y(bu\013er,)i(and)f(a)g(message)g(created)i (from)c(it)i(con)o(tains)g(no)g(data.)19 b(The)c(handle)f(still)f(need)i(b)q (e)g(committed)d(b)q(efore)75 2554 y(it)i(is)f(used.)158 2604 y(The)h(same)e(en)o(try)i(ma)o(y)d(app)q(ear)i(more)f(than)i(once)f(in)g(a)g (bu\013er)h(comp)q(onen)o(t,)e(and)h(distinct)g(bu\013er)h(comp)q(o-)75 2654 y(nen)o(ts)i(ma)o(y)d(o)o(v)o(erlap.)21 b(If)15 b(a)g(bu\013er)h (descriptor)g(with)f(replicated)h(en)o(tries)g(is)f(used)h(to)f(receiv)o(e)h (a)f(message,)f(then)75 2704 y(the)g(\014nal)g(v)n(alue)f(of)g(an)h(en)o(try) g(that)g(o)q(ccurs)i(more)d(than)g(once)i(in)e(the)i(bu\013er)g(descriptor)g (is)f(unde\014ned.)p eop %%Page: 12 13 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(12)75 45 y Fh(1.5.3)55 b(Comm)n(unication)21 b(Bu\013ers)d({)g(Extension)1931 72 y Fe(TO)11 b(DISCUSS)75 200 y Fj(Discussion:)158 246 y Fi(James)g(Co)o(wnie)h(prop)q(oses)g(the)g (follo)o(wing)h(extension)g(to)e(bu\013er)i(descriptors.)18 b(Its)11 b(main)h(feature)f(is)h(the)f(addition)j(of)75 292 y(a)g(bu\013er)h(comp)q(onen)o(t)g(that)g(allo)o(ws)g(for)f(the)g(e\016cien)o (t)h(represen)o(tation)h(and)f(manipulation)j(of)13 b(arra)o(ys)i(of)f(C)f (structures.)75 337 y(This)h(extension)h(has)e(not)g(b)q(een)h(discussed)h(b) o(y)e(MPIF.)158 387 y(\(The)g(original)j(prop)q(osal)f(has)e(b)q(een)h(repro) q(cessed)g(b)o(y)f(Rust)o(y)h(and)g(Marc.\))158 520 y Fn(F)m(ortran)d(and)f (C)h(b)q(oth)f(allo)o(w)f(to)i(pro)o(vide)g(hierarc)o(hical)f(de\014nitions)h (of)f(comp)q(osite)g(data)g(structures;)k(on)c(the)75 569 y(other)15 b(hand,)g(bu\013er)h(descriptors)g(are)f(\015at)g(structures.)23 b(Th)o(us,)15 b(it)g(is)f(incon)o(v)o(enien)o(t)h(to)g(use)g(bu\013er)h (descriptors)75 619 y(in)d(order)i(to)f(transfer)h(records.)158 748 y Fj(Discussion:)158 797 y Fi(There)i(also)h(is)g(an)g(e\016ciency)g (consideration.)33 b(Consider)18 b(the)f(transfer)h(of)f(an)g(arra)o(y)h(of)f (structures,)h(as)g(in)g(the)75 847 y(example)g(giv)o(en)h(for)e(an)g Fd(hvector)d Fi(comp)q(onen)o(t.)30 b(The)17 b(arra)o(y)h(is)f(co)o(v)o(ered) h(with)g(strided)g(v)o(ectors)f(with)h(homogeneous)75 897 y(comp)q(onen)o (ts;)c(the)f(fact)g(that)g(the)g(en)o(tire)h(data)g(structure)f(is)h(con)o (tiguous)h(in)f(memory)f(is)h(lik)o(ely)h(to)e(b)q(e)g(lost.)158 1029 y Fn(Bu\013er)i(descriptors)h(p)q(erform)d(t)o(w)o(o)g(functions)h(:)126 1112 y(1.)20 b(They)12 b(describ)q(e)h(the)f(bu\013er)g Fk(t)o(yp)q(e)p Fn(,)f(i.e.)16 b(the)c(data)f(t)o(yp)q(e)h(of)f(eac)o(h)h(elemen)o(t)f(of)g (the)g(bu\013er,)i(and)e(the)h(relativ)o(e)179 1162 y(displacemen)o(t)h(of)g (the)i(elemen)o(ts)e(in)h(memory)m(.)126 1245 y(2.)20 b(They)14 b(describ)q(e)i(the)e(address)h(in)f(memory)d(of)i(eac)o(h)i(elemen)o(t)e(of) g(the)i(bu\013er.)158 1328 y(Consider,)f(for)f(example,)g(the)h(follo)o(wing) d(fragmen)o(t)i(of)g(co)q(de.)75 1420 y Fl(int)21 b(a[10];)75 1469 y(...)75 1519 y(mpi_create_buffer)o(\(&b1,)d(MPI_EPHEMERAL\);)75 1569 y(mpi_append_contig)o(uous\()o(b1,)h(\(void)h(*\)a,)h(3,)h(MPI_INT\);)75 1619 y(mpi_create_buffer)o(\(&b2,)c(MPI_EPHEMERAL\);)75 1669 y(mpi_append_contig)o(uous\()o(b2,)h(\(void)h(*\)a,)h(3,)h(MPI_INT\);)75 1719 y(mpi_create_buffer)o(\(&b3,)c(MPI_EPHEMERAL\);)75 1768 y(mpi_append_contig)o(uous\()o(b3,)h(\(void)h(*\)\(a+2\),)h(3,)g(MPI_INT\);) 158 1860 y Fn(Then)c Fl(b1)f Fn(and)h Fl(b2)f Fn(b)q(oth)g(p)q(oin)o(t)g(to)h (bu\013er)g(descriptors)i(for)d(the)h(same)f(bu\013er,)h(whereas)h Fl(b3)e Fn(p)q(oin)o(ts)h(to)f(a)75 1909 y(bu\013er)f(descriptor)g(for)f (another)g(bu\013er;)g(ho)o(w)o(ev)o(er,)g(all)f(three)i(bu\013er)g (descriptors)g(ha)o(v)o(e)f(the)h(same)e(t)o(yp)q(e.)158 1959 y(In)f(order)g(to)g(pro)o(vide)f(a)h(recursiv)o(e)h(de\014nition)e(of)g (bu\013er)i(descriptors,)g(one)f(needs)h(an)f(ob)r(ject)g(that)g(sp)q (eci\014es)75 2009 y(the)g(bu\013er)g(t)o(yp)q(e,)f(but)g(not)g(its)g(lo)q (cation.)16 b(By)c(analogy)m(,)d(in)h(a)h(language)f(suc)o(h)i(as)f(C,)f(a)h (struct)h(de\014nition)f(pro)o(vides)75 2059 y(the)i(structure)i(t)o(yp)q(e,) e(and)g(its)g(declaration)f(pro)o(vides)h(\(implicitly\))d(the)k(address)g (of)e(the)h(structure.)20 b(The)14 b(struct)75 2109 y(de\014nition)f(can)i(b) q(e)f(used)h(recursiv)o(ely)g(as)f(a)f(comp)q(onen)o(t)g(of)h(a)f(new)i (struct)g(de\014nition.)158 2159 y(Rather)d(then)g(adding)e(new)i(functions)g (and)f(new)h(ob)r(jects,)h(w)o(e)e(prop)q(ose)i(to)e(use)h(bu\013er)h (descriptors)g(in)e(b)q(oth)75 2208 y(roles:)23 b(\\b)q(ound",)16 b(to)g(indicate)g(the)h(t)o(yp)q(e)g(and)f(lo)q(cation)g(of)f(a)h(bu\013er,)i (and)e(\\un)o(b)q(ound",)g(to)g(indicate)g(a)g(t)o(yp)q(e,)75 2258 y(that)e(can)g(b)q(e)h(used)f(as)g(a)g(comp)q(onen)o(t)f(of)g(a)h(new)g (bu\013er)h(de\014nition.)158 2308 y(F)m(ormally)m(,)d(the)k Fl(MPI)p 485 2308 14 2 v 15 w(APPEND)p 632 2308 V 14 w(xxx)f Fn(functions)h(detailed)f(ab)q(o)o(v)o(e)g(tak)o(e)h(a)f(handle)h(to)f(a)g (previously)g(de\014ned,)75 2358 y(uncommitted)8 b(bu\013er)k(descriptor)f (as)g(their)g Fl(datatype)e Fn(argumen)o(t)g(\(rather)i(than)g(a)f(state)h(v) n(ariable\).)16 b(The)11 b(\(com-)75 2408 y(p)q(osite\))j(elemen)o(t)g (describ)q(ed)h(b)o(y)f(this)g(argumen)o(t)f(is)h(an)f(elemen)o(t)h(of)f(the) h(same)f(t)o(yp)q(e)i(as)f(the)g(bu\013er)h(argumen)o(t.)158 2457 y(Example)d(2:)75 2549 y Fl(int)21 b(i[2])g(=)h({)f(2,)h(0};)75 2599 y(int)f(a[100];)75 2648 y(...)75 2698 y(mpi_create_buffer)o(\()e(&b1,)i (MPI_PERSISTENT\);)p eop %%Page: 13 14 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(13)75 45 y Fl(mpi_append_indexe)o(d\()19 b(b1,)i(\(void)g (*\)a,)g(2,)g(i,)h(MPI_INT\);)75 145 y(mpi_create_buffer)o(\()d(&b2,)i (MPI_PERSISTENT\);)75 195 y(mpi_append_contig)o(uous\()d(b2,)j(\(void)g (*\)a,)g(3,)h(b1\);)75 294 y(mpi_create_buffer)o(\()d(&b3,)i (MPI_PERSISTENT\);)75 344 y(mpi_append_indexe)o(d\()e(b3,)i(\(void)g(*\)a,)g (2,)g(i,)h(b2\);)75 394 y(mpi_append_vec\()d(b3,)i(\(void)g(*\)\(a+50\),)f (3,)h(3,)g(2,)h(b1\);)158 478 y(b1)12 b Fn(is)g(asso)q(ciated)h(with)f(a)g (bu\013er)h(consisting)g(of)e(the)i(t)o(w)o(o)f(elemen)o(ts)g Fl(a[2],)21 b(a[0])p Fn(.)c(When)12 b(used)h(un)o(b)q(ound,)75 528 y(it)k(indicates)h(a)f(comp)q(onen)o(t)f(with)h(t)o(w)o(o)g(in)o(tegers)h (that)f(are)h(one)g(in)o(teger)f(appart;)i(this)e(comp)q(onen)o(t)g(o)q (ccupies)75 578 y(three)e(in)o(teger)g(storage)f(units)g(\(12)f(b)o(ytes,)i (if)e(in)o(tegers)h(o)q(ccup)o(y)h(four)e(b)o(ytes\).)158 628 y(The)e(bu\013er)g(asso)q(ciated)g(with)e Fl(b2)h Fn(has)g(3)g(comp)q(onen)o (ts)g(of)g(the)g(t)o(yp)q(e)h(indicated)f(b)o(y)g Fl(b1)p Fn(.)16 b(It)11 b(consists)g(of)e(the)i(ar-)75 678 y(ra)o(y)g(en)o(tries)h Fl(a[2],)21 b(a[0],)f(a[5],)h(a[3],)g(a[8],)g(a[6])p Fn(.)16 b(When)11 b(used)h(un)o(b)q(ound,)f(it)g(indicates)g(a)g(comp)q(onen)o(t)75 727 y(with)j(six)f(in)o(tegers)i(that)f(o)q(ccup)o(y)g(nine)g(in)o(teger)h (storage)f(units.)158 777 y(The)h(bu\013er)h(asso)q(ciated)g(with)f Fl(b3)f Fn(has)h(t)o(w)o(o)g(comp)q(onen)o(t)f(of)g(the)i(t)o(yp)q(e)f (indicated)g(b)o(y)g Fl(b2)f Fn(and)h(three)h(com-)75 827 y(p)q(onen)o(ts)f (of)e(the)i(t)o(yp)q(e)f(indicated)g(b)o(y)f Fl(b1)p Fn(.)18 b(It)c(consists)h(of)e(the)i(arra)o(y)e(en)o(tries)75 905 y Fl(a[20],)21 b(a[18],)f(a[23],)h(a[21],)f(a[26],)h(a[24],)97 954 y(a[2],)42 b(a[0],)h(a[5],)f(a[3],)h(a[8],)g(a[6],)75 1004 y(a[52],)21 b(a[50],)f(a[55],)h(a[53],)f(a[59],)h(a[57].)158 1082 y Fn(An)o(y)15 b(uncommitted)e(bu\013er)k(descriptor)f(ma)o(y)d(b)q(e)j (used)g(as)g(a)f(data)g(t)o(yp)q(e.)22 b(MPI)16 b(prede\014nes)h(\(handles)f (to\))75 1132 y(uncommitted)10 b(bu\013er)k(descriptors)g Fl(MPI)p 723 1132 14 2 v 16 w(INT)p Fn(,)d Fl(MPI)p 894 1132 V 15 w(FLOAT)p Fn(,)g(etc.,)i(for)g(eac)o(h)g(basic)g(datat)o(yp)q(e)f(of)g(F)m(ortran)h (and)f(C.)75 1181 y(Eac)o(h)f(of)f(these)i(descriptors)g(can)f(b)q(e)g (though)o(t)g(as)f(describing)i(a)e(bu\013er)i(with)e(one)h(elemen)o(t)f(of)g (the)h(corresp)q(onding)75 1231 y(t)o(yp)q(e.)158 1281 y(When)g(a)f(bu\013er) i(descriptor)g(isd)e(used)i(as)e(a)h(t)o(yp)q(e)g(descriptor)h(then)f(MPI)g (ensures)h(the)f(alignmen)o(ts)e(required)75 1331 y(b)o(y)14 b(the)g(underlying)g(arc)o(hitecture)h(and)f(add,)f(if)g(required,)i(padding) e(space.)158 1381 y(Example)f(3:)75 1430 y(Consider)i(the)h(follo)o(wing)c(C) j(co)q(de)h(whic)o(h)e(builds)h(a)f(bu\013er)i(descriptor)g(for)f(an)g(arra)o (y)f(of)h(structures.)75 1558 y Fl(/*)21 b(A)h(brain)f(damaged)f(structure)g (which)h(needs)g(both)f(internal)h(and)g(external)97 1608 y(*)g(padding)g(on) g(a)h(32)f(bit)g(machine)g(with)g(strict)f(alignment)g(requirements)97 1658 y(*/)75 1707 y(struct)h(foostruct)e({)140 1757 y(int)j(i;)140 1807 y(/*)g(four)f(bytes)f(of)i(padding)e(to)i(get)f(back)g(to)g(double)g (alignment)f(*/)140 1857 y(double)h(d[5];)140 1907 y(char)g(b;)140 1956 y(/*)h(trailing)e(7)h(bytes)g(of)h(padding)e(to)h(enforce)g(alignment)f (again)g(*/)75 2006 y(};)75 2106 y(struct)h(foostruct)e(foo[100];)75 2205 y(MPI_BUFFER)h(t;)75 2255 y(MPI_BUFFER)g(fb;)75 2355 y (mpi_create_buffer)o(\(&t,)40 b(MPI_EPHEMERAL\);)75 2405 y(mpi_create_buffer) o(\(&fb,)18 b(MPI_PERSISTENT\);)75 2504 y(/*)j(Describe)g(the)g(data)g(type)g (in)g(t)h(*/)75 2554 y(mpi_append_contig)o(uous\()o(t,)d(&\(\(struct)h (foostruct)g(*\)0\)->i,1,MPI_IN)o(T\);)75 2604 y(mpi_append_contig)o(uous\()o (t,)f(&\(\(struct)h(foostruct)g(*\)0\)->d,5,MPI_DO)o(UBLE\))o(;)75 2654 y(mpi_append_contig)o(uous\()o(t,)f(&\(\(struct)h(foostruct)g (*\)0\)->b,1,MPI_BY)o(TE\);)p eop %%Page: 14 15 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(14)75 45 y Fl(/*)21 b(Now)h(describe)e(foo)h(itself,)f(a)i (contiguous)e(array)g(of)i(100)f(elements)97 95 y(*)g(of)h(the)f(derived)f (type.)97 145 y(*/)75 195 y(mpi_append_contig)o(uous\()o(fb,&f)o(oo,10)o (0,t\);)75 244 y(mpi_commit\(fb\);)158 324 y Fn(Example)12 b(4:)75 374 y(Consider)17 b(the)g(follo)o(wing)d(co)q(de)k(whic)o(h)f (expresses)i(as)e(a)f(single)g(bu\013er)i(descriptor)g(a)e(subset)j(of)d(an)g (arbitrary)75 424 y(dimensional)c(arra)o(y)m(.)162 503 y Fl(real)21 b(a\(100,100,100\))75 553 y(c)65 b(declare)21 b(the)g(buffer)f(descriptors) 162 603 y(integer)h(onedslice)162 653 y(integer)g(twodslice)162 702 y(integer)g(message)75 802 y(c)65 b(Slice)21 b(out)g(the)g(section)g (a\(1:21:2,1:10,1:)o(11\))d(as)k(a)f(single)g(buffer)g(descriptor)162 902 y(call)g(mpi_create_buffer\()o(ondes)o(lice,)d(MPI_EPHEMERAL\))162 951 y(call)j(mpi_create_buffer\()o(twods)o(lice,)d(MPI_EPHEMERAL\))162 1001 y(call)j(mpi_create_buffer\()o(messa)o(ge,)62 b(MPI_EPHEMERAL\))162 1101 y(call)21 b(mpi_append_vec)e(\(onedslice,)h(a,)h(1+\(21-1\)/2,)140 1151 y($)458 b(2,)21 b(1,)h(MPI_REAL\))162 1200 y(call)f(mpi_append_hvec\(tw) o(odsli)o(ce,)e(a,)195 b(10,)140 1250 y($)458 b(100*MPI_REAL_SIZE)o(,)19 b(1,)i(ondeslice\))162 1300 y(call)g(mpi_append_hvec\(me)o(ssage)o(,)63 b(a,)348 b(11,)140 1350 y($)458 b(100*100*MPI_REAL_)o(SIZE)o(,1,tw)o(odsli)o (ce\))162 1450 y(call)21 b(mpi_commit\(message)o(\))75 1549 y(c)65 b(all)21 b(done...)158 1707 y Fj(Discussion:)158 1753 y Fi(F)m(or)16 b(p)q(ortabilit)o(y)i(this)f(example)g(requires)g(access)f(to) g(the)g(size)g(in)h(b)o(ytes)f(of)g(an)g(MPI)f(data)i(t)o(yp)q(e.)25 b(\(In)16 b(general)h(a)75 1799 y(bu\013er)d(descriptor\).)k(W)m(e)c(should)g (probably)h(pro)o(vide)g(an)e Fd(MPI)p 995 1799 12 2 v 13 w(SIZEOF\()k (bufhandle)f(\))d Fi(function)h(to)f(ac)o(hiev)o(e)i(this.)158 2010 y Fj(Discussion:)158 2055 y Fi(W)m(e)g(should)i(probably)f(restrict)g(a) f(bu\013er)g(descriptor)i(that)e(is)g(used)h(as)f(a)f(t)o(yp)q(e)i(parameter) f(to)g(describ)q(e)h(a)f(bu\013er)75 2101 y(that)g(is)g(con)o(tained)h (within)g(one)f(\(comp)q(osite\))h(v)n(ariable)g(\(arra)o(y)m(,)f(structure,) g(etc.\).)21 b(The)14 b(w)o(orry)g(is)i(that)e(on)h(a)f(mac)o(hine)75 2147 y(with)i(a)g(segmen)o(ted)h(address)g(space)f(di\013eren)o(t)i(v)n (ariables)g(ma)o(y)e(not)g(b)q(elong)h(to)f(the)g(same)g(address)h(segmen)o (t,)g(so)f(that)75 2192 y(their)f(relativ)o(e)i(displacemen)o(t)g(is)e(not)g (de\014ned)h(\(think)f(of)g(a)f(bu\013er)i(that)f(con)o(tains)h(a)e(heap)i(v) n(ariable,)g(a)f(stac)o(k)g(v)n(ariable)75 2238 y(and)f(a)f(static)g(v)n (ariable\).)75 2428 y Fk(Alternativ)o(e)g(within)g(alternativ)o(e)75 2504 y Fn(A)g(v)n(arian)o(t)f(on)h(the)g(previous)h(prop)q(osal)e(is)h(to)g (k)o(eep)h(un)o(b)q(ound)f(bu\013er)h(descriptors)g(syn)o(tactically)e (distinct)i(from)75 2554 y(b)q(ound)d(bu\013er)i(descriptors.)19 b(A)11 b(bu\013er)h(descriptor)h(is)e Fk(un)o(b)q(ound)e Fn(if)h(it)h(is)g (not)g(asso)q(ciated)i(with)e(a)g(\014xed)g(memory)75 2604 y(address.)23 b(A)15 b(bu\013er)h(descriptor)g(can)f(b)q(e)h(used)g(as)f(a)f (datat)o(yp)q(e)i(if)e(and)h(only)f(if)g(it)g(is)h(un)o(b)q(ound.)22 b(This)15 b(prev)o(en)o(ts)75 2654 y(the)f(p)q(ossible)g(confusion)g(arising) e(from)g(using)i(the)g(same)f(ob)r(ject)h(for)g(t)o(w)o(o)f(di\013eren)o(t)h (functions,)g(one)g(where)g(the)75 2704 y(starting)g(address)h(is)f (signi\014can)o(t)f(and)h(one)g(where)h(the)g(starting)e(address)j(should)d (b)q(e)i(ignored.)p eop %%Page: 15 16 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(15)158 45 y(An)20 b(un)o(b)q(ound)g(bu\013er)h(descriptor)g (is)e(created)i(b)o(y)f(using)f(the)i(sp)q(ecial)f Fl(MPI)p 1400 45 14 2 v 15 w(NULL)f Fn(v)n(alue)g(as)h(the)g Fl(start)75 95 y Fn(parameter)14 b(in)f Fl(MPI)p 391 95 V 15 w(APPEND)p 538 95 V 15 w(xxx)g Fn(op)q(erations)h(that)g(create)h(this)f(descriptor.)20 b(An)14 b(un)o(b)q(ound)g(bu\013er)h(descriptor)75 145 y(can)f(also)f(b)q(e)i (created)g(b)o(y)f Fk(un)o(bind)o(in)o(g)d Fn(a)j(b)q(ound)g(bu\013er)g (descriptor.)158 230 y Fk(MPI)p 257 230 15 2 v 17 w(UNBIND\(handlein,)f (handleout\))75 357 y(IN)j(handlein)i Fn(handle)c(to)f(b)q(ound)h(bu\013er)h (descriptor)75 440 y Fk(OUT)h(handleout)h Fn(handle)g(to)g(un)o(b)q(ound)g (bu\013er)h(descriptor)g(with)f(the)g(datat)o(yp)q(e)g(of)g(the)g(input)g (bu\013er)h(de-)179 490 y(scriptor)158 659 y Fj(Discussion:)158 705 y Fi(The)12 b Fd(MPI)p 298 705 12 2 v 13 w(UNBIND)e Fi(function)k(allo)o (ws)g(to)e(create)h(a)g(bu\013er)g(datat)o(yp)q(e)h(after)e(the)g(pattern)h (of)g(an)f(existing)j(bu\013er)e(and,)75 751 y(therefor,)g(after)g(the)g (pattern)g(of)g(an)g(existing)j(data)d(structure)h(in)f(memory)m(.)158 883 y Fn(W)m(e)h(rep)q(eat)h(the)f(last)g(three)h(examples,)d(using)i(this)g (alternativ)o(e)g(notation.)158 933 y(Example)e(2':)75 1024 y Fl(int)21 b(i[2])g(=)h({)f(2,)h(0};)75 1074 y(int)f(a[100];)75 1124 y(...)75 1174 y(mpi_create_buffer)o(\()e(&b1,)i(MPI_PERSISTENT\);)75 1224 y(mpi_append_indexe)o(d\()e(b1,)i(MPI_NULL,)f(2,)h(i,)h(MPI_INT\);)75 1323 y(mpi_create_buffer)o(\()d(&b2,)i(MPI_PERSISTENT\);)75 1373 y(mpi_append_contig)o(uous\()d(b2,)j(MPI_NULL,)f(3,)i(b1\);)75 1473 y(mpi_create_buffer)o(\()d(&b3,)i(MPI_PERSISTENT\);)75 1523 y(mpi_append_indexe)o(d\()e(b3,)i(\(void)g(*\)a,)g(2,)g(i,)h(b2\);)75 1572 y(mpi_append_vec\()d(b3,)i(\(void)g(*\)\(a+50\),)f(3,)h(3,)g(2,)h(b1\);) 158 1664 y Fn(Example)12 b(3:)75 1846 y Fl(struct)21 b(foostruct)e({)140 1896 y(int)j(i;)140 1946 y(double)f(d[5];)140 1996 y(char)g(b;)75 2046 y(};)75 2145 y(struct)g(foostruct)e(foo[100];)75 2245 y(MPI_BUFFER)h(t,)h(tt;)75 2295 y(MPI_BUFFER)f(fb;)75 2394 y(mpi_create_buffer)o(\(&t,)40 b(MPI_EPHEMERAL\);)75 2444 y (mpi_create_buffer)o(\(&fb,)18 b(MPI_PERSISTENT\);)75 2544 y(/*)j(Describe)g(the)g(data)g(type)g(in)g(t)h(*/)75 2593 y (mpi_append_contig)o(uous\()o(t,)d(&\(\(struct)h(foostruct)g (*\)0\)->i,1,MPI_IN)o(T\);)75 2643 y(mpi_append_contig)o(uous\()o(t,)f (&\(\(struct)h(foostruct)g(*\)0\)->d,5,MPI_DO)o(UBLE\))o(;)75 2693 y(mpi_append_contig)o(uous\()o(t,)f(&\(\(struct)h(foostruct)g (*\)0\)->b,1,MPI_BY)o(TE\);)p eop %%Page: 16 17 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(16)75 95 y Fl(/*)21 b(Abstract)g(buffer)f(datatype)g(*/)75 145 y(mpi_unbind\(t,)f(tt\);)75 244 y(/*)i(Now)h(describe)e(foo)h(itself,)f (a)i(contiguous)e(array)g(of)i(100)f(elements)97 294 y(*)g(of)h(the)f (derived)f(type.)97 344 y(*/)75 394 y(mpi_append_contig)o(uous\()o(fb,&f)o (oo,10)o(0,tt\))o(;)75 444 y(mpi_commit\(fb\);)158 521 y Fn(Example)12 b(4:)162 649 y Fl(real)21 b(a\(100,100,100\))75 699 y(c)65 b(declare)21 b(the)g(buffer)f(descriptors)162 749 y(integer)h(onedslice)162 799 y(integer)g(twodslice)162 848 y(integer)g(message)75 948 y(c)65 b(Slice)21 b(out)g(the)g(section)g(a\(1:21:2,1:10,1:)o(11\))d(as)k(a)f (single)g(buffer)g(descriptor)162 1048 y(call)g(mpi_create_buffer\()o(ondes)o (lice,)d(MPI_EPHEMERAL\))162 1097 y(call)j(mpi_create_buffer\()o(twods)o (lice,)d(MPI_EPHEMERAL\))162 1147 y(call)j(mpi_create_buffer\()o(messa)o(ge,) 62 b(MPI_EPHEMERAL\))162 1247 y(call)21 b(mpi_append_vec)e(\(onedslice,)h (MPI_NULL,)f(1+\(21-1\)/2,)140 1297 y($)458 b(2,)21 b(1,)h(MPI_REAL\))162 1346 y(call)f(mpi_append_hvec\(tw)o(odsli)o(ce,)e(MPI_NULL,)41 b(10,)140 1396 y($)458 b(100*MPI_REAL_SIZE)o(,)19 b(1,)i(ondeslice\))162 1446 y(call)g(mpi_append_hvec\(me)o(ssage)o(,)e(a,)239 b(11,)140 1496 y($)458 b(100*100*MPI_REAL_)o(SIZE)o(,1,tw)o(odsli)o(ce\))162 1596 y(call)21 b(mpi_commit\(message)o(\))75 1695 y(c)65 b(all)21 b(done...)75 1802 y Fk(T)o(yp)q(e)16 b(matc)o(hing)75 1879 y Fn(The)j(rules)h(for)f(bu\013er)h(t)o(yp)q(e)f(matc)o(hing)e(are)j (extended)g(to)f(comp)q(osite)f(bu\013er)i(descriptors:)30 b(A)19 b(send)h(bu\013er)75 1928 y(descriptor)15 b(matc)o(hes)f(a)g(receiv)o (e)h(bu\013er)g(descriptor)g(is)f(the)h(successiv)o(e)h(basic)e(en)o(tries)h (in)e(eac)o(h)i(descriptor)g(ha)o(v)o(e)75 1978 y(iden)o(tical)e(names.)158 2028 y(Example:)75 2106 y Fl(struct)21 b({)140 2156 y(int)h(a[5];)140 2205 y(char)f(c;)228 2255 y(})g(st[100];)75 2355 y(...)75 2405 y(mpi_create_buffer)o(\(&b1,)d(MPI_PERSISTENT\);)75 2455 y(mpi_append_contig) o(uous\()o(b1,)h(\(void)h(*\)st,)h(5,)h(MPI_INT\);)75 2504 y(mpi_append_contig)o(uous\()o(b1,)d(\(void)h(*\)&\(st.c\),)g(1,)h (MPI_CHAR\);)75 2554 y(mpi_append_contig)o(uous\()o(b1,)e(\(void)h(*\)st,)h (5,)h(MPI_INT\);)75 2604 y(mpi_append_contig)o(uous\()o(b1,)d(\(void)h (*\)&\(st.c\),)g(1,)h(MPI_CHAR\);)75 2654 y(...)75 2704 y(mpi_create_buffer)o (\(&b2,)d(MPI_PERSISTENT\);)p eop %%Page: 17 18 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(17)75 45 y Fl(mpi_append_contig)o(uous\()o(b2,)19 b(\(void)h(*\)st,)h(5,)h(MPI_INT\);)75 95 y(mpi_append_contig)o(uous\()o(b2,) d(\(void)h(*\)&\(st.c\),)g(1,)h(MPI_CHAR\);)75 195 y(mpi_create_buffer)o (\(&b3,)d(MPI_PERSISTENT\);)75 244 y(mpi_append_contig)o(uous\()o(b3,)h (\(void)h(*\)st,)h(2,)h(b2\);)158 327 y Fn(The)15 b(bu\013er)g(descriptors)i (asso)q(ciated)e(with)f Fl(b1)g Fn(and)g Fl(b3)g Fn(ha)o(v)o(e)h(matc)o(hing) d(t)o(yp)q(es.)21 b(They)15 b(b)q(oth)f(corresp)q(ond)75 377 y(to)19 b(a)g(sequence)i(of)e Fl(int,)i(int,)g(char,)f(int,)h(int,)g(char)p Fn(.)33 b(\(in)19 b(fact,)h(b)q(oth)f(bu\013ers)i(consist)f(of)e(the)i(same) 75 427 y(sequence)c(of)d(v)n(ariables\).)158 555 y Fj(Discussion:)158 601 y Fi(A)f(more)g(restrictiv)o(e)i(de\014nition)g(of)e(t)o(yp)q(e)h(matc)o (hing)g(is)g(to)f(require)i(\\structural)f(matc)o(hing":)18 b(I.e.,)11 b(t)o(w)o(o)h(descriptors)75 647 y(matc)o(h)k(if)f(all)i(corresp)q (onding)h(top)d(\(comp)q(osite\))i(comp)q(onen)o(ts)g(recursiv)o(ely)g(matc)o (h)f(pairwise.)25 b(With)17 b(suc)o(h)e(de\014nition,)75 692 y(the)e(t)o(w)o(o)g(bu\013er)h(descriptors)h(in)f(the)f(last)h(example)h (don't)e(ha)o(v)o(e)h(matc)o(hing)h(t)o(yp)q(es.)j(I)13 b(\(Marc\))g(b)q (eliev)o(e)i(that)e(the)h(w)o(eak)o(er)75 738 y(de\014nition)20 b(mak)o(es)d(more)g(sense)g(in)h(our)f(en)o(vironmen)o(t,)j(since)e(w)o(e)e (do)i(not,)f(in)h(general,)h(require)f(an)o(y)g(corrsp)q(ondance)75 784 y(b)q(et)o(w)o(een)13 b(data)h(alignemen)o(t)h(at)e(sender)h(and)f(data)h (alignmen)o(t)h(at)e(receiv)o(er.)158 916 y Fn(T)o(yp)q(e)h(matc)o(hing)e(of) i(bu\013er)g(descriptor)i(and)d(host)i(language)d(is)i(dealt)g(similarly)m(.) 75 1024 y Fk(Message)i(length)75 1101 y Fn(The)10 b(length)g(of)f(a)g (message)g(is)g(de\014ned)i(as)f(the)g(n)o(um)o(b)q(er)f(of)g(basic)g(elemen) o(ts)h(\(comp)q(onen)o(ts)f(with)h(basic)f(datat)o(yp)q(es\))75 1150 y(in)k(the)i(message.)j(This)13 b(is)h(the)h(v)n(alue)e(returned)i(in)f (the)g(status)h(of)e(a)h(receiv)o(e)h(op)q(eration)f(for)f(that)h(message.) 158 1279 y Fj(Discussion:)158 1324 y Fi(An)f(alternativ)o(e)i(de\014nition)h (is)e(to)f(coun)o(t)h(message)f(length)i(as)e(the)g(n)o(um)o(b)q(er)h(of)f (top)h(comp)q(onen)o(ts)g(in)g(the)f(message.)75 1370 y(If)j(w)o(e)g(accept)g (the)h(w)o(eak)o(er)f(de\014nition)j(of)e(t)o(yp)q(e)f(compatibili)q(t)o(y)m (,)k(this)d(has)g(the)f(e\013ect)h(that)f(the)h(length)h(of)e(a)g(message)75 1416 y(sen)o(t)e(ma)o(y)h(b)q(e)f(v)o(ery)h(di\013eren)o(t)g(from)f(the)g (length)i(of)e(the)g(message)h(receiv)o(ed)g({)f(ev)o(en)h(in)g(a)f (homogeneous)i(en)o(vironmen)o(t.)75 1461 y(Consider)g(in)g(the)f(previous)h (example)h(the)d(case)i(of)e(message)i(sen)o(t)f(with)g Fd(b1)f Fi(and)h(receiv)o(ed)h(with)g Fd(b3)d Fi({)i(the)g(length)h(has)75 1507 y(halv)o(ed.)158 1557 y(On)d(the)g(other)h(hand,)g(if)f(w)o(e)g(accept)g (the)g(stronger)h(de\014nition)i(of)d(t)o(yp)q(e)g(compatibil)q(it)o(y)m(,)j (coun)o(ting)f(top)e(elemen)o(ts)h(is)75 1607 y(plausible.)20 b(Since)13 b(top)f(comp)q(onen)o(ts)h(are)f(required)h(to)f(matc)o(h)g (pairwise,)h(the)f(message)h(length)g(is)f(in)o(teger,)h(and)f(iden)o(tical) 75 1657 y(at)h(sending)i(and)e(receiving)j(sites.)75 1847 y Fk(Bu\013er)f(descriptors)e(as)i(datat)o(yp)q(es)g(in)g(comm)o(unicati)o(on)e (op)q(erations)75 1924 y Fn(W)m(e)h(prop)q(ose)h(to)f(allo)o(w)e(the)j(use)g (of)e(bu\013er)i(descriptor)h(handles)e(as)g(datat)o(yp)q(es)h(in)f(con)o (tiguous)g(bu\013er)h(comm)o(u-)75 1974 y(nication)e(op)q(erations)h(\(e.g.)k (in)c Fl(MPI)p 651 1974 14 2 v 15 w(SENDC)p Fn(\))158 2023 y(If)i(w)o(e)h(do)f(so,)h(then)g(w)o(e)g(don't)f(need)h(an)o(ymore)e(the)i (more)f(general)g(comm)o(unication)d(op)q(erations)k(with)f(a)75 2073 y Fl(buffer)p 210 2073 V 14 w(handle)d Fn(argumen)o(t!!!)75 2123 y Fl(MPI)p 144 2123 V 15 w(SEND\(buffer)p 401 2123 V 13 w(handle,)21 b(dest,)g(tag,)g(context\))75 2173 y Fn(is)14 b(replaced)h(b)o(y)75 2223 y Fl(MPI)p 144 2223 V 15 w(SENDC\(start,)k(1,)j (buffer)p 640 2223 V 14 w(handle,)f(dest,)f(tag,)h(context\))p Fn(,)75 2272 y(where)14 b Fl(start)d Fn(is)i(the)g(\014rst)g(bu\013er)h(lo)q (cation.)i(\(If)d(un)o(b)q(ound)f(bu\013er)i(descriptors)g(are)f(k)o(ept)g (distinct)g(from)e(b)q(ound)75 2322 y(bu\013er)j(descriptors,)g(then)f(the)h (argumen)o(t)d Fl(buffer)p 897 2322 V 15 w(handle)g Fn(gets)j(replaced)f(b)o (y)g(the)g(un)o(b)q(ound)g(datat)o(yp)q(e)g(of)f(the)75 2372 y(bu\013er.\))158 2422 y(Suc)o(h)f(simpli\014cation)d(w)o(ould)i(b)q(e)h(a)f (big)g(win,)h(as)f(comm)o(unication)d(with)k(general)g(bu\013ers)h(b)q (ecomes)e(a)h(sp)q(ecial)75 2472 y(case)18 b(of)e(comm)o(unicatio)o(n)e(with) j(con)o(tiguous)f(bu\013ers.)28 b(One)18 b(ca)o(v)o(eat:)23 b(When)17 b(a)g(bu\013er)h(descriptor)g(is)e(used)i(as)75 2522 y(a)d(datat)o(yp)q(e,)g(w)o(e)g(ma)o(y)e(ha)o(v)o(e)i(restrictions)h(on)f (where)h(from)d(the)j(di\013eren)o(t)g(v)n(ariables)e(app)q(earing)h(in)f (the)i(bu\013er)75 2571 y(descriptor)e(come)f(from:)j(They)d(need)h(to)f(all) f(come)g(from)g(the)i(same)e(con)o(tiguous)h(address)h(sap)q(ce,)g(so)f(that) h(their)75 2621 y(relativ)o(e)i(displacemen)o(t)g(is)g(w)o(ell)f(de\014ned.) 27 b(It)16 b(means)g(that)g(w)o(e)h(shall)e(only)h(b)q(e)g(able)g(to)h(pac)o (k)f(in)g(one)g(message)75 2671 y(v)n(ariables)c(that)i(b)q(elong)f(to)g(the) g(same)g(con)o(tiguous)g(address)h(space.)19 b(W)m(e)13 b(ha)o(v)o(e)g(to)g (think)f(ab)q(out)i(syn)o(tactic)f(rules)p eop %%Page: 18 19 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(18)75 45 y(to)15 b(enforce)g(this)g(restriction:)21 b(E.g.,)14 b(is)g(it)h(su\016cien)o(t)g(to)g(sa)o(y)f(in)g(C)h(that)g(a)g (bu\013er)g(is)g(either)h(all)d(in)h(stac)o(k,)h(or)g(all)75 95 y(in)e(heap,)h(or)g(all)f(in)g(static?)19 b(What)13 b(is)h(the)g(equiv)n (alen)o(t)f(F)m(ortran)h(rule?)k(Are)d(suc)o(h)g(rules)f(acceptable?)158 227 y Fj(Discussion:)35 b Fi(OPINIONS?)75 418 y Fk(Other)14 b(issues)75 495 y Fn(An)i(ephemeral)f(bu\013er)h(descriptor)h(can)f(b)q(e)g (freed)h(as)e(so)q(on)h(as)g(it)f(has)h(b)q(een)g(used)h(as)e(a)h(t)o(yp)q(e) g(parameter)f(in)g(a)75 544 y(bu\013er)g(descriptor)g(creation)f(function.) 158 594 y(A)j(bu\013er)g(descriptor)h(handle)e(cannot)h(b)q(e)g(passed)g(to)f (the)h(same)f Fl(MPI)p 1295 594 14 2 v 15 w(APPEND)p 1442 594 V 14 w(xxx)g Fn(function)g(b)q(oth)h(as)f(a)75 644 y(bu\013er)f(descriptor)g (handle)f(and)g(as)g(a)f(datat)o(yp)q(e)h(\(no)g(self-references\).)158 777 y Fj(Discussion:)35 b Fi(The)13 b(last)h(constrain)o(t)g(can)g(b)q(e)f (remo)o(v)o(ed)h(if)f(someb)q(o)q(dy)h(cares.)75 975 y Fh(1.5.4)55 b(T)n(yp)r(e)18 b(Matc)n(hing)75 1052 y Fn(One)d(can)f(think)f(of)h(message)f (transmission)g(as)h(consisting)g(of)f(three)i(phases:)126 1135 y(1.)20 b(Data)13 b(is)h(pulled)f(out)h(of)f(the)i(send)g(bu\013er)f (and)g(a)g(message)f(is)h(assem)o(bled)126 1218 y(2.)20 b(A)14 b(message)f(is)h(transferred)i(from)c(sender)j(to)f(receiv)o(er)126 1301 y(3.)20 b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming)c(message)j(and) g(disassem)o(bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 1384 y(T)o(yp)q(e)c(matc)o(hing)e(has)i(to)f(b)q(e)h(observ)o(ed)h(at)f(eac)o (h)g(of)f(these)i(three)g(phases:)17 b(The)11 b(t)o(yp)q(e)g(of)f(eac)o(h)i (v)n(ariable)d(in)h(the)75 1434 y(sender)15 b(bu\013er)g(has)f(to)g(matc)o(h) f(the)h(t)o(yp)q(e)g(of)g(the)g(corresp)q(onding)h(en)o(try)f(in)g(the)g (bu\013er)h(descriptor;)g(the)f(t)o(yp)q(e)h(of)75 1484 y(an)e(en)o(try)g(in) f(the)h(bu\013er)h(descriptor)g(used)g(to)e(send)i(a)f(message)f(has)h(to)f (matc)o(h)g(the)h(t)o(yp)q(e)h(of)e(the)h(corresp)q(onding)75 1534 y(en)o(try)f(in)e(the)i(bu\013er)g(descriptor)g(used)g(to)f(receiv)o(e)h (the)g(message;)f(and)g(the)h(t)o(yp)q(e)f(of)g(eac)o(h)g(v)n(ariable)f(in)h (the)h(receiv)o(e)75 1583 y(bu\013er)j(has)g(to)f(matc)o(h)f(the)i(corresp)q (onding)h(en)o(try)f(in)e(the)i(receiv)o(er)h(bu\013er)g(descriptor.)k(A)15 b(program)e(that)h(fails)75 1633 y(to)g(observ)o(e)h(these)g(three)g(rules)g (is)e(erroneous.)158 1683 y(T)m(o)h(de\014ne)h(t)o(yp)q(e)g(matc)o(hing)d (more)h(precisely)m(,)i(w)o(e)f(need)h(to)g(deal)f(with)g(t)o(w)o(o)f (issues:)20 b(matc)o(hing)13 b(of)g(t)o(yp)q(es)j(of)75 1733 y(the)h(host)g(language)f(with)g(bu\013er)h(descriptor)h(t)o(yp)q(es;)g(and)f (matc)o(hing)d(of)i(bu\013er)i(descriptor)g(t)o(yp)q(es)f(at)f(sender)75 1783 y(and)c(receiv)o(er.)18 b(W)m(e)12 b(restrict)h(ourselv)o(es)g(to)f(the) g(case)h(where)g(the)f(sending)g(and)g(receiving)g(programs)e(are)i(written) 75 1832 y(in)h(the)i(same)e(language.)158 1882 y(The)f(t)o(yp)q(e)h(of)e(an)h (en)o(try)g(in)g(the)g(sender)i(bu\013er)f(descriptor)g(matc)o(hes)f(the)g(t) o(yp)q(e)h(of)e(an)h(en)o(try)g(in)g(the)g(receiv)o(er)75 1932 y(bu\013er)e(descriptor)h(if)e(they)h(ha)o(v)o(e)f(iden)o(tical)g(names:)15 b Fl(MPI)p 966 1932 V 15 w(INTEGER)8 b Fn(matc)o(hes)h Fl(MPI)p 1365 1932 V 15 w(INTEGER)p Fn(,)f Fl(MPI)p 1620 1932 V 15 w(REAL)g Fn(matc)o(hes)75 1982 y Fl(MPI)p 144 1982 V 15 w(REAL)p Fn(,)13 b(and)g(so)h(on.)158 2032 y(The)i(t)o(yp)q(e)g(of)e(a)h(v)n(ariable)g(in)g(a) g(host)g(program)f(matc)o(hes)h(the)h(t)o(yp)q(e)g(of)e(the)i(en)o(try)g(in)f (a)g(bu\013er)i(descriptor)75 2082 y(if)e(the)i(name)e(of)g(the)i(bu\013er)g (descriptor)g(en)o(try)g(t)o(yp)q(e)g(corresp)q(onds)h(to)e(the)g(basic)g(t)o (yp)q(e)h(of)f(the)g(host)h(program)75 2131 y(v)n(ariable:)24 b(an)17 b(en)o(try)h(with)f(t)o(yp)q(e)h(name)e Fl(MPI)p 805 2131 V 15 w(INTEGER)g Fn(matc)o(hes)h(a)g(F)m(ortran)g(v)n(ariable)f(of)h(t)o (yp)q(e)h Fl(INTEGER)p Fn(,)d(an)75 2181 y(en)o(try)h(with)f(t)o(yp)q(e)h (name)e Fl(MPI)p 557 2181 V 15 w(REAL)g Fn(matc)o(hes)h(a)g(F)m(ortran)g(v)n (ariable)f(of)h(t)o(yp)q(e)h Fl(REAL)p Fn(,)e(and)h(so)g(on.)22 b(There)17 b(is)e(one)75 2231 y(exception)i(to)f(this)g(last)f(rule:)23 b(An)16 b(en)o(try)g(with)g(t)o(yp)q(e)h(name)d Fl(MPI)p 1137 2231 V 15 w(BYTE)i Fn(can)g(b)q(e)g(used)h(to)f(matc)o(h)f(an)o(y)g(b)o(yte)i (of)75 2281 y(storage)f(\(on)f(a)g(b)o(yte-addressable)h(mac)o(hine\),)e (irresp)q(ectiv)o(e)j(of)d(the)i(datat)o(yp)q(e)f(of)g(the)h(v)n(ariable)e (that)h(con)o(tains)75 2331 y(this)j(b)o(yte.)30 b(The)18 b(v)n(alue)g(of)f (the)h(message)g(en)o(try)g(will)e(b)q(e)j(the)f(binary)g(v)n(alue)f(of)g (the)h(corresp)q(onding)h(b)o(yte)f(in)75 2380 y(memory)m(.)158 2430 y(W)m(e)c(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)137 2513 y Fc(\017)21 b Fn(Comm)o(uni)o(cation)d(of)j(t)o(yp)q(ed)h(v)n(alues)g (\(e.g.,)g(with)f(datat)o(yp)q(e)h(di\013eren)o(t)g(from)e Fl(MPI)p 1541 2513 V 15 w(BYTE)p Fn(\),)h(where)h(the)179 2563 y(datat)o(yp)q(es)c(of)f(the)i(corresp)q(onding)f(en)o(tries)h(in)e(the)h (sender)i(program,)c(in)i(the)g(sender)h(bu\013er)g(descrip-)179 2613 y(tor,)13 b(in)h(the)g(receiv)o(er)i(bu\013er)f(descriptor)g(and)e(in)h (the)g(receiv)o(er)i(program)c(should)i(all)e(matc)o(h.)p eop %%Page: 19 20 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(19)137 45 y Fc(\017)21 b Fn(Comm)o(uni)o(cation)12 b(of)i(un)o(t)o(yp)q(ed)i(v)n(alues)e(\(e.g.,)g(of)h(datat)o(yp)q(e)g Fl(MPI)p 1214 45 14 2 v 15 w(BYTE)p Fn(\),)f(where)i(the)f(the)h(corresp)q (onding)179 95 y(en)o(tries)f(in)e(the)i(sender)g(and)f(the)g(receiv)o(er)h (bu\013er)g(descriptors)h(ha)o(v)o(e)d(b)q(oth)h(datat)o(yp)q(e)g Fl(MPI)p 1631 95 V 16 w(BYTE)p Fn(.)e(In)i(this)179 145 y(case,)k(there)g (are)f(no)g(requiremen)o(ts)g(on)g(the)g(datat)o(yp)q(es)h(of)e(the)i (corresp)q(onding)f(en)o(tries)h(in)f(the)g(sender)179 195 y(and)c(the)i(receiv)o(er)g(programs,)e(nor)h(is)f(it)h(required)h(that)f (they)g(b)q(e)g(the)h(same.)75 311 y Fh(1.5.5)55 b(Data)19 b(con)n(v)n(ersion)75 387 y Fn(W)m(e)13 b(use)i(the)g(follo)o(wing)c (terminology:)75 470 y Fk(t)o(yp)q(e)k(con)o(v)o(ersion)j Fn(c)o(hanges)d (the)f(datat)o(yp)q(e)g(of)f(a)h(v)n(alue,)f(e.g.)18 b(b)o(y)13 b(rounding)h(a)f Fl(REAL)g Fn(to)h(an)g Fl(INTEGER)p Fn(.)75 553 y Fk(represen)o(tati)o(on)e(con)o(v)o(ersion)18 b Fn(c)o(hanges)e(the)g (binary)f(represen)o(tation)i(of)e(a)g(v)n(alue,)f(e.g.)23 b(from)13 b(Hex)j(\015oating)179 603 y(p)q(oin)o(t)d(to)h(IEEE)g(\015oating)f (p)q(oin)o(t.)158 686 y(Represen)o(tation)20 b(con)o(v)o(ersion)f(is)g (needed)h(when)g(data)e(is)h(mo)o(v)o(ed)e(across)j(mac)o(hines)e(that)h(use) h(di\013eren)o(t)75 736 y(binary)9 b(enco)q(dings)h(for)f(the)h(same)f(datat) o(yp)q(e.)16 b(Represen)o(tation)11 b(con)o(v)o(ersion)e(preserv)o(es,)k(to)c (the)h(greatest)h(p)q(ossible)75 786 y(extend,)h(the)e Fk(v)m(alue)g Fn(represen)o(ted.)20 b(Ho)o(w)o(ev)o(er,)11 b(rounding)e(errors)j(and)e(o)o (v)o(er\015o)o(w)g(and)g(under\015o)o(w)h(exceptions)g(ma)o(y)75 836 y(o)q(ccur)17 b(during)e(\015oating)f(p)q(oin)o(t)h(con)o(v)o(ersions;)h (con)o(v)o(ersion)g(of)f(in)o(tegers)h(ma)o(y)e(also)g(lead)h(to)h (exceptions)g(when)g(a)75 885 y(v)n(alue)d(that)h(can)g(represen)o(ted)j(in)c (one)h(system)g(cannot)g(b)q(e)h(represen)o(ted)h(in)e(the)g(other)h(system.) i(MPI)d(do)q(es)h(not)75 935 y(sp)q(ecify)f(rules)h(for)e(represen)o(tation)j (con)o(v)o(ersion.)158 985 y(The)j(t)o(yp)q(e)f(matc)o(hing)e(rules)j(imply)d (that)i(MPI)g(comm)o(unication)d(nev)o(er)k(en)o(tails)e(t)o(yp)q(e)i(con)o (v)o(ersion.)31 b(On)75 1035 y(the)15 b(other)g(hand,)f(MPI)g(requires)i (that)e(a)g(represen)o(tation)i(con)o(v)o(ersion)e(b)q(e)h(p)q(erformed)f (when)h(a)f(t)o(yp)q(ed)g(v)n(alue)g(is)75 1085 y(transferred)g(across)g(en)o (vironmen)o(ts)d(that)i(use)g(di\013eren)o(t)g(represen)o(tations)i(for)d (the)h(datat)o(yp)q(e)g(of)e(this)i(v)n(alue.)k(An)75 1135 y(exception)f(o)q(ccurring)g(during)f(represen)o(tation)i(con)o(v)o(ersion)f (results)g(in)f(a)h(failure)e(of)h(the)h(comm)o(unicatio)o(n;)d(an)75 1184 y(error)i(o)q(ccurs)g(either)g(in)e(the)i(send)g(op)q(eration,)e(or)h (the)g(receiv)o(e)h(op)q(eration,)f(or)g(b)q(oth.)158 1234 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o(t)o(yp)q(ed)i (\(i.e.,)e(of)g(t)o(yp)q(e)i Fl(MPI)p 1180 1234 V 15 w(BYTE)p Fn(\),)e(then)h(the)h(binary)e(represen)o(tation)75 1284 y(of)h(the)g(b)o (yte)h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o(tical)f(to)h(the)h (binary)f(represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded)e(at)h(the)75 1334 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender)g(and)e (receiv)o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o(vironmen)o (ts;)75 1384 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g(required.)158 1433 y(Note)f(that)g(no)g(con)o(v)o(ersions)g(ev)o(er)h(o)q(ccur)g(when)g(an) e(MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f(system,)75 1483 y(where)j(all)e(pro)q(cesses)j(run)f(in)e(the)i(same)e(en)o(vironmen)o (t.)k(Also)c(note)i(the)f(di\013eren)o(t)h(b)q(eha)o(vior)f(of)f Fl(MPI)p 1695 1483 V 15 w(BYTE)g Fn(and)75 1533 y(of)18 b Fl(MPI)p 196 1533 V 15 w(CHARACTER)p Fn(.)f(A)h(bu\013er)i(descriptor)g(en)o(try)f (with)g(datat)o(yp)q(e)g(of)f Fl(MPI)p 1320 1533 V 15 w(CHARACTER)f Fn(can)i(only)f(matc)o(h)f(a)75 1583 y(F)m(ortran)e(v)n(ariable)f(of)h(t)o (yp)q(e)g Fl(CHARACTER)p Fn(;)e(and)i(represen)o(tation)i(con)o(v)o(ersion)e (ma)o(y)e(o)q(ccur)j(when)g(v)n(alues)f(of)f(t)o(yp)q(e)75 1633 y Fl(MPI)p 144 1633 V 15 w(CHARACTER)e Fn(are)i(transferred.,)h(e.g.,)e (from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q(ding.)158 1683 y(Consider)g(the)h(follo)o(wing)c(examples.)75 1774 y Fl(!)43 b(FIRST)21 b(EXAMPLE)75 1824 y(INTEGER)f(A\(100\))75 1873 y(...)75 1923 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 1973 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h (MPI_CHARACTER\))158 2064 y Fn(This)15 b(program)f(is)i(erroneous,)g(since)h (the)f(t)o(yp)q(e)g(of)e(the)j(bu\013er)f(descriptor)h(do)q(es)f(not)f(matc)o (h)g(the)h(t)o(yp)q(e)g(of)75 2114 y(the)g(v)n(ariable)f(in)g(the)i (bu\013er.)25 b(The)16 b(b)q(eha)o(vior)f(or)h(this)g(program)e(is)i (unde\014ned.)25 b(A)16 b(desirable)g(b)q(eha)o(vior)g(is)f(for)75 2164 y(that)f(error)h(to)e(b)q(e)i(detected)h(\(at)e(compile)e(time)h(or)h (run)g(time\))e(and)i(result)h(in)e(an)h(error)h(condition.)75 2255 y Fl(!)43 b(SECOND)21 b(EXAMPLE)75 2305 y(INTEGER)f(A\(100\))75 2355 y(...)75 2405 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 2455 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_BYTE\)) 158 2546 y Fn(This)11 b(program)f(is)h(correct.)19 b(The)12 b(b)o(yte)g(that)f(will)f(b)q(e)i(transmitted)f(in)g(a)g(message)g(created)i (with)e(this)h(bu\013er)75 2596 y(descriptor)h(is)f(the)g(\014rst)h(b)o(yte)f (in)g(the)g(storage)h(o)q(ccupied)f(b)o(y)g Fl(A\(1\))p Fn(.)17 b(The)12 b(v)n(alue)f(of)h(this)f(b)o(yte)i(is)e(implemen)o(tation)75 2646 y(dep)q(enden)o(t,)17 b(and)e(will)f(v)n(ary)h(according)g(to)g(the)h(n) o(um)o(b)q(er)f(of)f(b)o(ytes)i(used)h(to)e(store)h(F)m(ortran)f(v)n (ariables)g(of)f(t)o(yp)q(e)75 2695 y Fl(INTEGER)e Fn(and)i(the)h(b)o(yte)f (ordering)g(on)f(the)i(host)f(mac)o(hine.)p eop %%Page: 20 21 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(20)75 45 y Fl(!)43 b(THIRD)21 b(EXAMPLE)75 95 y(INTEGER)f(A\(100\))75 145 y(...)75 195 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH1,) d(MPI_PERSISTENT\))75 244 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH1,)j (A\(1\),)g(1,)g(MPI_INTEGER\))75 294 y(...)75 344 y(CALL)g(MPI_CREATE_BUFFER) o(\(BH2,)d(MPI_PERSISTENT\))75 394 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d (\(BH2,)j(A\(1\),)g(4,)g(MPI_BYTE\))158 485 y Fn(Consider)14 b(the)h(follo)o(wing)c(three)k(cases:)126 568 y(1.)20 b(A)10 b(message)g(is)g(sen)o(t)h(using)e(bu\013er)j(descriptor)f(handle)f Fl(BH1)f Fn(and)h(receiv)o(ed)i(using)e(bu\013er)h(descriptor)g(handle)179 618 y Fl(BH2)p Fn(.)19 b(The)d(program)d(is)h(erroneous,)i(since)f(the)h (sender)g(and)e(receiv)o(er)i(bu\013er)g(descriptors)g(don't)f(matc)o(h.)179 668 y(The)d(b)q(eha)o(vior)f(of)g(the)h(program)e(is)i(unde\014ned,)h(and)e (a)g(p)q(ossible)h(\(desirable)g(but)g(not)g(required\))g(b)q(eha)o(vior)179 718 y(is)i(for)f(an)h(error)h(condition)e(to)g(o)q(ccur.)126 801 y(2.)20 b(Both)d(sender)h(and)f(receiv)o(er)i(use)e(bu\013er)h (descriptor)g(handle)f Fl(BH1)p Fn(.)26 b(The)18 b(program)d(is)i(correct,)h (and)f(an)179 850 y(in)o(teger)12 b(v)n(alue)f(is)g(transmitted)g(from)f (sender)j(to)f(receiv)o(er.)19 b(The)12 b(v)n(alue)f(assigned)h(to)f Fl(A\(1\))g Fn(at)g(the)h(receiv)o(er)179 900 y(is)h(iden)o(tical)f(to)h(the) g(v)n(alue)g(of)f Fl(A\(1\))g Fn(at)h(the)h(sender,)g(but)f(the)h(binary)f (represen)o(tation)h(of)e(this)i(v)n(alue)e(ma)o(y)179 950 y(b)q(e)i(di\013eren)o(t,)h(if)e(sender)i(and)f(receiv)o(er)h(run)g(in)e (di\013eren)o(t)i(en)o(vironmen)o(ts.)126 1033 y(3.)20 b(Both)c(sender)h(and) e(receiv)o(er)i(use)g(bu\013er)f(descriptor)h(handle)f Fl(BH2)p Fn(.)22 b(The)16 b(program)e(is)h(correct.)25 b(Assume)179 1083 y(that)14 b(b)q(oth)h(sender)h(and)e(receiv)o(er)i(use)f(four)f(b)o (ytes)h(to)f(store)i(in)o(tegers.)k(Then)15 b(the)g(v)n(alue)f(stored)h(in)f Fl(A\(1\))179 1133 y Fn(at)i(the)h(receiv)o(er)h(has)e(the)h(same)f(binary)g (represen)o(tation)i(as)e(the)h(v)n(alue)f(of)f Fl(A\(1)h Fn(at)g(the)h (sender.)27 b(If)16 b(the)179 1183 y(same)c(b)o(yte)h(ordering)g(and)g(the)h (same)e(enco)q(ding)h(of)g(in)o(teger)g(v)n(alues)g(is)g(used)g(at)g(b)q(oth) g(ends)h(then)g Fl(A\(1\))e Fn(at)179 1232 y(the)j(receiv)o(er)h(is)f(set)g (to)g(the)g(v)n(alue)f(of)g Fl(A\(1\))g Fn(at)h(the)g(sender;)h(otherwise)g (the)f(same)f(32)g(bits)h(ma)o(y)e(enco)q(de)179 1282 y(di\013eren)o(t)i(v)n (alues.)158 1365 y(Data)h(represen)o(tation)h(con)o(v)o(ersion)g(also)e (applies)h(to)g(the)h(en)o(v)o(elop)q(e)g(of)e(a)h(message:)23 b(source,)17 b(destination)75 1415 y(and)d(tag)f(are)i(all)d(in)o(tegers)j (that)f(ma)o(y)e(need)j(to)e(b)q(e)i(con)o(v)o(erted.)158 1543 y Fj(Implemen)o(tation)e(note:)158 1589 y Fi(The)d(curren)o(t)g(de\014nition) j(do)q(es)d(not)h(require)g(messages)g(to)e(carry)i(t)o(yp)q(e)f (information.)18 b(A)9 b(message)i(can)f(b)q(e)h(comp)q(osed)75 1635 y(and)g(sen)o(t)f(using)i(only)f(the)f(information)i(in)f(the)g(sender)g (bu\013er)g(descriptor,)h(and)e(can)h(b)q(e)f(receiv)o(ed)i(and)e(stored)h (using)h(only)75 1680 y(the)i(information)i(in)e(the)g(receiv)o(er)h (bu\013er)g(descriptor.)21 b(If)13 b(messages)h(are)g(sen)o(t)g(b)q(et)o(w)o (een)g(di\013eren)o(t)i(mac)o(hines)f(then)f(one)75 1726 y(can)e(either)h (use)g(a)f(\\univ)o(ersal")j(data)d(enco)q(ding)i(for)e(messages,)h(use)f (kno)o(wledge)i(of)d(the)i(receiv)o(er)g(en)o(vironmen)o(t)h(in)e(order)75 1772 y(to)i(con)o(v)o(ert)h(data)f(at)g(the)h(sender,)f(or)h(use)f(kno)o (wledge)i(of)e(the)g(sender)h(en)o(vironmen)o(t)h(in)f(order)f(to)g(con)o(v)o (ert)h(data)g(at)f(the)75 1817 y(receiv)o(er.)k(In)c(either)g(case)f(the)h (lo)q(cal)g(bu\013er)g(descriptor)h(can)f(b)q(e)f(used)h(to)f(deriv)o(e)i (the)e(t)o(yp)q(es)h(of)f(the)g(v)n(alues)h(transferred.)75 1863 y(Ho)o(w)o(ev)o(er,)e(additional)17 b(t)o(yp)q(e)c(information)i (carried)f(b)o(y)g(messages)g(can)f(pro)o(vide)i(b)q(etter)e(error)g (detection.)158 2074 y Fj(Discussion:)158 2120 y Fi(There)i(is)g(no)g (agreemen)o(t)h(y)o(et)f(on)g(the)g(need)g(for)g(a)g(de\014nition)i(and,)f (if)f(needed,)h(on)f(the)g(righ)o(t)g(de\014nition)j(of)c(t)o(yp)q(e)75 2165 y(matc)o(hing)j(and)e(con)o(v)o(ersion)i(b)q(et)o(w)o(een)f(programs)g (written)g(in)g(di\013eren)o(t)g(languages.)26 b(One)15 b(prop)q(osal)i(is)f (to)f(allo)o(w)h(b)q(oth)57 b Fe(TO)11 b(DISCUSS)75 2211 y Fi(F)m(ortran)16 b(routines)i(to)e(sp)q(ecify)h(C)e(datat)o(yp)q(es)j(in)e (their)h(bu\013er)g(descriptors,)h(and)f(for)e(C)h(routines)h(to)f(sp)q (ecify)h(F)m(ortran)75 2257 y(data)o(yp)q(es.)j(The)13 b(draft)h(will)h(sp)q (ecify)f(rules)h(for)e(t)o(yp)q(e)h(matc)o(hing)h(b)q(et)o(w)o(een)f(F)m (ortran)g(v)n(ariables)h(and)g(C)e(datat)o(yp)q(es)h(in)h(the)75 2302 y(bu\013er)f(descriptors)h(and)f(b)q(et)o(w)o(een)f(C)g(v)n(ariables)j (and)d(F)m(ortran)h(datat)o(yp)q(es)g(in)g(the)g(bu\013er)g(descriptor.)19 b(\(E.g.,)12 b(a)h(F)m(ortran)75 2348 y(v)n(ariable)h(of)e(t)o(yp)q(e)g Fd(REAL)f Fi(ma)o(y)h(matc)o(h)g(an)g(en)o(try)h(of)f(t)o(yp)q(e)g Fd(MPI)p 976 2348 12 2 v 13 w(FLOAT)e Fi(in)j(a)f(bu\013er)h(descriptor,)g (but)g(not)f(an)g(en)o(try)h(of)e(t)o(yp)q(e)75 2394 y Fd(MPI)p 137 2394 V 13 w(INTEGER)p Fi(.)j(Data)j(represen)o(tation)i(con)o(v)o(ersion) g(o)q(ccurs)f(if)f(corresp)q(onding)j(C)d(and)h(F)m(ortran)f(t)o(yp)q(es)h (ha)o(v)o(e)f(di\013eren)o(t)75 2439 y(represen)o(tations)i(on)e(the)g(same)g (pro)q(cess.)29 b(There)16 b(still)j(is)e(the)g(requiremen)o(t)h(that)f (bu\013er)h(descriptor)g(en)o(tries)g(on)f(the)75 2485 y(send)12 b(side)h(matc)o(h)f(v)o(erbatim)g(bu\013er)h(descriptor)g(en)o(tries)g(on)e (the)h(receiv)o(e)h(side,)f(and)g(there)g(is)g(no)g(c)o(hange)h(in)f(in)o (terpro)q(cess)75 2531 y(comm)o(unication.)158 2576 y(This)i(prop)q(osal)h (giv)o(es)g(a)e(precise)h(meaning)h(to)e(in)o(terlanguage)k(comm)o (unication.)j(It)13 b(has)h(t)o(w)o(o)e(dra)o(wbac)o(ks:)19 b(It)13 b(ma)o(y)75 2622 y(require)k(additional)i(data)d(con)o(v)o(ersions)i (when)d(messages)i(are)e(assem)o(bled)j(or)d(disassem)o(bled,)k(and)d(it)g (imp)q(oses)h(a)f(\014xed)75 2668 y(corresp)q(ondence)i(b)q(et)o(w)o(een)e(C) g(and)h(F)m(ortran)f(t)o(yp)q(e)g(whic)o(h)h(ma)o(y)f(not)h(b)q(e)f(the)g (preferred)h(one)f(on)g(the)h(target)f(mac)o(hine.)p eop %%Page: 21 22 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(21)75 45 y Fi(\(Some)12 b(mac)o(hines)h(ma)o(y)f(w)o(an)o(t)g (F)m(ortran)g Fd(REAL)e Fi(to)h(matc)o(h)h(C)g Fd(FLOAT)p Fi(;)d(other)j(ma)o (y)g(w)o(an)o(t)f(it)h(to)g(matc)o(h)g(C)f Fd(double)p Fi(;)f(MPI)h(will)75 91 y(enforce)i(one)h(of)e(the)i(t)o(w)o(o.\))158 136 y(Another)i (\(non\)solution)i(is)f(to)e(ignore)i(the)e(issue)i(and)f(let)g(the)g(b)q (eha)o(vior)h(of)e(C)g(to)h(F)m(ortran)f(comm)o(unication)k(b)q(e)75 182 y(unde\014ned.)32 b(The)18 b(user)g(can)g(alw)o(a)o(ys)g(matc)o(h)g (language)h(at)f(sending)h(and)f(receiving)i(end)e(if)f(he)h(or)g(she)g(is)g (willing)i(to)75 228 y(struggle)14 b(with)g(the)f(issue)h(of)f(calling)i(MPI) e(C)g(functions)i(from)d(F)m(ortran)i(or)f(vice-v)o(ersa.)75 448 y Fm(1.6)70 b(Receiv)n(e)20 b(Criteria)75 539 y Fn(The)e(selection)g(of)f (a)g(message)g(b)o(y)g(a)g(receiv)o(e)i(op)q(eration)e(is)h(done)f(uniquely)g (according)h(to)f(the)h(v)n(alue)f(of)f(the)75 588 y(message)c(en)o(v)o(elop) q(e.)18 b(The)13 b(receiv)o(e)h(op)q(eration)e(sp)q(eci\014es)i(an)e Fk(en)o(v)o(elop)q(e)g(pattern)p Fn(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o (ed)75 638 y(b)o(y)j(that)g(receiv)o(e)h(op)q(eration)f(only)f(if)g(its)h(en) o(v)o(elop)q(e)g(matc)o(hes)f(that)h(pattern.)25 b(A)16 b(pattern)h(sp)q (eci\014es)h(v)n(alues)d(for)75 688 y(the)i Fl(source)p Fn(,)f Fl(tag)f Fn(and)i Fl(context)e Fn(\014elds)i(of)f(the)h(message)f(en)o(v)o (elop)q(e.)26 b(In)17 b(addition,)f(the)h(v)n(alue)f(for)g(the)h Fl(dest)75 738 y Fn(\014eld)e(is)g(set,)h(implicitl)o(y)m(,)c(to)i(b)q(e)i (equal)f(to)g(the)g(receiving)g(pro)q(cess)i(id.)k(The)16 b(receiv)o(er)g(ma) o(y)d(sp)q(ecify)j(a)e(wildcard)75 788 y Fl(MPI)p 144 788 14 2 v 15 w(ANY)p 225 788 V 15 w(SOURCE)d Fn(v)n(alue)g(for)g Fl(source)p Fn(,)g(and/or)g(a)h(wildcard)f Fl(MPI)p 1105 788 V 15 w(ANY)p 1186 788 V 15 w(TAG)g Fn(v)n(alue)g(for)h Fl(tag)p Fn(,)f(indicating)g(that)g(an)o(y)75 838 y(source)18 b(and/or)d(tag)h(are)h (acceptable.)26 b(It)16 b(cannot)h(sp)q(ecify)g(a)f(wildcard)f(v)n(alue)h (for)g Fl(context)f Fn(or)h Fl(dest)p Fn(.)24 b(Th)o(us,)75 887 y(a)16 b(message)g(can)g(b)q(e)h(receiv)o(ed)g(b)o(y)f(a)g(receiv)o(e)h (op)q(eration)f(only)g(if)f(it)h(is)f(addressed)j(to)e(the)h(receiving)f (task,)h(has)75 937 y(a)e(matc)o(hing)e(con)o(text,)i(has)g(matc)o(hing)e (source)j(unless)g(source=)p Fl(MPI)p 1175 937 V 17 w(ANY)p 1258 937 V 15 w(SOURCE)d Fn(in)i(the)g(pattern,)h(and)e(has)h(a)75 987 y(matc)o(hing)d(tag)i(unless)g(tag=)p Fl(MPI)p 609 987 V 15 w(ANY)p 690 987 V 15 w(TAG)g Fn(in)f(the)h(pattern.)158 1037 y(The)h(length)f(of)g(the)g(receiv)o(ed)i(message)e(m)o(ust)f(b)q(e)i (less)g(then)g(or)f(equal)g(the)h(length)f(of)g(the)h(receiv)o(e)g(bu\013er.) 75 1087 y(I.e.,)d(all)f(incoming)f(data)i(m)o(ust)f(\014t,)i(without)f (truncation,)g(in)o(to)g(the)h(receiv)o(e)g(bu\013er.)19 b(It)12 b(is)h(erroneous)g(to)g(receiv)o(e)75 1136 y(a)h(message)f(whic)o(h)h(length) g(exceed)i(the)e(receiv)o(e)h(bu\013er,)g(and)e(the)i(outcome)e(of)g(program) g(where)i(this)f(o)q(ccurs)h(is)75 1186 y(undetermined.)75 1323 y Fm(1.7)70 b(Comm)n(unication)20 b(Mo)r(de)75 1414 y Fn(A)14 b(send)h(op)q(eration)f(can)g(o)q(ccur)h(in)e(one)h(of)f(three)j(mo)q (des:)75 1497 y Fk(ST)l(AND)o(ARD)k Fn(The)14 b(send)h(ma)o(y)d(b)q(e)i (started)h(whether)h(or)d(not)h(a)g(matc)o(hing)e(receiv)o(e)j(has)f(b)q(een) h(p)q(osted.)75 1580 y Fk(READ)o(Y)20 b Fn(The)15 b(send)g(ma)o(y)c(b)q(e)k (started)g(only)e(if)g(a)h(matc)o(hing)e(receiv)o(e)j(has)f(b)q(een)h(p)q (osted.)75 1663 y Fk(SYNCHR)o(ONOUS)20 b Fn(The)c(send)g(op)q(eration)f(will) f(not)h(complete)g(successfully)h(un)o(til)e(it)h(is)g(guaran)o(teed)h(that) 179 1713 y(the)e(message)g(sen)o(t)h(will)d(b)q(e)i(receiv)o(ed.)158 1796 y(The)19 b(start)g(and)f(completion)f(of)h(a)g Fk(standard)h(send)f Fn(do)g(not)g(dep)q(end)i(on)e(the)h(status)h(of)e(a)g(matc)o(hing)75 1846 y(receiv)o(e.)158 1896 y(A)f Fk(ready)i(send)c Fn(should)i(b)q(e)h (started)g(only)e(if)g(a)g(matc)o(hing)f(receiv)o(e)j(is)f(already)g(p)q (osted;)i(otherwise)e(the)75 1946 y(op)q(eration)j(is)f(erroneous)i(and)f (its)g(outcome)f(is)g(unde\014ned.)37 b(In)20 b(some)f(systems,)i(this)e (allo)o(ws)g(the)h(remo)o(v)n(al)75 1996 y(of)f(a)f(hand-shak)o(e)i(op)q (eration)f(that)g(is)g(otherwise)h(required,)h(and)e(results)i(in)d(impro)o (v)o(ed)g(p)q(erformance.)33 b(Its)75 2045 y(completion)12 b(do)q(es)j(not)f(dep)q(end)h(on)e(the)i(status)g(of)e(a)g(matc)o(hing)f (receiv)o(e.)158 2095 y(A)k Fk(sync)o(hronous)f(send)g Fn(can)g(b)q(e)i (started)g(irresp)q(ectiv)o(e)g(of)e(the)i(status)f(of)f(a)h(matc)o(hing)e (receiv)o(e.)25 b(It)15 b(will)75 2145 y(complete)i(only)g(if)g(a)h(matc)o (hing)e(receiv)o(e)j(is)f(p)q(osted,)h(and)f(the)h(receiv)o(e)g(op)q(eration) f(is)f(committed)f(to)i(receiv)o(e)75 2195 y(the)e(message)f(sen)o(t)h(b)o(y) f(the)h(secure)h(send.)24 b(\(I.e.,)14 b(the)i(receiv)o(e)h(has)e(b)q(een)i (p)q(osted,)f(the)g(incoming)d(message)i(has)75 2245 y(b)q(een)i(matc)o(hed)e (to)g(this)h(p)q(osted)h(receiv)o(e,)g(and)e(the)h(receiv)o(e)h(op)q(eration) f(has)g(already)f(started)i(and)e(cannot)h(b)q(e)75 2294 y(cancelled)f(an)o (ymore.\))158 2344 y(There)g(is)f(only)f(one)h(receiv)o(e)h(mo)q(de,)e(whic)o (h)g(can)h(matc)o(h)f(an)o(y)h(of)f(the)h(send)h(mo)q(des.)158 2394 y(Comm)o(uni)o(cation)e(imp)q(oses)j(an)g(order)h(on)g(the)g(ev)o(en)o (ts)g(o)q(ccurring)g(at)f(the)h(comm)o(unicating)c(no)q(des.)27 b(It)16 b(is)75 2444 y(alw)o(a)o(ys)d(the)i(case)h(that)e(the)h(completion)d (of)i(a)g(receiv)o(e)i(o)q(ccurs)g(after)e(the)h(start)g(of)f(the)h(matc)o (hing)d(send.)20 b(If)14 b(the)75 2494 y(sync)o(hronous)i(send)g(mo)q(de)f (is)g(used,)h(then)g(it)f(is)g(also)g(the)h(case)g(that)f(the)h(completion)e (of)h(a)g(send)h(o)q(ccurs)h(after)75 2543 y(the)f(start)g(of)g(the)g(matc)o (hing)e(receiv)o(e.)24 b(Of)16 b(course,)h(on)e(eac)o(h)h(pro)q(cess,)i(a)d (comm)o(unicatio)o(n)e(completes)i(after)h(it)75 2593 y(is)i(started.)30 b(No)18 b(other)g(order)g(is)g(imp)q(osed)e(b)o(y)i(MPI.)f(I.e.,)h(if)f(the)h (standard)g(send)h(mo)q(de)d(is)i(used,)h(the)f(send)75 2643 y(op)q(eration)c(ma)o(y)e(complete)h(b)q(efore)i(the)f(matc)o(hing)e(receiv)o (e)j(w)o(as)f(started.)p eop %%Page: 22 23 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(22)158 45 y Fj(Implemen)o(tation)13 b(note:)35 b Fi(A)13 b(ready)g(send)h(can)g(b)q(e)f(implemen)o(ted)i(as)f(a)f(regular)h (send;)g(in)g(suc)o(h)f(case)h(there)f(will)75 91 y(b)q(e)j(no)h(p)q (erformance)g(adv)n(an)o(tage)g(\(or)f(disadv)n(an)o(tage\))j(for)d(the)g (use)h(of)f(ready)g(send;)i(and)f(some)g(erroneous)g(programs)75 136 y(ma)o(y)c(go)g(undetected)i(\(and)e(ma)o(y)g(still)i(pro)q(duce)g (correct)e(results\).)158 182 y(A)g(p)q(ossible)i(comm)o(unication)h(proto)q (col)e(for)f(the)g(v)n(arious)i(comm)o(unication)g(mo)q(des)f(is)g(outlined)h (b)q(elo)o(w:)158 228 y Fd(ready)j(send)p Fi(:)d(The)e(message)h(is)g(sen)o (t)f(asap;)h(an)f(ac)o(kno)o(wledge)i(is)f(returned)g(b)o(y)g(receiv)o(er.)k (The)13 b(proto)q(col)i(is)f(\(send,)75 273 y(ac)o(k\).)158 319 y Fd(synchronou)o(s)i(send:)25 b Fi(The)18 b(sender)g(sends)h(a)f (request-to-send)h(message.)31 b(The)18 b(receiv)o(er)h(stores)f(this)g (request.)75 365 y(When)e(a)e(matc)o(hing)j(receiv)o(e)e(is)h(p)q(osted,)f (the)g(receiv)o(er)h(sends)f(bac)o(k)h(a)f(p)q(ermission-to-send)i(message,)f (and)f(the)g(sender)75 410 y(no)o(w)e(sends)h(the)f(message.)18 b(The)12 b(proto)q(col)j(is)f(\(req,)e(ac)o(k-req,)h(send,)h(ac)o(k\).)158 456 y Fd(standard)j(send:)d Fi(First)e(proto)q(col)g(ma)o(y)g(b)q(e)f(used)h (for)f(short)g(messages,)h(and)g(second)g(proto)q(col)h(for)e(long)h (messages.)158 506 y(Of)g(course,)i(there)f(are)g(man)o(y)h(other)f(p)q (ossible)i(c)o(hoices.)75 726 y Fm(1.8)70 b(Comm)n(unication)20 b(Ob)t(jects)75 817 y Fn(An)15 b(opaque)g(comm)o(unication)d(ob)r(ject)k (iden)o(ti\014es)g(v)n(arious)e(prop)q(erties)j(of)e(a)g(comm)o(uni)o(cation) d(op)q(eration,)j(suc)o(h)75 867 y(as)h(the)g(bu\013er)h(descriptor)h(that)d (is)h(asso)q(ciated)h(with)e(it,)h(its)g(con)o(text,)g(the)h(tag)e(and)h (destination)g(parameters)75 916 y(to)f(b)q(e)h(used)g(for)f(a)g(send,)h(or)f (the)h(tag)f(and)g(source)h(parameters)g(to)f(b)q(e)g(used)i(for)d(a)h (receiv)o(e.)24 b(In)15 b(addition,)f(this)75 966 y(ob)r(ject)g(stores)h (information)10 b(ab)q(out)j(the)h(status)g(of)e(the)i(last)f(comm)o (unication)d(op)q(eration)j(that)g(w)o(as)g(p)q(erformed)75 1016 y(with)h(this)f(ob)r(ject.)19 b(This)14 b(ob)r(ject)h(is)f(accessed)i (using)d(a)h(comm)o(unicatio)o(n)d(handle.)158 1066 y(One)k(can)f(consider)h (comm)o(uni)o(cation)c(op)q(erations)j(to)g(consist)g(of)g(the)g(follo)o (wing)d(sub)q(op)q(erations:)75 1157 y Fk(INIT\(op)q(eration,)j(params,)i (handle\))i Fn(Pro)q(cess)c(pro)o(vides)f(all)f(relev)n(an)o(t)g(parameters)h (for)f(its)h(participation)179 1207 y(in)i(the)i(comm)o(unication)c(op)q (eration)j(\(t)o(yp)q(e)g(of)g(op)q(eration,)g(data)g(bu\013er,)h(tag,)e (participan)o(ts,)i(etc.\).)25 b(An)179 1257 y(ob)r(ject)14 b(is)g(created)i(that)e(iden)o(ti\014es)g(the)g(op)q(eration.)75 1340 y Fk(ST)l(AR)l(T\(handle\))k Fn(The)c(comm)o(unication)c(op)q(eration)k (is)g(started)75 1423 y Fk(COMPLETE\(handle\))19 b Fn(The)14 b(comm)o(unication)c(op)q(eration)k(is)g(completed.)75 1506 y Fk(FREE\(handle\))k Fn(The)d(comm)o(unicati)o(on)c(ob)r(ject,)j(and)g(asso) q(ciated)h(resources)h(are)e(freed.)75 1597 y(Correct)h(in)o(v)o(o)q(cation)e (of)g(these)i(sub)q(op)q(erations)g(is)f(a)f(sequence)k(of)c(the)h(form)587 1688 y Fk(INIT)j Fn(\()p Fk(ST)l(AR)l(T)f(COMPLETE)p Fn(\))1187 1671 y Fa(\003)1223 1688 y Fk(FREE)p Fb(:)75 1780 y Fn(I.e.,)10 b(an)h(ob)r(ject)g(needs)h(b)q(e)f(created)i(b)q(efore)e(comm)o(unication)c (o)q(ccurs;)13 b(it)d(can)h(b)q(e)g(reused)i(only)c(after)i(the)h(previous)75 1830 y(use)k(has)g(completed;)f(and)g(it)g(needs)i(to)e(b)q(e)h(freed)h(ev)o (en)o(tually)d(\(of)h(course,)i(one)f(can)f(assume)g(that)h(all)e(ob)r(jects) 75 1879 y(are)g(freed)h(at)f(program)e(termination,)g(b)o(y)h(default\).)158 1929 y(The)21 b(ab)q(o)o(v)o(e)g(scenario)g(p)q(ertains)g(to)g Fg(p)n(ersistent)f Fn(ob)r(jects.)39 b(One)22 b(can)f(also)f(create)i Fg(ephemer)n(al)e Fn(ob)r(jects.)75 1979 y(Suc)o(h)c(an)g(ob)r(ject)h(p)q (ersists)h(only)d(un)o(til)g(the)i(comm)o(uni)o(cation)c(op)q(eration)j(is)g (completed,)f(at)h(whic)o(h)g(p)q(oin)o(t)f(it)h(is)75 2029 y(destro)o(y)o(ed.)i(Th)o(us)12 b(the)g(correct)i(in)o(v)o(o)q(cation)c(of)h (sub)q(op)q(erations)i(with)e(an)g(ephemeral)g(ob)r(ject)i(is)e Fk(INIT)j(ST)l(AR)l(T)75 2079 y(COMPLETE)p Fn(.)158 2128 y(A)19 b(user)i(ma)o(y)c(directly)j(in)o(v)o(ok)o(e)f(these)h(sub)q(op)q(erations.) 36 b(This)19 b(w)o(ould)g(allo)o(w)e(the)j(amortization)e(of)g(the)75 2178 y(o)o(v)o(erhead)d(of)g(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o (er)j(man)o(y)e(successiv)o(e)k(uses)f(of)f(the)g(same)g(handle,)f(and)h (allo)o(ws)75 2228 y(the)d(o)o(v)o(erlap)f(of)g(comm)o(unication)e(and)i (computation.)16 b(Simpler)10 b(comm)o(unicatio)o(n)f(op)q(erations)j(com)o (bine)e(sev)o(eral)75 2278 y(of)17 b(these)i(sub)q(op)q(erations)g(in)o(to)e (one)h(op)q(eration,)g(th)o(us)g(simplifying)d(the)j(use)h(of)e(comm)o (unicatio)o(n)e(primitiv)o(es.)75 2328 y(Th)o(us,)h(one)h(only)e(needs)j(to)e (sp)q(ecify)g(precisely)h(the)g(seman)o(tics)f(of)f(these)j(sub)q(op)q (erations)f(in)f(order)g(to)g(sp)q(ecify)75 2378 y(the)e(seman)o(tics)g(of)f (MPI)h(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unicatio)o(n)d(op)q(erations.) 158 2427 y(W)m(e)i(sa)o(y)g(that)g(a)g(comm)o(unicatio)o(n)d(op)q(eration)j (\(send)i(or)e(receiv)o(e\))h(is)f Fk(p)q(osted)f Fn(once)i(a)f Fk(start)e Fn(sub)q(op)q(eration)75 2477 y(has)18 b(b)q(een)h(in)o(v)o(ok)o (ed;)g(the)g(op)q(eration)f(is)g Fk(completed)e Fn(once)i(the)h Fk(complete)d Fn(sub)q(op)q(eration)j(has)f(completed.)75 2527 y(A)e(send)h(and)f(a)g(receiv)o(e)i(op)q(eration)e Fk(matc)o(h)f Fn(if)g(the)i(receiv)o(e)h(pattern)f(sp)q(eci\014ed)g(b)o(y)f(the)h(receiv)o (e)h(matc)o(hes)d(the)75 2577 y(message)f(en)o(v)o(elop)q(e)g(created)h(b)o (y)f(the)g(send.)p eop %%Page: 23 24 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(23)75 45 y Fh(1.8.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)75 122 y Fn(An)g(ob)r(ject)g(for)g(a)f(send)i(op)q (eration)f(is)f(created)j(b)o(y)d(a)h(call)e(to)i Fk(MPI)p 1199 122 15 2 v 17 w(INIT)p 1322 122 V 18 w(xSEND)p Fn(.)g(The)g(pre\014x)g (to)g Fl(SEND)75 172 y Fn(indicates)13 b(the)g(comm)o(unication)d(mo)q(de:)16 b(none)d(for)g Fl(standard)p Fn(,)e Fl(R)h Fn(for)h Fl(ready)p Fn(,)e(and)i Fl(S)f Fn(for)h Fl(synchronous)p Fn(.)i(A)e(call)75 221 y(to)f Fk(MPI)p 223 221 V 18 w(INIT)p 347 221 V 18 w(RECV)h Fn(is)f(similarly)d(used)14 b(for)e(creating)h(an)f(ob)r(ject)h(for)f(a)g (receiv)o(e)i(op)q(eration.)k(The)13 b(creation)75 271 y(of)g(a)h(comm)o (unicatio)o(n)d(ob)r(ject)k(is)f(a)f(lo)q(cal)g(op)q(eration)h(that)g(need)h (not)e(in)o(v)o(olv)o(e)g(comm)o(unicatio)o(n)e(with)j(a)f(remote)75 321 y(pro)q(cess.)158 406 y Fk(MPI)p 257 406 V 17 w(INIT)p 380 406 V 18 w(SEND)j(\(handle,)e(bu\013er)p 856 406 V 15 w(handle,)g(dest,)h (tag,)h(con)o(text,)f(p)q(ersistence\))158 491 y Fn(Creates)g(a)f(comm)o (unicatio)o(n)d(ob)r(ject)k(for)f(standard)g(mo)q(de)f(send.)19 b(P)o(arameters)14 b(are)75 571 y Fk(OUT)i(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g (ob)r(ject)h(b)q(efore)g(the)179 621 y(call.)75 703 y Fk(IN)e(bu\013er)p 273 703 V 16 w(handle)j Fn(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 784 y Fk(IN)h(dest)k Fn(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o (teger\))75 865 y Fk(IN)i(tag)21 b Fn(user)15 b(tag)e(for)h(messages)g(sen)o (t)g(with)g(this)g(handle)g(\(in)o(teger\))75 947 y Fk(IN)i(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75 1028 y Fk(IN)i(p)q(ersistence)i Fn(handle)10 b(p)q(ersistence)i(\(state)e(t)o (yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fl(MPI)p 1318 1028 14 2 v 15 w(PERSISTENT)8 b Fn(and)h Fl(MPI)p 1703 1028 V 15 w(EPHEMERAL)p Fn(\))158 1144 y Fk(MPI)p 257 1144 15 2 v 17 w(INIT)p 380 1144 V 18 w(RSEND)16 b(\(handle,)d(bu\013er)p 891 1144 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text,)f(p)q(ersistence\))158 1229 y Fn(Creates)g(a)f(comm)o(unicatio)o(n)d(ob)r(ject)k(for)f(standard)g (mo)q(de)f(send.)19 b(P)o(arameters)14 b(are)75 1309 y Fk(OUT)i(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q (ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the)179 1359 y(call.)75 1440 y Fk(IN)e(bu\013er)p 273 1440 V 16 w(handle)j Fn(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 1521 y Fk(IN)h(dest)k Fn(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o (teger\))75 1603 y Fk(IN)i(tag)21 b Fn(user)15 b(tag)e(for)h(messages)g(sen)o (t)g(with)g(this)g(handle)g(\(in)o(teger\))75 1684 y Fk(IN)i(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75 1766 y Fk(IN)i(p)q(ersistence)i Fn(handle)10 b(p)q(ersistence)i(\(state)e(t)o (yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fl(MPI)p 1318 1766 14 2 v 15 w(PERSISTENT)8 b Fn(and)h Fl(MPI)p 1703 1766 V 15 w(EPHEMERAL)p Fn(\))158 1881 y Fk(MPI)p 257 1881 15 2 v 17 w(INIT)p 380 1881 V 18 w(SSEND)16 b(\(handle,)d(bu\013er)p 882 1881 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text,)e(p)q(ersistence\))158 1966 y Fn(Creates)h(a)f(comm)o(unicatio)o(n)d(ob)r(ject)k(for)f(sync)o (hronous)g(mo)q(de)f(send.)19 b(P)o(arameters)14 b(are)75 2046 y Fk(OUT)i(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not) g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the)179 2096 y(call.)75 2177 y Fk(IN)e(bu\013er)p 273 2177 V 16 w(handle)j Fn(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 2259 y Fk(IN)h(dest)k Fn(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o (teger\))75 2340 y Fk(IN)i(tag)21 b Fn(user)15 b(tag)e(for)h(messages)g(sen)o (t)g(with)g(this)g(handle)g(\(in)o(teger\))75 2422 y Fk(IN)i(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75 2503 y Fk(IN)i(p)q(ersistence)i Fn(handle)10 b(p)q(ersistence)i(\(state)e(t)o (yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fl(MPI)p 1318 2503 14 2 v 15 w(PERSISTENT)8 b Fn(and)h Fl(MPI)p 1703 2503 V 15 w(EPHEMERAL)p Fn(\))158 2618 y Fk(MPI)p 257 2618 15 2 v 17 w(INIT)p 380 2618 V 18 w(RECV)17 b(\(handle,)d(bu\013er)p 862 2618 V 15 w(handle,)g(source,)h(tag,)h(con)o(text,)f(p)q(ersistence\))158 2704 y Fn(Creates)g(a)f(receiv)o(e)h(handle.)j(P)o(arameters)c(are)p eop %%Page: 24 25 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(24)75 45 y Fk(OUT)16 b(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g (ob)r(ject)h(b)q(efore)g(the)179 95 y(call.)75 178 y Fk(IN)e(bu\013er)p 273 178 15 2 v 16 w(handle)j Fn(handle)13 b(to)h(receiv)o(e)h(bu\013er)g (descriptor.)75 261 y Fk(IN)h(source)k Fn(rank)14 b(in)f(con)o(text)i(of)e (source,)i(or)f Fl(MPI)p 897 261 14 2 v 15 w(ANY)p 978 261 V 15 w(SOURCE)f Fn(\(in)o(teger\).)75 344 y Fk(IN)j(tag)21 b Fn(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h (or)g Fl(MPI)p 1232 344 V 15 w(ANY)p 1313 344 V 15 w(TAG)f Fn(\(in)o(teger\).)75 427 y Fk(IN)j(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 510 y Fk(IN)i(p)q(ersistence)i Fn(handle)10 b(p)q(ersistence)i(\(state)e(t)o(yp)q (e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fl(MPI)p 1318 510 V 15 w(PERSISTENT)8 b Fn(and)h Fl(MPI)p 1703 510 V 15 w(EPHEMERAL)p Fn(\))158 593 y(See)15 b(Section)f(1.4.2)e(for)i(a)f(discussion)i(of)e (source,)i(tag)e(and)h(con)o(text.)158 721 y Fj(Discussion:)158 767 y Fi(Ephemeral)f(\\p)q(ersisten)o(t)h(handles")f(are)f(not)g(terribly)h (useful)g(as)f(suc)o(h.)17 b(But)12 b(their)g(existence)h(allo)o(w)g(us)f(to) f(describ)q(e)75 813 y(all)j(the)f(higher)i(lev)o(el)f(send/receiv)o(e)h(op)q (erations)g(in)f(terms)f(of)g(basic)h(op)q(erations)h(on)e(handles.)75 1012 y Fh(1.8.2)55 b(Comm)n(unication)21 b(Start)75 1124 y Fk(MPI)p 174 1124 15 2 v 17 w(ST)l(AR)l(T\(handle\))75 1242 y(IN)16 b(handle)j Fn(comm)o(unication)10 b(handle)158 1325 y(The)16 b Fl(MPI)p 314 1325 14 2 v 15 w(START)f Fn(function)h(starts)g(the)h (execution)f(of)g(a)f(comm)o(unication)e(op)q(eration)i(\(send)i(or)f(receiv) o(e\).)75 1375 y(A)e(send)h(can)g(b)q(e)f(started)i(pro)o(vided)e(that)g(the) h(previous)f(send)h(using)f(the)h(same)e(handle)h(has)h(completed,)e(or)h(as) 75 1425 y(so)q(on)j(as)h(the)g(handle)f(has)g(b)q(een)i(created,)g(if)d (there)j(w)o(as)e(no)g(preceding)h(send.)30 b(If)17 b(the)g(send)i(mo)q(de)d (is)h Fl(ready)75 1475 y Fn(then,)e(in)e(addition,)g(a)h(matc)o(hing)f (receiv)o(e)i(should)f(ha)o(v)o(e)g(b)q(een)i(already)e(p)q(osted.)20 b(The)15 b(send)g(bu\013er)g(should)f(not)75 1524 y(b)q(e)h(up)q(dated)f (after)g(the)h(send)f(is)g(started,)h(un)o(til)e(the)h(op)q(eration)g (completes.)158 1574 y(A)f(receiv)o(e)h(can)f(b)q(e)g(started)h(as)f(so)q(on) g(as)g(the)g(preceding)h(receiv)o(e)g(using)e(the)i(same)d(handle)i(has)g (completed,)75 1624 y(or)18 b(as)g(so)q(on)g(as)g(the)h(handle)e(has)i(b)q (een)g(created)g(is)f(there)h(w)o(as)f(no)g(preceding)h(receiv)o(e.)31 b(The)19 b(receiv)o(e)g(bu\013er)75 1674 y(should)14 b(not)f(b)q(e)i (accessed)h(after)e(the)h(receiv)o(e)g(is)f(started,)g(un)o(til)f(the)i(op)q (eration)f(completes.)158 1724 y(A)e(program)f(that)h(do)q(es)h(not)g (satisfy)f(these)h(conditions)f(is)h(erroneous)g(and)f(its)h(outcome)e(is)h (undetermined.)158 1773 y(The)i(call)f(returns)j(as)e(so)q(on)f(as)h(p)q (ossible.)k(A)c(call)f(that)h(starts)h(a)f(send)g(can)g(return)h(b)q(efore)g (the)f(data)g(to)f(b)q(e)75 1823 y(sen)o(t)i(has)f(b)q(een)i(cleared)f(out)f (of)f(the)i(send)g(bu\013er)g(and)f(b)q(efore)h(an)o(y)f(matc)o(hing)e (receiv)o(e)k(has)e(b)q(een)i(p)q(osted)f(\(this)75 1873 y(is)h(true)h(for)e (all)g(send)i(mo)q(des,)e(including)g Fl(synchronous)p Fn(\).)22 b(A)16 b(call)f(that)h(starts)h(a)f(receiv)o(e)h(can)f(return)h(b)q(efore)75 1923 y(andy)d(data)f(has)h(b)q(een)h(receiv)o(ed)g(in)f(the)g(receiv)o(e)i (bu\013er.)75 2039 y Fh(1.8.3)55 b(Comm)n(unication)21 b(Completion)75 2116 y Fn(The)c(completion)e(of)h(a)h(send)g(op)q(eration)g(indicates)g(that) g(the)g(sender)i(is)d(no)o(w)h(free)g(to)g(up)q(date)g(the)g(lo)q(cations)75 2166 y(in)d(the)h(send)h(bu\013er,)f(or)g(an)o(y)f(other)h(lo)q(cation)f (that)h(can)f(b)q(e)i(referenced)h(b)o(y)d(the)h(send)h(op)q(eration.)k(It)15 b(do)q(es)g(not)75 2215 y(indicate)c(that)f(the)h(message)g(has)g(b)q(een)g (receiv)o(ed;)i(rather,)f(it)e(ma)o(y)e(ha)o(v)o(e)j(b)q(een)h(bu\013ered)g (b)o(y)e(the)h(comm)o(unication)75 2265 y(subsystem.)21 b(Ho)o(w)o(ev)o(er,) 15 b(if)g(a)f Fl(synchronous)f Fn(mo)q(de)h(send)h(w)o(as)g(used,)h(the)f (completion)e(of)i(the)g(send)h(op)q(eration)75 2315 y(indicates)d(that)h(a)f (matc)o(hing)e(receiv)o(e)j(w)o(as)g(initiated,)e(and)h(that)g(the)h(message) e(will)g(ev)o(en)o(tually)h(b)q(e)h(receiv)o(ed)g(b)o(y)75 2365 y(this)g(matc)o(hing)e(receiv)o(e.)158 2415 y(The)20 b(completion)e(of)i (a)f(receiv)o(e)i(op)q(eration)f(indicates)g(that)g(the)h(receiv)o(er)g(is)f (no)o(w)f(free)i(to)f(access)h(the)75 2464 y(lo)q(cations)15 b(in)g(the)h(receiv)o(e)h(bu\013er,)f(whic)o(h)f(con)o(tain)g(the)h(receiv)o (ed)h(message,)e(or)h(an)o(y)f(other)h(lo)q(cation)e(that)i(can)75 2514 y(b)q(e)g(referenced)i(b)o(y)d(the)h(receiv)o(e)h(op)q(eration.)23 b(It)15 b(do)q(es)h(not)g(indicate)f(that)h(the)g(matc)o(hing)d(send)j(op)q (eration)g(has)75 2564 y(completed)d(\(but)i(indicates,)e(of)h(course,)g (that)g(the)h(send)g(w)o(as)e(initiated\).)158 2649 y Fk(MPI)p 257 2649 15 2 v 17 w(W)-5 b(AIT)17 b(\()e(handle,)f(status\))p eop %%Page: 25 26 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(25)75 45 y Fk(IN)16 b(handle)j Fn(comm)o(unication)10 b(handle)75 127 y Fk(OUT)16 b(status)j Fn(handle)i(that)f(is)h(asso)q(ciated) g(with)g(return)g(status)h(ob)r(ject.)39 b(The)21 b(return)h(status)g(ob)r (ject)f(is)179 177 y(up)q(dated)14 b(b)o(y)g(call.)158 257 y(A)g(call)f(to)h Fk(MPI)p 429 257 15 2 v 17 w(W)-5 b(AIT)14 b Fn(returns)i(when)e(the)h(send)f(op)q(eration)g(iden)o(ti\014ed)g(b)o(y)g Fl(handle)f Fn(is)g(complete.)158 307 y(The)20 b(call)e(returns)j(a)e(handle) h(to)f(an)g(opaque)g(ob)r(ject)h(that)g(con)o(tains)f(information)d(on)k(the) g(completed)75 356 y(op)q(eration)14 b({)f(the)i Fk(return)f(status)e Fn(ob)r(ject.)158 442 y Fk(MPI)p 257 442 V 17 w(TEST)k(\(handle,)e(\015ag,)i (status\))75 557 y(IN)g(handle)j Fn(comm)o(unication)10 b(handle)75 639 y Fk(OUT)16 b(\015ag)k Fn(logical)75 721 y Fk(OUT)c(status)j Fn(handle)14 b(that)g(is)f(asso)q(ciated)i(with)f(return)h(status)f(ob)r (ject.)158 801 y(A)k(call)e(to)i Fk(MPI)p 440 801 V 17 w(TEST)g Fn(returns)h Fl(flag=true)c Fn(if)i(the)h(op)q(eration)g(iden)o(ti\014ed)f(b) o(y)g Fl(handle)g Fn(is)g(complete.)75 851 y(In)f(suc)o(h)g(case,)g(the)h (return)f(handle)g(p)q(oin)o(ts)f(to)h(an)f(opaque)h(ob)r(ject)g(that)g(con)o (tains)f(information)e(on)i(the)h(com-)75 900 y(pleted)f(op)q(eration.)k(It)c (returns)h Fl(flag=false)p Fn(,)c(otherwise.)20 b(In)15 b(suc)o(h)g(case,)g (the)g(v)n(alue)f(of)f(the)i(return)h(handle)e(is)75 950 y(unde\014ned.)158 1000 y(A)f(call)g(to)g Fl(MPI)p 397 1000 14 2 v 15 w(WAIT)f Fn(blo)q(c)o(ks)h(only)f(the)i(executing)g(thread.)k(If)13 b(the)h(executing)g(pro)q(cess)h(is)e(m)o(ultithreaded,)75 1050 y(then)h(other)g(threads)g(within)f(the)h(pro)q(cess)h(can)f(b)q(e)f(sc) o(heduled)i(for)e(execution.)19 b(The)13 b(use)i(of)d(a)h(blo)q(c)o(king)g (receiv)o(e)75 1100 y(op)q(eration)e(\()p Fl(MPI)p 343 1100 V 15 w(WAIT)p Fn(\))g(allo)o(ws)f(the)i(op)q(erating)f(system)h(to)f(desc)o (hedule)i(the)f(blo)q(c)o(k)o(ed)f(thread)i(and)e(sc)o(hedule)h(an-)75 1150 y(other)f(thread)g(for)e(execution,)j(if)d(suc)o(h)i(is)f(a)o(v)n (ailable.)k(The)d(use)g(of)e(a)h(non)o(blo)q(c)o(king)f(receiv)o(e)i(op)q (eration)f(\()p Fl(MPI)p 1759 1150 V 16 w(TEST)p Fn(\))75 1199 y(allo)o(ws)j(the)h(user)h(to)f(sc)o(hedule)h(alternativ)o(e)f(activities)f (within)g(a)h(single)g(thread)g(of)f(execution.)158 1249 y(The)j(in)o(tended) h(implem)o(en)o(tation)c(of)i Fl(MPI)p 833 1249 V 15 w(TEST)g Fn(is)g(for)h(that)g(op)q(eration)f(to)h(return)h(as)f(so)q(on)f(as)h(p)q (ossible.)75 1299 y(Ho)o(w)o(ev)o(er,)e(if)f(rep)q(eatedly)i(called)f(for)f (an)h(op)q(eration)g(that)f(is)h(enabled,)g(it)f(m)o(ust)g(ev)o(en)o(tually)h (succeed.)158 1349 y(The)g(return)h(status)g(ob)r(ject)g(for)e(a)h(send)h(op) q(eration)e(carries)i(no)f(information.)h(The)f(return)h(status)g(ob)r(ject) 75 1399 y(for)h(a)g(receiv)o(e)i(op)q(eration)f(carries)g(information)d(on)i (the)i(source,)g(tag)e(and)g(length)h(of)f(the)h(receiv)o(ed)h(message.)75 1448 y(These)13 b(\014elds)g(are)f(required)h(b)q(ecause)h(the)e(receiv)o(e)i (op)q(eration)e(ma)o(y)e(ha)o(v)o(e)h(sp)q(eci\014ed)j Fl(DONTCARE)c Fn(in)i(either)h(source)75 1498 y(or)h(tag)f(\014eld,)h(and)g(the)g(message)g (ma)o(y)e(ha)o(v)o(e)h(b)q(een)i(shorter)g(than)f(the)h(receiv)o(e)g (bu\013er.)158 1583 y Fk(MPI)p 257 1583 15 2 v 17 w(RETURN)p 484 1583 V 18 w(ST)l(A)l(TUS\()g(handle,)f(coun)o(t,)g(source,)h(tag\))75 1699 y(IN)h(handle)j Fn(handle)14 b(to)g(return)h(status)f(ob)r(ject)75 1781 y Fk(OUT)i(coun)o(t)j Fn(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(receiv) o(ed.)75 1862 y Fk(OUT)i(source)j Fn(rank)14 b(of)g(message)f(sender)j(in)d (message)h(con)o(text)g(\(in)o(teger\).)75 1944 y Fk(OUT)i(tag)k Fn(tag)14 b(of)f(receiv)o(ed)i(message)f(\(in)o(teger\).)158 2060 y Fk(MPI)p 257 2060 V 17 w(CREA)l(TE)p 471 2060 V 19 w(ST)l(A)l(TUS\()h (handle\))158 2145 y Fn(Creates)g(a)f(new)g(return)h(status)g(ob)r(ject.)75 2225 y Fk(OUT)h(handle)i Fn(handle)c(to)g(return)h(status)g(ob)r(ject.)1931 2264 y Fe(TO)c(DISCUSS)158 2384 y Fj(Discussion:)158 2430 y Fi(Should)k(w)o(e)e(ha)o(v)o(e)g(p)q(ersisten)o(t/ephemeral)k(status)c(ob)r (jects?)158 2475 y(The)e(use)g(of)f(a)h(return)g(status)g(ob)r(ject,)g (rather)g(than)g(a)g(list)h(of)e(parameters)i(ma)o(y)f(simplify)i(the)e(use)g (of)f(MPI)h(routines,)75 2521 y(if)i(the)f(v)n(alues)i(stored)f(in)g(the)f (ob)r(ject)h(are)f(seldom)i(c)o(hec)o(k)o(ed.)j(A)12 b(prede\014ned)i(return) f(status)g(ob)r(ject)f(should)j(b)q(e)d(pro)o(vided,)75 2567 y(to)h(ease)g(programming.)19 b(The)12 b(main)i(reason)g(for)e(the)h(use)g (of)g(a)f(return-status)i(ob)r(ject)f(is)g(that)g(one)h(w)o(an)o(ts)e(to)h(b) q(e)g(able)h(to)75 2612 y(use)f(the)f(same)h Fd(MPI)p 364 2612 12 2 v 13 w(WAIT)e Fi(and)i Fd(MPI)p 602 2612 V 13 w(TEST)e Fi(calls)j(for)e(c)o(hec)o(king)i(on)f(di\013eren)o(t)h(t)o(yp)q(es)f(of)f (op)q(erations,)i(e.g.)j(b)q(oth)c(sends)g(and)75 2658 y(receiv)o(es,)i(and)f (p)q(erhaps)h(new)e(op)q(erations)j(in)e(the)g(future.)19 b(These)14 b(di\013eren)o(t)h(op)q(erations)g(return)g(di\013eren)o(t)g(information)75 2704 y(\(alb)q(eit)g(w)o(e)d(could)j(decree)e(that)g(a)g(send)h(returns)g (the)f(same)g(information)i(as)e(a)g(receiv)o(e\).)p eop %%Page: 26 27 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(26)158 45 y Fi(The)12 b(issue)i(of)e(memory)h(allo)q(cation)j (for)c(status)h(is)g(not)g(y)o(et)f(solv)o(ed.)18 b(There)13 b(is)g(a)f(desire)i(that)f(the)f(user)h(will)h(b)q(e)f(able)75 91 y(to)g(allo)q(cate)i(space)e(for)g(them)g(\(e.g.,)f(on)i(the)f(stac)o (k\).)158 136 y(W)m(e)j(return)g(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i(receiv) o(ed,)g(rather)e(than)i(b)o(ytes)e(receiv)o(ed)i(b)q(ecause)g(\\n)o(um)o(b)q (er)f(of)f(elemen)o(ts")i(is)75 182 y(closer)c(to)f(the)g(application)j (seman)o(tic)f(lev)o(el)f(\(user)f(do)q(es)h(not)f(need)h(to)f(b)q(e)g(a)o(w) o(are)g(of)g(the)g(size)h(of)f(elemen)o(ts\),)g(and)h(is)g(more)75 228 y(in)o(v)n(arian)o(t.)23 b(The)14 b(return)h(status)f(ob)r(ject)h (returns)g(the)f(n)o(um)o(b)q(er)h(of)f Fj(elemen)o(ts)g Fi(receiv)o(ed.)22 b(If)13 b(there)i(w)o(as)f(no)g(truncation,)75 273 y(then)h(this)g(is)g (equal)g(to)f(the)h(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)h(sen)o(t.)21 b(On)14 b(the)g(other)h(hand,)g(the)f(n)o(um)o(b)q(er)h(of)f(b)o(ytes)h (collected)h(from)75 319 y(the)d(sender)i(memory)m(,)e(the)g(n)o(um)o(b)q(er) h(of)f(b)o(ytes)h(sen)o(t)g(o)o(v)o(er)f(the)h(wire,)f(and)h(the)f(n)o(um)o (b)q(er)i(of)e(b)o(ytes)g(stored)h(in)g(the)g(receiv)o(er)75 365 y(memory)c(ma)o(y)g(all)i(b)q(e)e(di\013eren)o(t,)i(in)e(a)g (heterogeneous)i(en)o(vironmen)o(t.)18 b(The)10 b(n)o(um)o(b)q(er)g(of)g (elemen)o(ts)h(sen)o(t)f(can)h(b)q(e)f(computed)75 410 y(form)k(the)g(send)h (bu\013er)h(descriptor;)g(the)e(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(receiv)o (ed)f(can)g(b)q(e)g(computed)g(from)f(the)h(receiv)o(e)g(bu\013er)75 456 y(descriptor)g(and)e(the)g(length)i(\(in)e(b)o(ytes\))h(of)f(the)g (receiv)o(ed)h(message.)75 654 y Fh(1.8.4)55 b(Multiple)20 b(Completions)75 731 y Fn(It)c(is)g(con)o(v)o(enien)o(t)h(to)f(b)q(e)h(able)e (to)h(w)o(ait)g(for)g(the)g(completion)f(of)g(an)o(y)h(or)g(all)f(the)i(op)q (erations)f(in)g(a)g(set,)h(rather)75 781 y(than)e(ha)o(ving)g(to)g(w)o(ait)f (for)h(a)g(sp)q(eci\014c)i(message.)22 b(A)16 b(call)e(to)h Fl(MPI)p 1125 781 14 2 v 15 w(WAITANY)f Fn(or)h Fl(MPI)p 1426 781 V 16 w(TESTANY)e Fn(can)j(b)q(e)g(used)g(to)75 831 y(w)o(ait)d(for)h(the) h(completion)d(of)h(one)i(out)f(of)f(sev)o(eral)i(op)q(erations;)e(a)h(call)f (to)h Fl(MPI)p 1340 831 V 15 w(WAITALL)f Fn(can)h(b)q(e)h(used)g(to)f(w)o (ait)75 880 y(for)g(all)e(p)q(ending)i(op)q(erations)g(in)g(a)f(list.)158 966 y Fk(MPI)p 257 966 15 2 v 17 w(W)-5 b(AIT)l(ANY)17 b(\()f(list)p 628 966 V 15 w(of)p 682 966 V 16 w(handles,)e(coun)o(t,)h(index,)f(status\)) 158 1051 y Fn(Blo)q(c)o(ks)g(un)o(til)f(one)h(of)f(the)h(op)q(erations)g (asso)q(ciated)h(with)e(the)i(comm)o(uni)o(cation)c(handles)j(in)f(the)h (arra)o(y)g(has)75 1101 y(completed.)19 b(Returns)c(the)g(index)g(of)e(that)i (handle)f(in)g(the)h(arra)o(y)m(,)e(and)i(returns)h(the)f(status)g(of)f(that) g(op)q(eration)75 1151 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f(with)g(the)g (status.)19 b(The)14 b(parameters)g(are:)75 1239 y Fk(IN)i(list)p 215 1239 V 15 w(of)p 269 1239 V 17 w(handles)j Fn(list)13 b(of)g(handles)h (to)g(comm)o(unication)d(ob)r(jects.)75 1321 y Fk(IN)16 b(coun)o(t)j Fn(list)14 b(length)g(\(in)o(teger\))75 1402 y Fk(OUT)i(index)j Fn(index)14 b(of)f(handle)h(for)f(op)q(eration)h(that)g(completed)f(\(in)o (teger\).)75 1484 y Fk(OUT)j(status)j Fn(handle)14 b(that)g(is)h(asso)q (ciated)g(with)f(return)h(status)g(ob)r(ject.)20 b(Ob)r(ject)c(is)e(set)h(to) f(return)h(status)h(of)179 1534 y(op)q(eration)e(that)f(completed.)158 1622 y(The)20 b(successful)i(execution)f(of)e Fl(MPI)p 758 1622 14 2 v 15 w(WAITANY\(list)p 1037 1622 V 13 w(of)p 1094 1622 V 15 w(handles,)h(index,)h(status\))d Fn(has)i(the)h(same)75 1672 y(e\013ect)16 b(as)f(the)g(successful)h(execution)g(of)d Fl(MPI)p 801 1672 V 15 w(WAIT\(handle[i],)19 b(status\))p Fn(,)13 b(where)j Fl(i)e Fn(is)g(the)h(v)n(alue)f(returned)75 1722 y(b)o(y)i Fl(index)f Fn(and)g Fl(handle[i])f Fn(is)i(the)h Fl(i)p Fn(-th)f(handle)f(in)h(the)g(list,)g(and)g(the)g(cancellation)g(of)f (all)g(remaining)f(w)o(ait)75 1772 y(op)q(erations.)158 1821 y(If)h(more)e(then)j(one)f(op)q(eration)g(is)g(enabled)g(and)g(can)g (terminate,)f(one)h(is)g(arbitrarily)f(c)o(hosen)i(\(sub)r(ject)g(to)75 1871 y(the)e(restrictions)i(on)d(op)q(eration)h(termination)e(order,)i(and)g (fairness,)g(see)h(Chapter)g Fk(??)p Fn(\).)158 1921 y Fl(MPI)p 227 1921 V 15 w(WAITANY)20 b(\()i(list)p 548 1921 V 15 w(of)p 607 1921 V 15 w(handles,)e(count,)h(index,)f(status\))13 b Fn(is)140 2002 y Fl({MPI_WAIT)20 b(\(handle[0],)g(return_handle\);)f(index)h (=)i(0})f(||)h(...)75 2051 y(||)140 2101 y({MPI_WAIT)e(\(handle[count-1],)f (return_handle\);)f(index)j(=)h(count-1})158 2182 y Fn(\(\\)p Fc(jj)p Fn(")13 b(indicates)h(c)o(hoice;)g(one)g(of)f(the)i(alternativ)o(es)f (is)f(c)o(hosen,)i(nondeterministically)m(.\))158 2267 y Fk(MPI)p 257 2267 15 2 v 17 w(TEST)l(ANY)i(\()f(list)p 621 2267 V 15 w(of)p 675 2267 V 16 w(handles,)e(coun)o(t,)h(index,)f(status\))158 2352 y Fn(Causes)20 b(either)h(one)f(or)f(none)h(of)f(the)h(op)q(erations)g (asso)q(ciated)g(with)f(the)h(comm)o(unication)c(handles)k(to)75 2402 y(return.)f(In)13 b(the)h(former)d(case,)j(it)f(has)g(the)h(same)e (return)i(seman)o(tics)e(as)i(a)e(call)g(to)h Fl(MPI)p 1449 2402 14 2 v 15 w(WAIT)p 1552 2402 V 15 w(ANY)p Fn(.)f(In)h(the)h(latter)75 2452 y(case,)g(it)g(returns)h(a)f(v)n(alue)f(of)g({1)h(in)f Fl(index)g Fn(and)h Fl(status)e Fn(is)i(unde\014ned.)19 b(The)c(parameters)f (are:)75 2540 y Fk(IN)i(list)p 215 2540 15 2 v 15 w(of)p 269 2540 V 17 w(handles)j Fn(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob) r(jects.)75 2622 y Fk(IN)16 b(coun)o(t)j Fn(list)14 b(length)g(\(in)o (teger\))75 2704 y Fk(OUT)i(index)j Fn(index)14 b(of)f(handle)h(for)f(op)q (eration)h(that)g(completed,)f(or)h(-1)f(if)g(none)h(completed)g(\(in)o (teger\).)p eop %%Page: 27 28 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(27)75 45 y Fk(OUT)16 b(status)j Fn(handle)13 b(that)g(is)h(asso)q(ciated)g(with)f(return)h(status)g(ob)r(ject.)19 b(Set)14 b(to)f(return)h(status)g(of)f(op)q(eration)179 95 y(that)h(completed,)f(if)g(an)o(y;)g(unde\014ned)i(when)f Fl(index)21 b(=)g(-1)p Fn(.)158 222 y Fk(MPI)p 257 222 15 2 v 17 w(W)-5 b(AIT)l(ALL\(list)p 580 222 V 15 w(of)p 634 222 V 17 w(handles,)13 b(coun)o(t,)i(list)p 1047 222 V 15 w(of)p 1101 222 V 17 w(statuss\))158 307 y Fn(Blo)q(c)o(ks)k(un)o(til)e(all)g(comm)o(unicati)o(on)e(op)q(erations) k(asso)q(ciated)g(with)f(handles)g(in)g(the)g(list)g(complete,)g(and)75 357 y(return)d(the)g(status)f(of)g(all)e(these)j(op)q(erations.)k(The)14 b(parameters)g(are:)75 448 y Fk(IN)i(list)p 215 448 V 15 w(of)p 269 448 V 17 w(handles)j Fn(list)13 b(of)g(handles)h(to)g(comm)o(unication)d (ob)r(jects.)75 531 y Fk(IN)16 b(coun)o(t)j Fn(lists)14 b(length)g(\(in)o (teger\))75 614 y Fk(OUT)i(list)p 266 614 V 15 w(of)p 320 614 V 16 w(status)k Fn(Must)c(ha)o(v)o(e)f(the)h(same)f(length)g(as)h(the)g (\014rst)g(list.)23 b(Eac)o(h)15 b(return)i(status)f(ob)r(ject)h(is)e(set)179 664 y(to)f(the)g(return)h(status)g(of)e(the)i(corresp)q(onding)f(op)q (eration)g(in)g(the)g(\014rst)h(list.)158 834 y Fj(Discussion:)158 879 y Fi(The)9 b(fairness)i(requiremen)o(t)g(giv)o(en)g(in)f(Chapter)g Fj(??)16 b Fi(implies)c(that)e(the)f(use)h(of)f Fd(WAIT)p 1374 879 12 2 v 13 w(ANY)f Fi(cannot)j(lead)f(to)g(starv)n(ation:)75 925 y(If)15 b(the)i(sending)g(pro)q(cess)g(has)g(issued)g(a)f(send)h (complete)g(op)q(eration,)i(and)d(the)g(receiving)j(pro)q(cess)d(rep)q (eatedly)i(p)q(ost)f(a)75 971 y(receiv)o(e)e(for)f(the)h(message)g(sen)o(t,)f (then)h(that)f(message)h(m)o(ust)g(b)q(e)f(ev)o(en)o(tually)j(receiv)o(ed.)22 b(Section)16 b Fj(??)21 b Fi(has)14 b(not)h(y)o(et)f(b)q(een)75 1016 y(discussed.)35 b(The)18 b(fairness)h(requiremen)o(t)h(can)f(b)q(e)f (attac)o(k)o(ed)h(either)g(for)f(b)q(eing)i(to)q(o)f(w)o(eak)f(\(ev)o(en)o (tually)j(is)d(not)h(go)q(o)q(d)75 1062 y(enough\),)13 b(or)f(to)q(o)g (strong)g(\(hard)h(to)f(implemen)o(t\).)18 b(In)12 b(the)g(later)g(case,)g (if)g(the)g(requiremen)o(t)i(of)d(fairness)i(is)g(dropp)q(ed)g(from)75 1108 y(MPI)g(implemen)o(tation)q(s,)j(then)d(some)h(mec)o(hanism)h(need)e(b)q (e)h(pro)o(vided)h(to)e(the)h(user)f(to)h(ac)o(hiev)o(e)g(fairness)h(b)o(y)e (his)h(or)g(her)75 1153 y(o)o(wn)g(devices.)22 b(One)14 b(suc)o(h)h(prop)q (osal)h(is)f(to)f(b)q(e)g(able)i(to)e(sp)q(ecify)h(a)f(rotating)h(priorit)o (y)h(order)f(on)g(the)f(op)q(erations)i(p)q(osted)75 1199 y(b)o(y)d(a)g Fd(MPI)p 222 1199 V 13 w(WAITANY)p Fi(:)d(searc)o(h)k(the)f(list)h(sequen)o (tially)n(,)h(starting)f(from)f(a)g(user)g(sp)q(eci\014ed)i(p)q(osition.)428 b Fe(TO)11 b(DISCUSS)158 1245 y Fi(A)i(prop)q(osal)i(w)o(as)e(made)h(for)f (an)h Fd(MPI)p 707 1245 V 13 w(TESTALL)c Fi(function)15 b(that)e(returns)h (successfully)i(\()p Fd(flag)i(=)h(true)p Fi(\))11 b(if)j(all)h(op)q(er-)75 1290 y(ations)f(in)g(the)f(list)h(ha)o(v)o(e)g(terminated,)g(unsuccessfully)i (\()p Fd(flag)i(=)h(false)p Fi(\),)11 b(otherwise.)75 1510 y Fm(1.9)70 b(Blo)r(c)n(king)21 b(Comm)n(unication)75 1601 y Fn(Blo)q(c)o(king)14 b(send)i(and)f(receiv)o(e)h(op)q(erations)g(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(erations)j(in)o(to)g(one)g(call.) 21 b(The)75 1651 y(op)q(eration)16 b(returns)j(only)c(when)i(the)h(comm)o (unicati)o(on)c(completes)i(and)g(no)h(comm)o(unicatio)o(n)d(ob)r(ject)j(p)q (ersists)75 1701 y(after)d(the)h(call)e(completed.)k(Ho)o(w)o(ev)o(er,)d(the) h(bu\013er)f(descriptor)i(ob)r(ject)e(needs)i(b)q(e)e(created)h(ahead)f(of)g (the)g(call.)158 1751 y(W)m(e)9 b(use)h(the)h(same)d(naming)f(con)o(v)o(en)o (tion)j(as)f(for)g(comm)o(unication)d(handle)k(creation:)16 b(a)9 b(pre\014x)h(of)f Fl(R)h Fn(indicates)75 1801 y(the)k Fl(ready)f Fn(mo)q(de,)g(and)g(a)h(pre\014x)h(of)e Fl(S)g Fn(indicates)i(the) f(sync)o(hronous)h(mo)q(de.)158 1886 y Fk(MPI)p 257 1886 15 2 v 17 w(SEND)h(\(bu\013er)p 565 1886 V 15 w(handle,)e(dest,)h(tag,)h(con)o (text\))158 1971 y Fn(Blo)q(c)o(king)d(standard)i(send.)158 2021 y Fl(MPI)p 227 2021 14 2 v 15 w(SEND)21 b(\(buffer)p 505 2021 V 14 w(handle,)g(dest,)f(tag,)h(context\))13 b Fn(is)75 2104 y Fl(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)75 2154 y(MPI_EPHEMERAL\))75 2203 y(MPI_START\(handle\))75 2253 y(MPI_WAIT\(handle,)d(null\))158 2372 y Fk(MPI)p 257 2372 15 2 v 17 w(RSEND)e(\(bu\013er)p 601 2372 V 15 w(handle,)e(dest,)h(tag,)h (con)o(text\))158 2457 y Fn(blo)q(c)o(king)d(ready)h(send)158 2507 y Fl(MPI)p 227 2507 14 2 v 15 w(RSEND)21 b(\(buffer)p 527 2507 V 14 w(handle,)f(dest,)h(tag,)g(context\))12 b Fn(is)75 2590 y Fl(MPI_INIT_RSEND\(ha)o(ndle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_EPHEMERAL\))75 2640 y(MPI_START\(handle\))75 2689 y(MPI_WAIT\(handle,)e(null\))p eop %%Page: 28 29 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(28)158 45 y Fk(MPI)p 257 45 15 2 v 17 w(SSEND)16 b(\(bu\013er)p 592 45 V 15 w(handle,)e(dest,)h(tag,)h(con)o(text\))158 130 y Fn(Blo)q(c)o(king)d(sync)o(hronous)i(send.)158 180 y Fl(MPI)p 227 180 14 2 v 15 w(SSEND)21 b(\(buffer)p 527 180 V 14 w(handle,)f(dest,)h(tag,)g(context\))12 b Fn(is)75 258 y Fl(MPI_INIT_SSEND\(ha)o(ndle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_EPHEMERAL\))75 307 y(MPI_START\(handle\))75 357 y(MPI_WAIT\(handle,)e(null\))158 470 y Fk(MPI)p 257 470 15 2 v 17 w(RECV\(bu\013er)p 554 470 V 16 w(handle,)c(source,)h(tag,)h(con)o (text,)f(status\))158 555 y Fn(Blo)q(c)o(king)e(receiv)o(e)158 605 y Fl(MPI)p 227 605 14 2 v 15 w(RECV\(buffer)p 484 605 V 14 w(handle,)20 b(source,)g(tag,)h(context,)f(status\))13 b Fn(is)75 682 y Fl(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i (tag,)g(context,)f(MPI_EPHEMERAL\))75 732 y(MPI_START\(handle\))75 782 y(MPI_WAIT\(handle,)e(status\))158 938 y Fj(Implemen)o(tation)13 b(note:)158 984 y Fi(While)20 b(these)f(functions)g(can)g(b)q(e)f(implemen)o (ted)i(via)f(calls)h(to)e(functions)i(that)e(implemen)o(t)i(sub)q(op)q (erations,)i(as)75 1029 y(describ)q(ed)16 b(in)g(this)f(subsection,)h(an)f (e\016cien)o(t)g(implemen)o(tation)j(ma)o(y)d(optimize)h(a)o(w)o(a)o(y)e (these)h(m)o(ultiple)i(calls,)f(pro)o(vided)75 1075 y(it)d(do)q(es)h(not)f(c) o(hange)h(the)f(b)q(eha)o(vior)i(of)e(correct)g(programs.)75 1294 y Fm(1.10)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)75 1385 y Fn(Non)o(blo)q(c)o(king)11 b(send)i(and)f(receiv)o(e)h(op)q(erations)f (com)o(bine)f(the)i(\014rst)g(t)o(w)o(o)e(sub)q(op)q(erations)i(\()p Fl(INIT)f Fn(and)g Fl(START)p Fn(\))f(in)o(to)75 1434 y(one)16 b(call.)21 b(They)16 b(use)g(ephemeral)f(comm)o(uni)o(cation)d(ob)r(jects,)17 b(so)e(that)g(the)h(op)q(eration)f(is)h(completed,)e(and)h(the)75 1484 y(asso)q(ciated)i(resources)h(are)e(freed,)h(b)o(y)f(using)g(one)g(of)f (the)i(functions)f Fl(MPI)p 1265 1484 V 15 w(WAIT,)21 b(MPI)p 1477 1484 V 15 w(TEST,)g(MPI)p 1689 1484 V 15 w(WAITANY,)75 1534 y(MPI)p 144 1534 V 15 w(TESTANY)p Fn(,)13 b(or)i Fl(MPI)p 456 1534 V 15 w(WAITALL)p Fn(.)e(Here,)i(to)q(o,)g(a)f(bu\013er)i(ob)r(ject)g (has)e(to)h(b)q(e)h(created)g(ahead)f(of)f(the)h(comm)o(uni-)75 1584 y(cation)f(initiation)d(op)q(eration.)158 1634 y(W)m(e)j(use)i(the)f (same)f(naming)e(con)o(v)o(en)o(tion)i(as)h(for)f(blo)q(c)o(king)g(op)q (erations:)20 b(a)14 b(pre\014x)h(of)f Fl(R)h Fn(\()p Fl(S)p Fn(\))f(indicates)h(the)75 1683 y Fl(READY)g Fn(\()p Fk(SYNCHR)o(ONOUS)p Fn(\))i(mo)q(de.)25 b(In)16 b(addition,)g(a)g(pre\014x)h(of)f Fk(I)h Fn(is)f(used)h(to)g(indicate)f Fg(imme)n(diate)g Fn(\(i.e.,)75 1733 y(non)o(blo)q(c)o(king\))d(execution.)158 1818 y Fk(MPI)p 257 1818 15 2 v 17 w(ISEND)j(\(handle,)e(bu\013er)p 750 1818 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))158 1904 y Fn(Non)o(blo)q(c)o (king)d(standard)h(send.)158 1954 y Fl(MPI)p 227 1954 14 2 v 15 w(ISEND)21 b(\(handle,)f(buffer)p 701 1954 V 14 w(handle,)h(dest,)g (tag,)g(context\))12 b Fn(is)75 2031 y Fl(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)f(MPI_EPHEMERAL\))75 2081 y(MPI_START\(handle\))158 2193 y Fk(MPI)p 257 2193 15 2 v 17 w(IRSEND)c(\(handle,)e(bu\013er)p 786 2193 V 15 w(handle,)g(dest,)h (tag,)h(con)o(text\))158 2279 y Fn(Non)o(blo)q(c)o(king)d(ready)h(send.)158 2329 y Fl(MPI)p 227 2329 14 2 v 15 w(IRSEND)21 b(\(handle,)f(buffer)p 723 2329 V 14 w(handle,)h(dest,)f(tag,)h(context\))12 b Fn(is)75 2406 y Fl(MPI_INIT_RSEND\(ha)o(ndle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_EPHEMERAL\))75 2456 y(MPI_START\(handle\))158 2569 y Fk(MPI)p 257 2569 15 2 v 17 w(ISSEND)c(\(handle,)d(bu\013er)p 776 2569 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text\))158 2654 y Fn(Non)o(blo)q(c)o(king)d(sync)o(hronous)i(send.)158 2704 y Fl(MPI)p 227 2704 14 2 v 15 w(ISSEND)21 b(\(handle,)f(buffer)p 723 2704 V 14 w(handle,)h(dest,)f(tag,)h(context\))12 b Fn(is)p eop %%Page: 29 30 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(29)75 45 y Fl(MPI_INIT_SSEND\(ha)o(ndle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)f(MPI_EPHEMERAL\))75 95 y(MPI_START\(handle\))158 206 y Fk(MPI)p 257 206 15 2 v 17 w(IRECV\(handle,)15 b(bu\013er)p 740 206 V 15 w(handle,)f(source,)i(tag,)f (con)o(text\))158 291 y Fn(Non)o(blo)q(c)o(king)e(receiv)o(e)158 341 y Fl(MPI)p 227 341 14 2 v 15 w(IRECV\(handle,)19 b(buffer)p 679 341 V 15 w(handle,)h(source,)g(tag,)h(context\))12 b Fn(is)75 417 y Fl(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g (context,)f(MPI_EPHEMERAL\))75 467 y(MPI_START\(handle\))75 603 y Fm(1.11)70 b(Con)n(tiguous)23 b(Bu\013er)g(Comm)n(unication)e(Op)r (erations)75 694 y Fn(The)c(most)f(frequen)o(t)i(t)o(yp)q(e)g(of)e(bu\013er)i (used)g(is)f(a)g(bu\013er)h(with)f(one)g(con)o(tiguous)g(comp)q(onen)o(t.)27 b(W)m(e)16 b(sp)q(ecialize)75 743 y(the)e(functions)f(in)f(the)i(t)o(w)o(o)e (previous)i(subsections)g(to)f(this)g(case,)h(th)o(us)f(a)o(v)o(oiding)e(the) j(need)g(for)f(the)g(creation)h(of)75 793 y(a)h(bu\013er)i(descriptor)f(ob)r (ject.)24 b(W)m(e)15 b(use)h(the)g(same)e(naming)g(sc)o(heme)h(used)h(in)f (the)h(previous)g(subsections,)h(and)75 843 y(app)q(end)d(a)g Fk(C)g Fn(in)g(the)g(function)g(name,)e(for)i Fl(CONTIGUOUS)p Fn(.)158 928 y Fk(MPI)p 257 928 15 2 v 17 w(SENDC)j(\(start,)d(coun)o(t,)h (datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))158 1014 y Fn(Blo)q(c)o(king)d (send)i(of)e(con)o(tiguous)h(bu\013er)h(in)e(standard)i(mo)q(de.)158 1063 y Fl(MPI)p 227 1063 14 2 v 15 w(SENDC)21 b(\(start,)f(count,)h (datatype,)f(dest,)g(tag,)h(context\))13 b Fn(is)75 1139 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 1189 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h(start,)i(count,)f (datatype\))75 1239 y(MPI_COMMIT_BUFFER)o(\()f(buffer_handle\))75 1289 y(MPI_SEND)h(\(buffer_handle,)f(count,)h(dest,)h(tag,)g(context\))158 1400 y Fk(MPI)p 257 1400 15 2 v 17 w(RSENDC)16 b(\(start,)f(coun)o(t,)f (datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))158 1485 y Fn(Blo)q(c)o(king)d (send)i(of)e(con)o(tiguous)h(bu\013er)h(in)e(ready)i(mo)q(de.)158 1535 y Fl(MPI)p 227 1535 14 2 v 15 w(RSENDC)21 b(\(start,)f(count,)h (datatype,)e(dest,)i(tag,)g(context\))12 b Fn(is)75 1611 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 1661 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h(start,)i(count,)f (datatype\))75 1710 y(MPI_COMMIT_BUFFER)o(\()f(buffer_handle\))75 1760 y(MPI_RSEND\()h(buffer_handle,)f(count,)h(dest,)h(tag,)g(context\))158 1872 y Fk(MPI)p 257 1872 15 2 v 17 w(SSENDC)16 b(\(start,)f(coun)o(t,)f (datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))158 1957 y Fn(Blo)q(c)o(king)d (send)i(of)e(con)o(tiguous)h(bu\013er)h(in)e(sync)o(hron)o(uous)i(mo)q(de.) 158 2007 y Fl(MPI)p 227 2007 14 2 v 15 w(SSENDC)21 b(\(start,)f(count,)h (datatype,)e(dest,)i(tag,)g(context\))12 b Fn(is)75 2083 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2132 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h(start,)i(count,)f (datatype\))75 2182 y(MPI_COMMIT_BUFFER)o(\()f(buffer_handle\))75 2232 y(MPI_SSEND\()h(buffer_handle,)f(count,)h(dest,)h(tag,)g(context\))158 2343 y Fk(MPI)p 257 2343 15 2 v 17 w(RECV)o(C)c(\(start,)e(coun)o(t,)f(datat) o(yp)q(e,)h(source,)g(tag,)h(con)o(text,)f(status\))158 2428 y Fn(Blo)q(c)o(king)e(receiv)o(e)i(of)f(con)o(tiguous)f(bu\013er.)158 2478 y Fl(MPI)p 227 2478 14 2 v 15 w(RECVC)21 b(\(start,)f(count,)h (datatype,)f(source,)g(tag,)h(context,)f(status\))13 b Fn(is)75 2554 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2604 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h(start,)i(count,)f (datatype\))75 2654 y(MPI_COMMIT_BUFFER)o(\()f(buffer_handle\))75 2704 y(MPI_RECV\()h(buffer_handle,)f(source,)h(tag,)h(context,)f (return_handle\))p eop %%Page: 30 31 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(30)158 45 y Fk(MPI)p 257 45 15 2 v 17 w(ISENDC)17 b(\(handle,)c(start,)i(coun)o(t,)g(datat)o(yp)q(e,)g(dest,)g(tag,)g(con)o (text\))158 130 y Fn(Non)o(blo)q(c)o(king)e(send)h(of)g(con)o(tiguous)f (bu\013er)i(in)f(standard)g(mo)q(de.)158 180 y Fl(MPI)p 227 180 14 2 v 15 w(ISENDC)21 b(\(handle,)f(start,)g(count,)h(datatype,)f(dest,)h (tag,)g(context\))12 b Fn(is)75 263 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 313 y(MPI_APPEND_CONTIG)o(UOUS\()f (buffer_handle,)h(start,)i(count,)f(datatype\))75 363 y(MPI_COMMIT_BUFFER)o (\()f(buffer_handle\))75 413 y(MPI_ISEND\()h(handle,)g(buffer_handle,)f (dest,)i(tag,)g(context\))158 531 y Fk(MPI)p 257 531 15 2 v 17 w(IRSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)g(datat)o(yp)q(e,)f(dest,)h (tag,)h(con)o(text\))158 616 y Fn(Non)o(blo)q(c)o(king)d(send)h(of)g(con)o (tiguous)f(bu\013er)i(in)f(ready)g(mo)q(de.)158 666 y Fl(MPI)p 227 666 14 2 v 15 w(IRSENDC)20 b(\(handle,)h(start,)f(count,)h(datatype,)f (dest,)g(tag,)h(context\))13 b Fn(is)75 749 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 799 y(MPI_APPEND_CONTIG)o(UOUS\()f (buffer_handle,)h(start,)i(count,)f(datatype\))75 849 y(MPI_COMMIT_BUFFER)o (\()f(buffer_handle\))75 899 y(MPI_IRSEND\()h(handle,)g(buffer_handle,)f (dest,)i(tag,)f(context\))158 1017 y Fk(MPI)p 257 1017 15 2 v 17 w(ISSENDC)c(\(handle,)e(start,)h(coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g (tag,)h(con)o(text\))158 1102 y Fn(Non)o(blo)q(c)o(king)d(send)h(of)g(con)o (tiguous)f(bu\013er)i(in)f(sync)o(hronous)h(mo)q(de.)158 1152 y Fl(MPI)p 227 1152 14 2 v 15 w(ISSENDC)20 b(\(handle,)h(start,)f(count,)h (datatype,)f(dest,)g(tag,)h(context\))13 b Fn(is)75 1235 y Fl(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 1285 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h(start,)i(count,)f (datatype\))75 1335 y(MPI_COMMIT_BUFFER)o(\()f(buffer_handle\))75 1385 y(MPI_ISSEND\()h(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158 1503 y Fk(MPI)p 257 1503 15 2 v 17 w(IRECV)o(C\(handle,)15 b(start,)g(coun)o(t,)f(datat)o(yp)q(e,)h(source,)g(tag,)h(con)o(text\))158 1588 y Fn(Non)o(blo)q(c)o(king)d(receiv)o(e)i(of)e(con)o(tiguous)h(bu\013er.) 158 1638 y Fl(MPI)p 227 1638 14 2 v 15 w(IRECVC\(handle,)19 b(start,)i(count,)f(datatype,)g(source,)g(tag,)h(context\))13 b Fn(is)75 1721 y Fl(MPI_CREATE_BUFFER)18 b(\(buffer_handle,)h (MPI_EPHEMERAL\))75 1771 y(MPI_APPEND_CONTIG)o(UOUS\()f(buffer_handle,)h (start,)i(count,)f(datatype\))75 1821 y(MPI_COMMIT_BUFFER)o(\()f (buffer_handle\))75 1871 y(MPI_IRECV\()h(handle,)g(buffer_handle,)f(source,)h (tag,)h(context\))75 2008 y Fm(1.12)70 b(Prob)r(e)23 b(and)h(Cancel)75 2099 y Fn(The)18 b Fl(MPI)p 233 2099 V 15 w(PROBE)e Fn(op)q(eration)h(allo)o (ws)f(incoming)f(messages)i(to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)g(without)e (actually)h(receiving)75 2149 y(them.)22 b(The)16 b(user)g(can)g(then)g (decide)h(where)f(to)g(receiv)o(e)g(them,)f(based)h(on)f(the)h(information)d (returned)k(b)o(y)e(the)75 2198 y(prob)q(e)i(\(basically)m(,)f(the)h (information)d(on)i(the)i(message)e(en)o(v)o(elop)q(e\).)27 b(An)17 b(additional)e(function,)h Fl(MPI)p 1716 2198 V 16 w(GET)p 1798 2198 V 15 w(LEN)75 2248 y Fn(allo)o(ws)11 b(the)j(amoun)o(t)c (of)i(storage)h(needed)i(to)d(receiv)o(e)i(the)f(message)g(to)f(b)q(e)h (computed,)f(when)h(this)g(length)g(is)f(not)75 2298 y(readily)h(computed)h (from)e(the)i(information)d(returned)16 b(b)o(y)e Fl(MPI)p 1094 2298 V 15 w(PROBE)p Fn(.)158 2348 y(The)f Fl(MPI)p 311 2348 V 15 w(CANCEL)e Fn(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o (unicatio)o(ns)d(to)i(b)q(e)h(cancelled.)19 b(This)12 b(is)g(required)h(for) 75 2398 y(clean)o(up.)20 b(P)o(osting)14 b(a)h(send)g(or)g(a)f(receiv)o(e)i (ties)f(user)g(resources)i(\(send)f(or)e(receiv)o(e)i(bu\013ers\),)g(and)e(a) h(cancel)g(ma)o(y)75 2447 y(b)q(e)g(needed)g(to)f(free)g(these)i(resources)g (gracefully)m(.)158 2533 y Fk(MPI)p 257 2533 15 2 v 17 w(IPR)o(OBE\()f (source,)g(tag,)h(con)o(text,)f(\015ag,)h(t)o(yp)q(e,)f(status\))75 2659 y(IN)h(source)k Fn(rank)14 b(in)f(con)o(text)i(of)e(source,)i(or)f Fl(MPI)p 897 2659 14 2 v 15 w(ANY)p 978 2659 V 15 w(SOURCE)f Fn(\(in)o(teger\).)p eop %%Page: 31 32 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(31)75 45 y Fk(IN)16 b(tag)21 b Fn(user)15 b(tag)e(for)h (messages)g(receiv)o(ed)h(with)f(this)f(handle,)h(or)g Fl(MPI)p 1232 45 14 2 v 15 w(ANY)p 1313 45 V 15 w(TAG)f Fn(\(in)o(teger\).)75 128 y Fk(IN)j(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i (with)d(this)h(handle.)75 211 y Fk(OUT)i(\015ag)k Fn(\(logical\))75 294 y Fk(IN)c(t)o(yp)q(e)k Fn(assumed)14 b(t)o(yp)q(e)g(of)f(data)h(in)f (message)h(\(status)h(v)n(ariable\).)75 377 y Fk(OUT)h(status)j Fn(handle)14 b(that)g(is)f(asso)q(ciated)i(with)f(return)h(status)f(ob)r (ject.)158 468 y Fl(MPI)p 227 468 V 15 w(IPROBE)9 b Fn(returns)j Fl(flag)21 b(=)g(true)9 b Fn(if)h(there)h(is)f(a)g(message)g(that)g(can)g(b)q (e)h(receiv)o(ed)g(and)f(that)h(matc)o(hes)e(the)75 517 y(pattern)j(sp)q (eci\014ed)h(b)o(y)e(the)h(parameters)g Fl(source)p Fn(,)e Fl(tag)p Fn(,)g(and)i Fl(context)p Fn(.)j(It)d(returns)h Fl(flag)21 b(=)g(false)p Fn(,)10 b(otherwise.)75 567 y(If)18 b Fl(MPI)p 190 567 V 15 w(PROBE)f Fn(returns)i Fl(flag)i(=)h(true)p Fn(,)c(then)h(the)f (length,)h(source)g(and)f(tag)g(of)g(the)g(message)g(matc)o(hed)g(are)75 617 y(returned)i(in)d(the)i(return)g(status)g(ob)r(ject.)32 b(These)19 b(are)g(the)g(same)e(v)n(alues)h(that)g(w)o(ould)f(ha)o(v)o(e)h(b) q(een)h(returned)75 667 y(b)o(y)d(a)g(call)f(to)h Fl(MPI)p 372 667 V 15 w(RECV)g Fn(or)g(to)g Fl(MPI)p 663 667 V 15 w(SRECV)f Fn(executed)j(at)e(the)h(same)e(p)q(oin)o(t)h(in)g(the)g(program)f(\(with)h (a)g(ca)o(v)o(eat)75 717 y(concerning)e(length;)f(see)h(b)q(elo)o(w\).)k (These)c(v)n(alues)f(can)g(b)q(e)h(deco)q(ded)g(from)e(the)h(return)i(status) f(ob)r(ject)g(using)e(the)75 766 y Fl(MPI)p 144 766 V 15 w(RETURN)p 291 766 V 15 w(STAT)h Fn(function.)k(The)e(return)g(status)f(ob)r(ject)h(is)f (unde\014ned)h(if)e Fl(flag=false)p Fn(.)158 816 y(The)18 b(length)g(v)n (alue)f(returned)i(b)o(y)e(the)i(return)f(status)h(ob)r(ject)f(is)g (\(correctly\))h(the)f(n)o(um)o(b)q(er)f(of)g(elemen)o(ts)75 866 y(in)h(the)h(message,)f(pro)o(vided)h(that)f(all)f(elemen)o(ts)h(in)g (the)h(message)f(are)h(of)e(the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(b)o(y)e(the)h Fl(type)75 916 y Fn(parameter;)13 b(otherwise)i(the)f(length)g(v)n(alue)f (returned)j(is)e(unde\014ned.)158 966 y(A)g(subsequen)o(t)j(receiv)o(e)e (executed)h(with)e(the)h(same)f(con)o(text,)g(and)h(the)g(source)g(and)f(tag) g(returned)i(b)o(y)e(the)75 1015 y(call)f(to)h Fl(MPI)p 271 1015 V 15 w(IPROBE)f Fn(will)f(receiv)o(e)k(the)e(message)g(that)g(w)o(as)g (matc)o(hed)f(b)o(y)h(the)g(prob)q(e,)h(if)e(no)g(other)i(in)o(terv)o(ening) 75 1065 y(receiv)o(e)d(o)q(ccurred)g(after)f(the)g(prob)q(e.)18 b(If)10 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o(ultithreaded,)g(it)g(is)h (the)g(user)g(resp)q(onsibilit)o(y)75 1115 y(to)j(ensure)h(that)f(the)h(last) e(condition)g(holds.)158 1200 y Fk(MPI)p 257 1200 15 2 v 17 w(PR)o(OBE\()i(source,)g(tag,)h(con)o(text,)f(t)o(yp)q(e,)g(status\))75 1327 y(IN)h(source)k Fn(rank)14 b(in)f(con)o(text)i(of)e(source,)i(or)f Fl(MPI)p 897 1327 14 2 v 15 w(ANY)p 978 1327 V 15 w(SOURCE)f Fn(\(in)o(teger\).)75 1410 y Fk(IN)j(tag)21 b Fn(user)15 b(tag)e(for)h (messages)g(receiv)o(ed)h(with)f(this)f(handle,)h(or)g Fl(MPI)p 1232 1410 V 15 w(ANY)p 1313 1410 V 15 w(TAG)f Fn(\(in)o(teger\).)75 1492 y Fk(IN)j(con)o(text)k Fn(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i (with)d(this)h(handle.)75 1575 y Fk(IN)i(t)o(yp)q(e)k Fn(assumed)14 b(t)o(yp)q(e)g(of)f(data)h(in)f(message)h(\(status)h(v)n(ariable\).)75 1658 y Fk(OUT)h(status)j Fn(handle)14 b(that)g(is)f(asso)q(ciated)i(with)f (return)h(status)f(ob)r(ject.)158 1749 y Fl(MPI)p 227 1749 V 15 w(PROBE)f Fn(b)q(eha)o(v)o(es)i(lik)o(e)f Fl(MPI)p 663 1749 V 15 w(IPROBE)f Fn(except)i(that)g(it)f(is)g(a)g(blo)q(c)o(king)f(call)g (whic)o(h)i(returns)g(only)f(after)g(a)75 1799 y(matc)o(hing)e(message)i(has) g(b)q(een)h(found.)158 1927 y Fj(Discussion:)158 1973 y Fi(MPI)10 b(guaran)o(tees)h(that)e(successiv)o(e)j(messages)e(sen)o(t)g(from)f(a)h (source)g(to)g(a)g(destination)i(within)f(the)f(same)g(con)o(text)g(are)75 2019 y(receiv)o(ed)i(in)f(the)g(order)g(they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h(either)f(explicitly)j(or)c (implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75 2064 y(to)i(manage)h (messages)g(b)q(et)o(w)o(een)f(eac)o(h)h(pair)g(of)f(pro)q(cesses,)h(for)f (eac)o(h)g(con)o(text.)17 b Fd(MPI)p 1309 2064 12 2 v 13 w(PROBE)10 b Fi(returns)j(information)h(on)f(the)75 2110 y(\014rst)i(matc)o(hing)g (message)g(in)h(this)f(FIF)o(O;)e(this)j(will)g(also)f(b)q(e)g(the)f(message) h(receiv)o(ed)h(b)o(y)f(the)f(\014rst)h(subsequen)o(t)h(receiv)o(e)75 2156 y(with)d(the)h(same)f(source,)g(tag)g(and)h(con)o(text)f(as)h(the)f (message)g(matc)o(hed)h(b)o(y)f Fd(MPI)p 1261 2156 V 13 w(PROBE)p Fi(.)158 2201 y(Message)19 b(passing)g(in)g(MPI)e(can)i(b)q(e)f(implemen)o (ted)i(without)e(app)q(ending)j(t)o(yp)q(e)d(information)i(to)d(messages.)32 b(A)75 2247 y(message)14 b(is)g(merely)h(a)f(string)g(of)g(b)o(ytes)g(and)g (the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h(a)e(sequence)i (of)e(t)o(yp)q(ed)i(elemen)o(ts)75 2293 y(is)f(done)f(using)i(the)e (information)i(in)e(the)g(bu\013er)h(descriptors)h(at)e(eac)o(h)g(end.)18 b(The)13 b(abilit)o(y)i(to)e(use)g(suc)o(h)h(implemen)o(tation)75 2338 y(strategy)d(is)g(deemed)g(to)f(b)q(e)h(an)g(imp)q(ortan)o(t)h(goal.)17 b(In)10 b(suc)o(h)h(implemen)o(tation)q(,)i(when)e(a)f(message)h(arriv)o(es,) h(it)f(is)g(not)f(kno)o(wn)75 2384 y(ho)o(w)16 b(man)o(y)g(elemen)o(ts)h(it)g (con)o(tains,)h(or)e(ev)o(en)g(ho)o(w)g(m)o(uc)o(h)g(storage)h(is)g(needed)g (to)e(receiv)o(e)i(that)f(message)h(\(b)q(ecause)g(of)75 2430 y(p)q(ossible)d(represen)o(tation)g(con)o(v)o(ersion)f(in)g(a)f (heterogeneous)h(en)o(vironmen)o(t\).)18 b(The)12 b(prob)q(e)h(function)g (cannot)f(use)g(a)g(bu\013er)75 2475 y(descriptor;)18 b(this)f(defeats)f(the) g(purp)q(ose)h(of)e(probing)j(in)e(order)h(to)e(decide)i(where)f(to)g(receiv) o(e)h(a)e(message.)26 b(Therefore,)75 2521 y(prob)q(e)17 b(cannot,)h(in)f (general,)h(return)f(correct)f(length)i(information.)28 b(Still,)19 b(it)e(is)g(often)f(the)g(case)h(that)f(prob)q(e)h(is)g(used)75 2567 y(to)f(decide)h(ho)o(w)f(m)o(uc)o(h)h(storage)f(to)g(allo)q(cate)i(in)f (order)f(to)g(receiv)o(e)h(a)f(message.)26 b(Enco)q(ding)18 b(suc)o(h)f(information)h(in)f(the)75 2612 y(message)d(tag)f(is)g(deemed)h (to)f(b)q(e)g(to)q(o)g(a)o(wkw)o(ard,)g(and)g(it)g(is)h(deemed)g(imp)q(ortan) o(t)g(for)f Fd(MPI)p 1396 2612 V 13 w(PROBE)d Fi(to)j(return)h(some)f(useful) 75 2658 y(size)k(information.)28 b(The)16 b(curren)o(t)h(de\014nition)i(of)c Fd(MPI)p 884 2658 V 13 w(PROBE)f Fi(is)j(a)f(compromise)i(b)q(et)o(w)o(een)e (these)h(t)o(w)o(o)e(goals.)28 b(F)m(or)16 b(the)75 2704 y(most)e(common)g (case)h(of)e(messages)i(where)f(all)h(en)o(tries)g(ha)o(v)o(e)f(the)g(same)g (t)o(yp)q(e,)g Fd(MPI)p 1318 2704 V 13 w(PROBE)e Fi(returns)i(the)g(correct)g (length)p eop %%Page: 32 33 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(32)75 45 y Fi(information;)20 b(the)d(more)g(esoteric)g(case)g (is)g(handled)i(b)o(y)e(the)g Fd(MPI)p 1076 45 12 2 v 13 w(GET)p 1149 45 V 12 w(LEN)f Fi(that)h(is)g(describ)q(ed)h(b)q(elo)o(w.)29 b(The)16 b(curren)o(t)75 91 y(solution)g(sa)o(v)o(es)e(us)g(the)f(need)h(for) g(one)g(additional)i(w)o(ord)e(p)q(er)g(message)g(that)g(w)o(ould)g (otherwise)h(b)q(e)f(needed)g(to)g(transfer)75 136 y(the)f(message)h(length)g (\(in)g(elemen)o(ts\))g(with)f(the)g(message.)158 304 y Fk(MPI)p 257 304 15 2 v 17 w(GET)p 376 304 V 18 w(LEN\()i(coun)o(t,)g(status,)g (bu\013er)p 948 304 V 15 w(descriptor\))158 390 y Fn(Computes)g(the)i(n)o(um) o(b)q(er)e(of)h(elemen)o(ts)g(that)g(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed,)i (if)d(the)i(message)e(that)h(is)g(asso)q(ciated)75 439 y(with)e(the)i(return) f(status)h(handle)f(w)o(ould)e(b)q(e)j(receiv)o(ed)g(in)e(the)h(bu\013er)h (asso)q(ciated)f(with)g(the)g(bu\013er)h(descriptor)75 489 y(handle.)75 564 y Fk(OUT)g(coun)o(t)j Fn(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(that)g(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed)h(\(in)o (teger\))75 643 y Fk(IN)h(status)k Fn(handle)13 b(to)h(return)h(status)g (descriptor)75 721 y Fk(IN)h(bu\013er)p 273 721 V 16 w(descriptor)i Fn(handle)13 b(to)h(bu\013er)h(descriptor)158 874 y Fj(Discussion:)158 920 y Fi(In)d(order)h(to)g(supp)q(ort)g(this)h(function,)f(an)g(additional)j (\014eld)d(is)g(needed)h(in)f(the)g(return)g(status)f(ob)r(ject,)h(i.e.,)f(n) o(um)o(b)q(er)75 966 y(of)h(b)o(ytes)g(in)h(the)f(incoming)i(message)f(matc)o (hed)g(b)o(y)f(prob)q(e.)158 1134 y Fk(MPI)p 257 1134 V 17 w(CANCEL\()k(handle,)d(\015ag\))75 1250 y(IN)i(handle)j Fn(handle)14 b(to)g(comm)o(uni)o(cation)d(ob)r(ject)75 1329 y Fk(OUT)16 b(\015ag)k Fn(\(logical\))158 1409 y(A)c(call)f(to)h Fl(MPI)p 405 1409 14 2 v 16 w(CANCEL)e Fn(cancels)j(a)f(p)q(ending)g(comm)o(unication) d(op)q(eration)j(\(send)h(or)f(receiv)o(e\).)26 b(The)17 b(call)75 1459 y(returns)12 b Fl(flag)21 b(=)g(true)9 b Fn(if)h(the)g(cancel)h(op)q (eration)f(succeeded,)j Fl(flag)21 b(=)h(false)9 b Fn(otherwise.)17 b(It)11 b(m)o(ust)e(b)q(e)h(the)h(case)75 1509 y(that)16 b(either)g(the)g (cancel)h(op)q(eration)e(succeeds)j(or)e(that)g(the)g(p)q(ending)g(comm)o (uni)o(cation)d(op)q(eration)i(completes)75 1559 y(\(but)f(not)g(b)q(oth\).) 158 1609 y(If)d(a)g(send)i(w)o(as)e(cancelled)h(successfully)h(then)f(the)g (message)f(sen)o(t)i(will)d(not)h(b)q(e)h(receiv)o(ed,)h(the)f(receiv)o(e)h (bu\013er)75 1659 y(of)h(an)o(y)h(p)q(osted)g(receiv)o(e)i(for)d(that)h (message)f(will)g(not)h(b)q(e)g(altered,)g(and)g(an)o(y)f(suc)o(h)i(receiv)o (e)g(has)f(to)g(b)q(e)g(satis\014ed)75 1708 y(b)o(y)g(another)h(send.)24 b(If)15 b(a)h(receiv)o(e)h(w)o(as)e(cancelled)h(successfully)m(,)g(then)h (the)f(receiv)o(e)h(bu\013er)f(p)q(osted)h(will)d(not)h(b)q(e)75 1758 y(altered,)f(and)g(an)o(y)f(send)i(matc)o(hing)d(this)i(receiv)o(e)h (has)f(to)g(b)q(e)g(satis\014ed)h(b)o(y)f(another)g(receiv)o(e.)158 1808 y(A)j(successful)i(cancel)f(op)q(eration)f(do)q(es)h(not)f(complete)f (the)i(cancelled)g(comm)o(uni)o(cation;)d(the)j(op)q(eration)75 1858 y(still)10 b(needs)i(to)e(b)q(e)h(completed)g(with)f(an)g Fl(MPI)p 773 1858 V 15 w(WAIT)g Fn(or)h Fl(MPI)p 1000 1858 V 15 w(TEST)e Fn(call,)i(if)e(it)i(is)f(an)h(non)o(blo)q(c)o(king)e(comm)o (unicatio)o(n.)53 b Fe(TO)11 b(DISCUSS)158 1986 y Fj(Discussion:)158 2032 y Fi(W)m(e)h(ma)o(y)h(w)o(an)o(t)f(a)g(non)o(blo)q(c)o(king)j(cancel,)f (that)e(returns)h(asap,)g(since,)g(an)o(yho)o(w,)f(one)h(has)g(to)f(w)o(ait)g (for)g(the)h Fd(MPI)p 1786 2032 12 2 v 13 w(TEST)75 2077 y Fi(or)k Fd(MPI)p 188 2077 V 13 w(WAIT)e Fi(to)i(return)g(b)q(efore)g(the)g (bu\013er)h(is)f(reused.)29 b(If)16 b(w)o(e)h(do)g(so,)g(then)h(the)e Fd(MPI)p 1402 2077 V 13 w(CANCEL)f Fi(call)j(will)g(not)f(return)75 2123 y(information)k(on)e(the)g(success)h(or)f(failure)h(of)f(the)g(cancel;)j (this)e(will)g(b)q(e)g(returned)f(b)o(y)h(the)f Fd(MPI)p 1563 2123 V 13 w(WAIT)e Fi(or)i Fd(MPI)p 1786 2123 V 13 w(TEST)75 2169 y Fi(op)q(eration.)158 2214 y(The)c Fd(MPI)p 301 2214 V 13 w(TEST)f Fi(executed)j(after)e(an)h(op)q(eration)h(has)f(b)q(een)g (cancelled)i(needs)e(to)g(return)f(an)h(indication)j(that)d(the)75 2260 y(op)q(eration)j(failed.)30 b(On)17 b(the)g(other)g(hand,)h(w)o(e)f(do)g (not)g(w)o(an)o(t)g(the)g(program)g(to)g(blo)o(w)h(up)f(in)h(suc)o(h)f (situation.)31 b(Th)o(us,)75 2306 y(this)16 b(indication)i(has)e(to)f(b)q(e)h (handled)h(as)e(a)g(nonfatal)i(error,)e(ev)o(en)h(in)g(the)f(execution)i(mo)q (de)f(where)f(an)o(y)h(unsuccessful)75 2351 y(comm)o(unication)g(is)d(fatal.) 18 b(Do)13 b(w)o(e)g(w)o(an)o(t)f(this)i(information)h(to)e(b)q(e)g(returned) h(in)g(the)f(error)g(co)q(de,)h(or)f(to)f(b)q(e)i(returned)g(as)75 2397 y(part)f(of)g(the)g(status?)158 2443 y(Can)g(a)g(thread)h(cancel)g(a)f (blo)q(c)o(king)j(comm)o(unication)f(executed)f(b)o(y)g(another)f(thread?)158 2493 y(Do)g(w)o(e)g(w)o(an)o(t)g Fd(MPI)p 432 2493 V 13 w(CANCEL)d Fi(to)j(free)g(the)g(comm)o(unication)j(ob)r(ject?)158 2704 y Fj(Implemen)o(tation)d(note:)p eop %%Page: 33 34 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(33)158 45 y Fi(The)13 b(cancel)h(op)q(eration)h(ma)o(y)e(b)q (e)g(exp)q(ensiv)o(e)j(to)c(implemen)o(t)j(on)f(some)f(systems.)75 265 y Fm(1.13)70 b(send-receiv)n(e)1931 306 y Fe(TO)11 b(DISCUSS)75 434 y Fj(Discussion:)158 484 y Fi(This)j(section)g(has)f(not)h(y)o(et)f(b)q (een)g(discussed)i(b)o(y)f(MPIF.)158 617 y Fn(The)g Fk(send-receiv)o(e)c Fn(op)q(erations)k(com)o(bines)e(in)h(one)h(call)e(the)i(sending)g(of)e(a)h (message)g(to)h(one)f(destination)75 667 y(and)h(the)g(receiving)h(of)e (another)h(message,)f(form)g(another)h(destination,)f(p)q(ossibly)h(the)g (same.)k(The)c Fk(exc)o(hange)75 716 y Fn(op)q(erations)20 b(are)g(the)h(same)e(as)h(send-receiv)o(e,)j(except)e(that)f(the)h(send)f (bu\013er)h(and)f(the)g(receiv)o(e)i(bu\013er)f(are)75 766 y(iden)o(tical.)j(A)17 b(send-receiv)o(e)h(op)q(eration)e(is)g(v)o(ery)h (useful)f(for)g(executing)h(a)f(shift)g(op)q(eration)g(across)h(a)f(c)o(hain) g(of)75 816 y(pro)q(cesses.)25 b(If)15 b(blo)q(c)o(king)f(sends)j(and)e (receiv)o(es)i(are)e(used)i(for)e(suc)o(h)h(shift,)f(then)g(one)h(needs)h(to) e(order)h(correctly)75 866 y(the)d(sends)g(and)f(receiv)o(es)i(\(e.g.)j(ev)o (en)c(sends,)g(next)g(receiv)o(es,)h(o)q(dd)e(receiv)o(es)i(\014rst,)e(next)h (sends\))h(so)e(as)g(to)g(prev)o(en)o(t)75 916 y(cyclic)j(dep)q(endencies)i (that)e(lead)f(to)h(deadlo)q(c)o(k.)20 b(When)15 b(a)f(send-receiv)o(e)j(ro)e (exc)o(hange)g(op)q(eration)g(is)f(used,)h(the)75 965 y(comm)o(unication)10 b(subsystem)15 b(tak)o(es)f(care)h(of)e(these)i(issues.)158 1015 y(A)j(handle)f(for)h(a)f(send-receiv)o(e)j(\(exc)o(hange\))e(op)q (eration)g(is)f(created)j(b)o(y)d(a)g(call)g(to)h Fl(MPI)p 1586 1015 14 2 v 15 w(INIT)p 1689 1015 V 15 w(SENDRECV)75 1065 y Fn(\()p Fl(MPI)p 160 1065 V 15 w(INIT)p 263 1065 V 15 w(EXCHANGE)p Fn(\).)f(The)j(op)q(erations)f(can)g(b)q(e)h(started)g(and)f(completed)f(lik) o(e)g(regular)h(comm)o(unication)75 1115 y(op)q(erations,)14 b(using)f(the)i Fl(MPI)p 538 1115 V 15 w(START)p Fn(,)d Fl(MPI)p 753 1115 V 15 w(WAIT)p Fn(,)h Fl(MPI)p 947 1115 V 15 w(WAITALL)p Fn(,)f(etc.)19 b(In)13 b(addition,)g(op)q(erations)h(are)g(pro)o(vided)75 1165 y(for)k(blo)q(c)o(king)f(and)g(non)o(blo)q(c)o(king)g(send-receiv)o(e)j (and)e(exc)o(hange)h(op)q(erations,)g(with)e(general)h(and)g(con)o(tiguous)75 1215 y(bu\013ers.)158 1264 y(A)f(message)f(sen)o(t)h(b)o(y)g(a)f(send-receiv) o(e)j(or)d(exc)o(hange)i(op)q(eration)e(can)h(b)q(e)g(receiv)o(ed)h(b)o(y)f (a)f(regular)h(receiv)o(e)75 1314 y(op)q(eration,)c(and)h(vice)g(v)o(ersa.) 158 1443 y Fj(Discussion:)158 1488 y Fi(The)h(correctness)g(c)o(hapter)h (need)f(a)g(section)h(on)f(send-receiv)o(es)i({)d(the)h(correctness)h(issue)g (is)f(not)g(en)o(tirely)i(trivial,)75 1534 y(b)q(ecause)e(of)e(p)q(ossible)j (cyclic)g(dep)q(endencies.)21 b(Ev)o(en)15 b(in)f(an)g(implemen)o(tation)j (where)d(messages)g(are)g(not)g(bu\013ered,)h(MPi)75 1580 y(has)j(to)g (guaran)o(tee)h(that)f(blo)q(c)o(king)i(calls)g(to)d(send-receiv)o(e)j(that)e (collectiv)o(ely)j(execute)d(a)g(shift)h(on)f(a)f(cycle)i(will)h(not)75 1625 y(deadlo)q(c)o(k.)f(Some)13 b(system)h(bu\013ering)h(is,)e(in)h(fact,)e (necessary)i(when)g(exc)o(hanges)g(are)f(used.)158 1793 y Fk(MPI)p 257 1793 15 2 v 17 w(INIT)p 380 1793 V 18 w(SENDRECV\(handle,)d(send)p 944 1793 V 16 w(bu\013er)p 1084 1793 V 16 w(handle,)g(dest,)g(recv)p 1461 1793 V 17 w(bu\013er)p 1602 1793 V 16 w(handle,)g(source,)75 1843 y(tag,)16 b(con)o(text,)e(p)q(ersistence\))158 1928 y Fn(Creates)h(a)f(comm)o(unicatio)o(n)d(ob)r(ject)k(for)f(a)f(send-receiv)o(e) j(op)q(eration.)75 2019 y Fk(OUT)g(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g (ob)r(ject)h(b)q(efore)g(the)179 2069 y(call.)75 2152 y Fk(IN)e(send)p 244 2152 V 16 w(bu\013er)p 384 2152 V 16 w(handle)j Fn(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 2235 y Fk(IN)h(dest)k Fn(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o(teger\))75 2318 y Fk(IN)i(recv)p 237 2318 V 18 w(bu\013er)p 379 2318 V 15 w(handle)j Fn(handle)14 b(to)f(receiv)o(e)i(bu\013er)g(descriptor)75 2401 y Fk(IN)h(source)k Fn(rank)14 b(of)f(source)i(in)f(con)o(text)g(\(in)o (teger\))75 2484 y Fk(IN)i(tag)21 b Fn(tag)13 b(used)i(b)q(oth)f(for)g(send)g (and)g(for)g(receiv)o(e)h(op)q(eration)f(\(in)o(teger\))75 2567 y Fk(IN)i(con)o(text)k Fn(con)o(text)14 b(b)q(oth)g(for)g(send)h(and)e (receiv)o(e)i(op)q(eration)f(\(handle\))75 2650 y Fk(IN)i(p)q(ersistence)i Fn(comm)o(unication)11 b(ob)r(ject)k(p)q(ersistence)p eop %%Page: 34 35 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(34)158 45 y Fk(MPI)p 257 45 15 2 v 17 w(INIT)p 380 45 V 18 w(EX)o(CHANGE\(handle,)18 b(bu\013er)p 991 45 V 16 w(handle,)e(dest,)i(source,)g(tag,)g(con)o(text,)g(p)q(ersis-)75 95 y(tence\))158 180 y Fn(Creates)d(a)f(comm)o(unicatio)o(n)d(ob)r(ject)k (for)f(an)f(exc)o(hange)i(op)q(eration.)75 257 y Fk(OUT)h(handle)i Fn(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q (ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the)179 307 y(call.)75 384 y Fk(IN)e(bu\013er)p 273 384 V 16 w(handle)j Fn(handle)13 b(to)h(send)h(and)f(receiv)o(e)h(bu\013er)g(descriptor)75 461 y Fk(IN)h(dest)k Fn(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o (teger\))75 539 y Fk(IN)i(source)k Fn(rank)14 b(of)f(source)i(in)f(con)o (text)g(\(in)o(teger\))75 616 y Fk(IN)i(tag)21 b Fn(tag)13 b(used)i(b)q(oth)f(for)g(send)g(and)g(for)g(receiv)o(e)h(op)q(eration)f(\(in) o(teger\))75 693 y Fk(IN)i(con)o(text)k Fn(con)o(text)14 b(b)q(oth)g(for)g (send)h(and)e(receiv)o(e)i(op)q(eration)f(\(handle\))75 770 y Fk(IN)i(p)q(ersistence)i Fn(comm)o(unication)11 b(ob)r(ject)k(p)q (ersistence)158 926 y Fj(Discussion:)158 972 y Fi(Do)e(w)o(e)g(also)h(w)o(an) o(t)f(send-receiv)o(e)h(and)g(exc)o(hange)g(op)q(erations)h(with)f(the)f Fd(ready)e Fi(and)i Fd(synchronous)c Fi(mo)q(de?)158 1017 y(Do)14 b(w)o(e)e(w)o(an)o(t)h Fd(tag)p Fi(?)k(\(Collectiv)o(e)e(comm)o(unication)h (don't)e(use)f(tag;)h(but)f(tag)h(is)f(needed)i(if)e(w)o(e)g(w)o(an)o(t)g(to) g(b)q(e)h(able)g(to)75 1063 y(mix)g(send-receiv)o(e)g(and)g(exc)o(hange)g (with)g(regular)g(sends)g(and)g(receiv)o(es.\))158 1109 y(Do)f(w)o(e)f(w)o (an)o(t)h(to)f(allo)o(w)i(wildcard)g(source)f(and/or)h(tag?)j(Since)d(I)f (assumed)g(that)g(the)g(message)g(sen)o(t)g(has)g(the)g(same)75 1154 y(tag)g(as)g(the)g(message)h(receiv)o(ed,)g(then)g(widcard)g(tag)f(do)q (es)g(not)h(mak)o(e)f(m)o(uc)o(h)g(sense.)158 1322 y Fk(MPI)p 257 1322 V 17 w(SENDRECV\(handle,)d(send)p 820 1322 V 16 w(bu\013er)p 960 1322 V 16 w(handle,)g(dest,)g(recv)p 1337 1322 V 17 w(bu\013er)p 1478 1322 V 16 w(handle,)g(source,)h(tag,)75 1372 y(con)o(text\))158 1457 y Fn(Blo)q(c)o(king)i(send-receiv)o(e)j(op)q(eration.)h(The)e(op)q (eration)e(returns)i(when)f(b)q(oth)g(send)h(and)e(receiv)o(e)i(ha)o(v)o(e)e (com-)75 1507 y(pleted.)158 1557 y Fl(MPI)p 227 1557 14 2 v 15 w(SENDRECV\(handle,)19 b(send)p 701 1557 V 14 w(buffer)p 847 1557 V 15 w(handle,)h(dest,)h(recv)p 1255 1557 V 15 w(buffer)p 1402 1557 V 14 w(handle,)g(source,)f(tag,)h(context,)75 1607 y(status\))12 b Fn(is)75 1684 y Fl(MPI_INIT_SENDRECV)o(\(hand)o(le,)19 b(send_buffer_hand)o(le,)f(dest,)140 1733 y(recv_buffer_handle,)g(source,)j (tag,)g(context,)f(MPI_EPHEMERAL\))75 1783 y(MPI_START\(handle\))75 1833 y(MPI_WAIT\(handle,)e(status\))158 1945 y Fk(MPI)p 257 1945 15 2 v 17 w(EX)o(CHANGE\(handle,)d(bu\013er)p 864 1945 V 16 w(handle,)f(dest,)h(source,)g(tag,)h(con)o(text,)e(status\))158 2031 y Fn(Blo)q(c)o(king)j(exc)o(hange)h(op)q(eration.)29 b(The)18 b(op)q(eration)f(returns)i(when)f(b)q(oth)g(send)g(and)f(receiv)o(e)i(ha)o(v) o(e)e(com-)75 2080 y(pleted.)158 2130 y Fl(MPI)p 227 2130 14 2 v 15 w(EXCHANGE\(handle,)i(buffer)p 745 2130 V 14 w(handle,)h(dest,)h (source,)f(tag,)h(context,)g(status\))12 b Fn(is)75 2207 y Fl(MPI_INIT_EXCHANGE)o(\(hand)o(le,)19 b(buffer_handle,)g(dest,)140 2257 y(source,)i(tag,)g(context,)f(MPI_EPHEMERAL\))75 2307 y(MPI_START\(handle\))75 2357 y(MPI_WAIT\(handle,)e(status\))158 2469 y Fk(MPI)p 257 2469 15 2 v 17 w(ISENDRECV\(handle,)25 b(send)p 853 2469 V 16 w(bu\013er)p 993 2469 V 15 w(handle,)g(dest,)g(recv)p 1399 2469 V 17 w(bu\013er)p 1540 2469 V 15 w(handle,)g(source,)75 2519 y(tag,)16 b(con)o(text\))158 2604 y Fn(Non)o(blo)q(c)o(king)d (send-receiv)o(e)j(op)q(eration.)158 2654 y Fl(MPI)p 227 2654 14 2 v 15 w(ISENDRECV\(handle,)i(send)p 722 2654 V 15 w(buffer)p 869 2654 V 15 w(handle,)i(dest,)h(recv)p 1277 2654 V 15 w(buffer)p 1424 2654 V 14 w(handle,)f(source,)h(tag,)g(context\))75 2704 y Fn(is)p eop %%Page: 35 36 bop 75 -100 a Ff(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)717 b Fn(35)75 45 y Fl(MPI_INIT_SENDRECV)o(\(hand)o(le,)19 b(send_buffer_hand)o(le,)f(dest,)140 95 y(recv_buffer_handle,)g(source,)j (tag,)g(context,)f(MPI_EPHEMERAL\))75 145 y(MPI_START\(handle\))158 271 y Fk(MPI)p 257 271 15 2 v 17 w(IEX)o(CHANGE\(handle,)15 b(bu\013er)p 882 271 V 16 w(handle,)f(dest,)h(source,)g(tag,)h(con)o(text\)) 158 357 y Fn(Non)o(blo)q(c)o(king)d(exc)o(hange)h(op)q(eration.)158 407 y Fl(MPI)p 227 407 14 2 v 15 w(IEXCHANGE\(handle,)k(buffer)p 766 407 V 15 w(handle,)i(dest,)h(source,)f(tag,)h(context\))12 b Fn(is)75 498 y Fl(MPI_INIT_EXCHANGE)o(\(hand)o(le,)19 b(buffer_handle,)g (dest,)140 548 y(source,)i(tag,)g(context,)f(MPI_EPHEMERAL\))75 597 y(MPI_START\(handle\))158 724 y Fk(MPI)p 257 724 15 2 v 17 w(SENDRECV)o(C\(handle,)10 b(send)p 853 724 V 16 w(start,)h(send)p 1088 724 V 16 w(coun)o(t,)f(send)p 1339 724 V 16 w(datat)o(yp)q(e,)h(dest,)g (recv)p 1763 724 V 17 w(start,)75 774 y(recv)p 166 774 V 17 w(coun)o(t,)k(recv)p 416 774 V 17 w(datat)o(yp)q(e,)g(source,)g(tag,)h(con)o (text,)e(status\))158 859 y Fn(Blo)q(c)o(king)f(send-receiv)o(e)j(with)e(con) o(tiguous)f(bu\013ers.)158 909 y Fl(MPI)p 227 909 14 2 v 15 w(SENDRECVC\(handle,)18 b(send)p 722 909 V 15 w(start,)j(send)p 978 909 V 15 w(count,)f(send)p 1233 909 V 15 w(datatype,)g(dest,)h(recv)p 1685 909 V 15 w(start,)f(recv)p 1940 909 V 15 w(count,)75 959 y(recv)p 166 959 V 15 w(datatype,)g(source,)g(tag,)h(context,)f(status\))13 b Fn(is)75 1050 y Fl(MPI_CREATE_BUFFER)o(\(send)o(_buff)o(er_ha)o(ndle,)18 b(MPI_EPHEMERAL\))75 1100 y(MPI_CREATE_BUFFER)o(\(recv)o(_buff)o(er_ha)o (ndle,)g(MPI_EPHEMERAL\))75 1150 y(MPI_APPEND_CONTIG)o(UOUS\()o(send_)o (buffe)o(r_han)o(dle,)g(send_start,)i(send_count,)184 1200 y(send_datatype\))75 1250 y(MPI_APPEND_CONTIG)o(UOUS\()o(recv_)o(buffe)o (r_han)o(dle,)e(recv_start,)i(recv_count,)184 1299 y(recv_datatype\))75 1349 y(MPI_COMMIT\(send_b)o(uffer)o(_hand)o(le\))75 1399 y (MPI_COMMIT\(recv_b)o(uffer)o(_hand)o(le\))75 1449 y(MPI_INIT_SENDRECV)o (\(hand)o(le,)f(send_buffer_hand)o(le,)f(dest,)140 1499 y (recv_buffer_handle,)g(source,)j(tag,)g(context,)f(MPI_EPHEMERAL\))75 1548 y(MPI_START\(handle\))75 1598 y(MPI_WAIT\(handle,)e(status\))158 1725 y Fk(MPI)p 257 1725 15 2 v 17 w(EX)o(CHANGEC\(handle,)e(start,)f(coun)o (t,)f(datat)o(yp)q(e,)h(dest,)g(source,)g(tag,)h(con)o(text\))158 1810 y Fn(Blo)q(c)o(king)d(exc)o(hange)i(with)e(con)o(tiguous)h(bu\013er)158 1860 y Fl(MPI)p 227 1860 14 2 v 15 w(EXCHANGEC\(handle,)k(start,)j(count,)f (datatype,)g(dest,)h(source,)g(tag,)f(context\))13 b Fn(is)75 1951 y Fl(MPI_CREATE_BUFFER)o(\(buff)o(er_ha)o(ndle,)18 b(MPI_EPHEMERAL\))75 2001 y(MPI_APPEND_CONTIG)o(UOUS\()o(buffe)o(r_han)o(dle,)g(start,)j(count,)f (datatype\))75 2051 y(MPI_COMMIT\(buffer)o(_hand)o(le\))75 2101 y(MPI_INIT_EXCHANGE)o(\(hand)o(le,)f(buffer_handle,)g(dest,)140 2151 y(source,)i(tag,)g(context,)f(MPI_EPHEMERAL\))75 2200 y(MPI_START\(handle\))75 2250 y(MPI_WAIT\(handle,)e(status\))158 2377 y Fk(MPI)p 257 2377 15 2 v 17 w(ISENDRECV)o(C\(handle,)11 b(send)p 872 2377 V 15 w(start,)g(send)p 1106 2377 V 16 w(coun)o(t,)f(send)p 1357 2377 V 16 w(datat)o(yp)q(e,)h(dest,)g(recv)p 1781 2377 V 17 w(start,)75 2427 y(recv)p 166 2427 V 17 w(coun)o(t,)k(recv)p 416 2427 V 17 w(datat)o(yp)q(e,)g(source,)g(tag,)h(con)o(text\))158 2512 y Fn(Non)o(blo)q(c)o(king)d(send-receiv)o(e)j(with)d(con)o(tiguous)h (bu\013ers.)158 2562 y Fl(MPI)p 227 2562 14 2 v 15 w(ISENDRECVC\(handle,)k (send)p 744 2562 V 15 w(start,)j(send)p 1000 2562 V 14 w(count,)g(send)p 1255 2562 V 15 w(datatype,)f(dest,)h(recv)p 1707 2562 V 14 w(start,)g(recv)p 1962 2562 V 15 w(count,)75 2612 y(recv)p 166 2612 V 15 w(datatype,)f(source,)g(tag,)h(context\))12 b Fn(is)p eop %%Page: 36 37 bop 75 -100 a Ff(CHAPTER)14 b(1.)32 b(POINT)14 b(TO)g(POINT)h(COMMUNICA)m (TION)712 b Fn(36)75 45 y Fl(MPI_CREATE_BUFFER)o(\(send)o(_buff)o(er_ha)o (ndle,)18 b(MPI_EPHEMERAL\))75 95 y(MPI_CREATE_BUFFER)o(\(recv)o(_buff)o (er_ha)o(ndle,)g(MPI_EPHEMERAL\))75 145 y(MPI_APPEND_CONTIG)o(UOUS\()o(send_) o(buffe)o(r_han)o(dle,)g(send_start,)i(send_count,)184 195 y(send_datatype\))75 244 y(MPI_APPEND_CONTIG)o(UOUS\()o(recv_)o(buffe)o (r_han)o(dle,)e(recv_start,)i(recv_count,)184 294 y(recv_datatype\))75 344 y(MPI_COMMIT\(send_b)o(uffer)o(_hand)o(le\))75 394 y(MPI_COMMIT\(recv_b)o (uffer)o(_hand)o(le\))75 444 y(MPI_INIT_SENDRECV)o(\(hand)o(le,)f (send_buffer_hand)o(le,)f(dest,)140 493 y(recv_buffer_handle,)g(source,)j (tag,)g(context,)f(MPI_EPHEMERAL\))75 543 y(MPI_START\(handle\))158 670 y Fk(MPI)p 257 670 15 2 v 17 w(IEX)o(CHANGEC\(handle,)c(start,)f(coun)o (t,)f(datat)o(yp)q(e,)h(dest,)g(source,)g(tag,)h(con)o(text\))158 755 y Fn(Non)o(blo)q(c)o(king)d(exc)o(hange)h(with)g(con)o(tiguous)f (bu\013er)158 805 y Fl(MPI)p 227 805 14 2 v 15 w(IEXCHANGEC\(handle,)18 b(start,)j(count,)f(datatype,)g(dest,)h(source,)f(tag,)h(context\))12 b Fn(is)75 896 y Fl(MPI_CREATE_BUFFER)o(\(buff)o(er_ha)o(ndle,)18 b(MPI_EPHEMERAL\))75 946 y(MPI_APPEND_CONTIG)o(UOUS\()o(buffe)o(r_han)o(dle,) g(start,)j(count,)f(datatype\))75 996 y(MPI_INIT_EXCHANGE)o(\(hand)o(le,)f (buffer_handle,)g(dest,)140 1046 y(source,)i(tag,)g(context,)f (MPI_EPHEMERAL\))75 1096 y(MPI_START\(handle\))158 1265 y Fj(Discussion:)158 1311 y Fi(Can)12 b(w)o(e)g(cancel)h(a)f(send-receiv)o(e)i(or)e(exc)o(hange?) 18 b(If)11 b(y)o(es,)i(should)g(it)g(b)q(e)f(the)g(case)h(that)f(either)h(w)o (e)e(cancel)i(b)q(oth)g(send)75 1357 y(and)g(receiv)o(e)g(or)f(neither?)18 b(This)13 b(seems)f(natural,)i(but)e(ma)o(y)g(cause)h(problems)g(\(The)f (receiv)o(e)h(has)g(already)h(executed,)e(but)75 1402 y(the)h(send)h(is)f(h)o (ung.)18 b(Ho)o(w)13 b(do)g(w)o(e)g(cancel?\))158 1448 y(This)h(is)f(the)g (pream)o(ble)i(to)e(a)g(m)o(uc)o(h)g(more)g(general)i(question:)j(can)c(w)o (e)e(cancel)j(collectiv)o(e)g(op)q(erations?)75 1668 y Fm(1.14)70 b(Null)21 b(pro)r(cesses)75 1842 y Fj(Discussion:)35 b Fi(This)14 b(section)g(has)f(not)h(b)q(een)f(review)o(ed)h(b)o(y)g(MPIF.)158 1974 y Fn(In)i(man)o(y)d(instances,)k(it)e(is)h(con)o(v)o(enien)o(t)g(to)f (sp)q(ecify)i(a)e(\\dumm)o(y")d(source)17 b(or)f(destination)f(for)g(comm)o (uni-)75 2024 y(cation.)27 b(This)17 b(simpli\014es)e(the)j(co)q(de)f(that)g (is)g(needed)i(for)d(dealing)g(with)h(b)q(oundaries,)h(e.g.,)e(in)g(the)i (case)g(of)e(a)75 2074 y(noncircular)e(shift)g(done)g(with)f(calls)h(to)g (send-receiv)o(e.)158 2124 y(The)d(sp)q(ecial)g(v)n(alue)f Fl(MPI)p 545 2124 V 15 w(PROCNULL)f Fn(can)i(b)q(e)g(used)h(instead)f(of)f(a) g(rank)h(wherev)o(er)h(a)e(source)i(or)f(a)f(destination)75 2173 y(parameter)k(is)g(required)h(in)f(a)g(call.)k(A)c(comm)o(unication)d (with)j(pro)q(cess)i Fl(MPI)p 1300 2173 V 15 w(PROCNULL)c Fn(has)j(no)f (e\013ect:)20 b(a)14 b(send)75 2223 y(to)k Fl(MPI)p 199 2223 V 15 w(PROCNULL)e Fn(succeeds)k(and)d(returns)j(asap.)29 b(A)18 b(receiv)o(e)h(from)d Fl(MPI)p 1287 2223 V 15 w(PROCNULL)g Fn(succeeds)k(and)d(returns)75 2273 y(asap)d(with)f(no)h(mo)q(di\014cations)e (to)i(the)g(receiv)o(e)h(bu\013er.)158 2401 y Fj(Discussion:)158 2447 y Fi(Need)h(to)g(decide)h(what)f(is)h(returned)g(in)g(the)f(status)h(ob) r(ject:)23 b(I)16 b(suggest)g Fd(source)i(=)h(MPI)p 1505 2447 12 2 v 13 w(PROCNULL)p Fi(,)12 b Fd(tag=0)i Fi(and)75 2493 y Fd(count=0)p Fi(.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Tue Jun 22 10:40:18 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA25378; Tue, 22 Jun 93 10:40:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11253; Tue, 22 Jun 93 10:39:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 22 Jun 1993 10:39:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11245; Tue, 22 Jun 93 10:39:07 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA10513; Tue, 22 Jun 93 14:39:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05585; Tue, 22 Jun 93 08:38:05 MDT Date: Tue, 22 Jun 93 08:38:05 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9306221438.AA05585@macaw.fsl.noaa.gov> To: SNIR@watson.ibm.com Subject: Re: latest pt2pt chapter draft... Cc: mpi-pt2pt@cs.utk.edu Marc, A wee nit-pick... :-) In the June 20 and May 28 pt2pt drafts, Section 1.5, first paragraph, a new sentence has been added since we voted on the second reading: "The syntax is designed to be extendable so that new types of buffers can (be) added by vendors." Do we really want to say this? I'd rather not. It seems to go against the whole idea of "standardization". IMHO, something like "The syntax is designed to be extendable so that new types of buffers can be added in future versions of MPI." would be better. Tom From owner-mpi-pt2pt@CS.UTK.EDU Mon Jun 28 10:54:14 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA11688; Mon, 28 Jun 93 10:54:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17041; Mon, 28 Jun 93 10:52:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 28 Jun 1993 10:52:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17030; Mon, 28 Jun 93 10:52:55 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA15953; Mon, 28 Jun 93 07:53:45 PDT Message-Id: <9306281453.AA15953@ocfmail.ocf.llnl.gov> Date: Mon, 28 Jun 1993 07:55:41 -0800 To: mpi-pt2pt@CS.UTK.EDU From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: counting the number of elements received During the pt2pt subcommittee meeting on Thursday evening, we discussed what the appropriate units were for the counts returned as the result of receive operations. There were two proposals remaining at the end of the session : 1) return the number of elements received according to the "flattened" type (i.e., the number of "basic type" elements received), and 2) return the number of elements received at the next level below the "top" level specified by a buffer descriptor or type. The latter proposal treats arrays differently than struts, since a buffer descriptor or type consisting of a list of basic types, structs and arrays used in a receive returns the number of basic types and structs received plus the number of elements received in the arrays. Furthermore, if a receive terminates without receiving enough data to complete the type or buffer descriptor, it is an error if the struct is only partially filled in, while it is not an error if the incomplete data received ends within one of the arrays specified in the type or buffer descriptor. After the meeting on Friday, some of us were having lunch and discussing this issue. Since my position is that we should use proposal #1, I was defending it. In the process I tried to come up with an example for which proposal #2 is not the suitable semantics. I couldn't do so on the spur of the moment and asked if I might retire from the discussion until I could give the matter some thought. The other participants graceously consented. On the flight home from the meeting, I worked out the details of such an example. While it is fictional, I believe it is plausable. I also believe given enough time and motivation I could come up with other examples that illustrate the problems with proposal #2. However, allow me to stick to this example for the moment. Consider someone designing a deep space vehicle that will be put in orbit around Jupiter and send probes into its atmosphere. The vehicle will collect data about the planet's atmosphere and send it back to earth. The designer wishes to use MPI to send the data messages. The characteristics of this engineering problem are as follows. The power budget for a probe is such that it is not possible for it to send back messages containing telemetric data periodically. Instead, it monitors both the temperature and pressure of its environment and when either has changed sufficiently, takes a sample of, say, the atmosphere's chemistry, returning a message with both the chemistry/press/temp data and its position. To make this concrete, suppose the chemistry data is represented by an array of 10 reals, press/temp by a double each and position by a struct. Since the chemistry sampling takes a finite time to carry out, the pressure and temperature will most likely change and so the probe sends back the pressure and temperature at the beginning of the sample, the temperature and pressure at the end of the sample along with the position and chemistry data. In order to reduce the amount of data the probe sends, the temperature, pressure and position data in most messages are not their absolute values. Rather, they are the difference between the current values and those represented in the previous message. These deltas are represented as reals rather than as doubles. However, every 100th message contains the more precise values for pressure, temperature and position. Consequently, the messages sent back have the following form (as represented in C) : Delta messages -------------- struct delta_data { struct position { real range; real phi; real theta; } position; real start_temp; real start_press; real end_temp; real end_press; real chemistry_data[10]; } delta_data; Full messages ------------- struct full_data { struct position { double range; double phi; double theta; } position; double start_temp; double start_press; double end_temp; double end_press; real chemistry_data[10]; } full_data; The space vehicle receives this data from the probe, archives it and at the end of the probe's descent, relays the archived message stream to earth. The probe descent takes approximately 4 hours, during which time much more data is generated than can fit into the space vehicle's RAM. Consequently, the space vehicle's computer system stores the data on an onboard tape system for later replay. The space vehicle is also running several other experiments and needs to archive this data as well. Therefore, the vehicle's platform computer periodically dumps its complete RAM to tape. This dump occurs after a suitable interval of time has elapsed and when the last of the delta messages in a 100 message sequence has been received. The amount of memory allocated to the probe message data is approximately 80% of the RAM available. The tape subsystem is architected with a cache memory equal in size to the RAM memory. This is also the size of a tape block. A tape copy begins by a fast RAM to cache transfer, followed by a much slower cache to tape copy. Since the probe's descent into the Jovian atmosphere will eventually destroy it, its data stream can end unpredictably. Furthermore, the "tail" data may be the most interesting and therefore cannot be simply dropped. When the probe data ends, the space vehicle rewinds its tape and reads it a block at a time (i.e., a RAM dump at a time), sending the message stream to earth. The MPI send/receive buffer descriptor corresponds to the following C data structure (assume the probe data is at the end of the RAM memory): struct memory_dump { struct experiment1_data { . . . } experiment1_data; struct experiment2_data { . . . } experiment2_data; struct experiment3_data { . . . } experiment3_data; struct experiment4_data { . . . } experiment4_data; struct experiment5_data { . . . } experiment5_data; int number_of_chunks; struct probe_message_chunk { int number_of_intermediates; struct full_data calibration; struct delta_data intermediates[99]; } probe_message_chunk[1000]; } memory_dump; Here I am assuming that 1000 "minor cycles" of probe_message_chunk is sufficient to hold all the probe data before a RAM write to tape occurs. If MPI receives behave according to proposal #2, the last RAM dump on the tape would cause an MPI receive error, since the last component in memory_dump (i.e., probe_message_chunk) will in general not be received completely. Let me anticipate some questions about this example and attempt to respond to them: QUESTION: Why not stride out the calibration data through an append_vec call and similarly stride out the intermediates data? ANSWER: Of course, that is possible. However, we were trying to come up with a way to send structures without using the "stride trick." QUESTION: Why not promote probe_message_chunk to the "top level in memory_dump? ANSWER: This would require 1000 copies of probe_message_chunk (called, I suppose, something like probe_message_chunk1, probe_message_chunk2, etc.) As a programmer, I would strongly object to being forced to do something like this. I realize that this is a somewhat artificial example. Given more time I think I could come up with one that is focused on scientific computing. However, I believe the principal is clear. At the very least, the programmer should have a way of turning off (e.g., through an environmental variable) the property that counts are based on a "level 2" view of the data and that receives will fail when only a partial struct is received. Regards, Dan P.S. I am about to go incommunicado for about a month, so I'll have to pick up the discussion of this issue when I get back. I wanted to get this out before I left on business travel and some vacation. From owner-mpi-pt2pt@CS.UTK.EDU Wed Jun 30 12:20:56 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA06559; Wed, 30 Jun 93 12:20:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09545; Wed, 30 Jun 93 12:18:50 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 30 Jun 1993 12:18:49 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from uchdcc.dcc.uchile.cl by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09528; Wed, 30 Jun 93 12:18:18 -0400 Received: by uchdcc.dcc.uchile.cl (/\==/\ Smail3.1.25.1 #25.3) id ; Wed, 30 Jun 93 12:17 CDT Received: by malloco.ing.puc.cl (4.1/5.17) id AA04815; Wed, 30 Jun 93 12:17:09 CST From: ricardo@ing.puc.cl (Ricardo Schmutzer von Oldershausen) Message-Id: <9306301617.AA04815@malloco.ing.puc.cl> Subject: Buffer descriptor generalization To: mpi-pt2pt@cs.utk.edu Date: Wed, 30 Jun 93 12:17:09 AST X-Mailer: ELM [version 2.3 PL11] Hello ! I studied James Cownie's extension for communication buffers (described in the draft of June 20) this weekend and even tried to write some code (actually a skeleton) of how you could implement buffers with the extension. Doing this I came along a "little detail" (or I may be missing something... ): Which is the main difference between bound and unbound buffers ? Bound buffers describe a variable in memory. Therefore the start argument is used as a pointer to a memory location. Unbound buffers describe a type (usually a structure or array; I agree with the restriction that a unbound buffer should only describe a composite variable). As we describe a type, no memory location should be involved. Therefore unbound buffers need to save the offsets of each element of the structure to the beginning of the structure (or to the first element of the structure, if you want). I will call this the "base address". As the MPI_APPEND_* functions receive a pointer (the start argument), the offset must be constructed using a KNOWN base address: offset = start - base The base address is the address of the first element of the structure. This address should be known as it is supplied in the first MPI_APPEND_* function (well, if it isn't the first, you can find it because it is the lowest start address supplied). But, and here I come to my "little problem", what happens if we do NOT want to transmit the first element of the structure ? Example: Let's suppose your program uses the same structure as described in the 3. example of the draft: struct foostruct { int i; double d[5]; char b; }; struct foostruct foo[100]; but you want to transmit only the array of doubles and the character: MPI_BUFFER t; MPI_BUFFER fb; mpi_create_buffer(&t, MPI_EPHEMERAL); mpi_create_buffer(&fb, MPI_PERSISTENT); /* describe the data type in t using only d and b */ mpi_append_contiguous(t, &((struct foostruct *)0)->d, 1, MPI_DOUBLE); mpi_append_contiguous(t, &((struct foostruct *)0)->c, 1, MPI_BYTE); /* describe foo itself */ mpi_append_contiguous(fb, &foo, 100, t); mpi_commit(fb); Did somebody tell the buffer t that the pointers are based on NULL pointers, i.e. that the base address is NULL ? Acording to the above description the base address is taken to be the lowest start address supplied, here it is the address of the array d. So, the structure actually described is: struct foostruct { double d[5]; char b; }; Note that the above problem arises independently if we describe struct foostruct with NULL pointers or with pointers to a variable of the same type, and independently of which alternative of the proposed extension we use (distinguishing syntactically between bound and unbound buffers or not). I do not know if there is some simple solution to solve the above problem, but it seems to me that there are two: 1. Restrict that unbound buffers may be described only with NULL pointers (in this case the base address is always NULL, and the above example would work). 2. Demand that the start argument of the first description of unbound buffers MUST be the base address. The above example would have to be rewritten as: /* describe the data type suppling first the base address */ /* Note that count is 0 in the first MPI_APPEND_* */ mpi_append_contiguous(t, &((struct foostruct *)0)->i, 0, MPI_INT); mpi_append_contiguous(t, &((struct foostruct *)0)->d, 1, MPI_DOUBLE); mpi_append_contiguous(t, &((struct foostruct *)0)->c, 1, MPI_BYTE); I do not like the first solution, because not everybody will be happy having to use NULL pointers (how would you do this in FORTRAN ?!), so the second solution seems to be better. =========================================================================== Comments about the first second alternative described in the draft: As bound buffers describe variables and unbound buffers types, we should distinguish between bound and unbound buffers (who knows if later on, somebody finds another restriction of unbound buffers), therefore I prefer the second alternative. But as it is a little bit difficult to create a unbound buffer with the MPI_UNBIND function (create a first buffer, describe it, create a second buffer, call MPI_UNBIND and finally free the first buffer), I would suggest to include a new argument to the MPI_CREATE_BUFFER function that can take two values: MPI_BOUND and MPI_UNBOUND indicating if we want to describe a bound or unbound buffer: MPI_CREATE_BUFFER(buffer, type, persistence) OUT buffer: buffer handle IN type: state type of buffer MPI_BOUND or MPI_UNBOUND IN persistence: buffer persistence The start pointers supplied to the MPI_APPEND_* functions describing the unbound buffer should be restricted to be: a) all based on NULL pointers of the same type or b) all based on a pointer to a single composite variable The first MPI_APPEND_* function must be given a pointer to: a) the first element of the composite type or b) the first element of the composite variable independent if it should be transmited or not (in the later case a count of 0 should be supplied to the MPI_APPEND_CONTIGUOUS function). The unbound buffer should be commited (the saved start pointers could be replaced by the corresponding offsets taking the first start pointer as the base address). This also keeps all MPI_APPEND_* functions describing the unbound buffer easily identifiable to verify the above restrictions. MPI_APPEND_* functions should receive only commited unbound buffers as datatypes. If the unbound buffer is declared ephemeral, it should be freed after it's first use in a MPI_APPEND_* function. Regards, Ricardo -- +----------------------+---------------------------------+--------------+ | Ricardo A. Schmutzer | Internet: ricardo@ing.puc.cl | Conectividad | | PUC, Santiago, Chile | | Distr. Comp. | +----------------------+---------------------------------+--------------+ From owner-mpi-pt2pt@CS.UTK.EDU Mon Jul 5 12:27:15 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01563; Mon, 5 Jul 93 12:27:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29490; Mon, 5 Jul 93 12:27:24 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 5 Jul 1993 12:27:23 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from uchdcc.dcc.uchile.cl by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29479; Mon, 5 Jul 93 12:26:36 -0400 Received: by uchdcc.dcc.uchile.cl (/\==/\ Smail3.1.25.1 #25.3) id ; Mon, 5 Jul 93 12:24 CDT Received: by malloco.ing.puc.cl (4.1/5.17) id AA04611; Mon, 5 Jul 93 12:25:10 CST From: ricardo@ing.puc.cl (Ricardo Schmutzer von Oldershausen) Message-Id: <9307051625.AA04611@malloco.ing.puc.cl> Subject: Re: Buffer descriptor generalization To: mpi-pt2pt@cs.utk.edu Date: Mon, 5 Jul 93 12:25:09 AST X-Mailer: ELM [version 2.3 PL11] Hello ... Please, do NOT read the first part of my previous post ! I feel rather silly, because it is nonsense, and the answer to my (nonexistent) problem is rather simple as the structure I described is not contiguous: MPI_BUFFER t, tt, fb; mpi_create_buffer(&t, MPI_EPHEMERAL); mpi_create_buffer(&fb, MPI_PERSISTENT); mpi_append_contiguous(t, &foo[0].d, 5, MPI_DOUBLE); mpi_append_contiguous(t, &foo[0].b, 1, MPI_BYTE); /* don't we need to mpi_create first the tt buffer ? */ /* the example 3 of the draft doesn't create it ... */ mpi_unbind(t, tt); mpi_append_hvec(fb, &foo[0].d[0], 100, sizeof(struct foostruct), 1, tt); mpi_commit(fb); Therefore it is not necesary to include always the first element of the structure as I thought... Sorry for the slip... Ricardo. -- To err is human, to really fool things around requires a computer ! +----------------------+---------------------------------+--------------+ | Ricardo A. Schmutzer | Internet: ricardo@ing.puc.cl | Conectividad | | PUC, Santiago, Chile | | Distr. Comp. | +----------------------+---------------------------------+--------------+ From owner-mpi-pt2pt@CS.UTK.EDU Sun Jul 18 11:22:39 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA08251; Sun, 18 Jul 93 11:22:39 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03193; Sun, 18 Jul 93 11:21:11 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 18 Jul 1993 11:21:10 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03185; Sun, 18 Jul 93 11:21:08 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19755; Sun, 18 Jul 93 10:21:07 CDT Date: Sun, 18 Jul 93 10:21:07 CDT From: Tony Skjellum Message-Id: <9307181521.AA19755@Aurora.CS.MsState.Edu> To: mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, mpi-core@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: All about threads Topic: Tacit Thread Safety Requirement of MPI1, Context Chapter, etc. Dear colleagues: In reviewing comments about the latest context draft, it has been repeatedly told me that we are at a crucial stage in MPI, because we have to agree on the context model, etc, as soon as possible. I concur with that assessment. In trying to find a consistent way to acquire safe communication space for groups, the issue of thread safety arises, because overlapping concurrent threads would have to work correctly. I am currently confident about the single-threaded case, and I am NOT CONFIDENT about the multi-threaded case. Does anyone have real experience with multi-threaded message passing (has it been done in an application setting, like we assume for MPI)? I need immediate guidance (specific guidance) about what multi-threaded programming MEANS in MPI, if this was in fact a reasonable requirement for MPI in the first place, and how multi-threading impacts point-to-point and collective communication (that is, real programs with examples). For instance, do we assume named threads or unnamed threads (and would this help). Is there an examplar threads package ??? Here is one problem in a nutshell. We discussed "statically initialized libraries" from time to time. Well, if there are multiple overlapping threads, then one would need to have separate contexts statically initialized for each concurrent thread. Such threads have group scope. Hence, groups would have to cache contexts for each concurrent thread (notice: groups cacheing contexts). I propose that we have a serious discussion on what thread safety really means for MPI1. I need for there to be well-formulated guidelines and in-depth debate immediately, so that the context committee can work effectively within these requirements, or give feedback as to why they are unreasonable. Otherwise, I/we can't really make the context chapter bullet-proof in time for the next meeting (except for the single-thread case). We have discussed how contexts provide group safety, but not temporal safety from multiple invocations of operations on a context (for which a programming paradihm must be described; e.g., synchronizing or implicitly synchronizing ... also could be called quiescent-at-exit). Now we need to have a notion of how to provide safety with multiple threads, or how to program the multi-threaded environment consistently, with interspersesd MPI calls. - - - To summarize, I seriously propose that in absence of an in-depth debate and specification of what thread safety means in MPI1, that we abandon this requirement altogether (analogous to the abandonment of non-blocking collective operations). If thread safety were to remain a de jure requirement of MPI1, then I ask that there be examples (analogous to or supersets of our contexts examples, pt2pt examples, and collective examples) illustrating same. If this is to be an added task of my subcommittee [which makes reasonablee sense to me] then I am eager for assistance nonetheless. I would want to see what people think existing thread practice is, what the design choices are, and which we choose to support, as well. It is not obvious to me that we really know what we mean (formally, practically) by "thread safety" for SPMD/MPMD message passing applications. Recall that there are at least three kinds of threads: O/S threads, compiler threads, user threads (we seeem to really mean the latter in our discussions). Thanks + please advise soonest. Tony Skjellum PS References to accessible texts or papers or software (eg, portable thread packages) are acceptable forms of advice. PPS I would like to have a new draft of the context chapter out by August 1 (with possible revisions by August 5). I am getting one extremely negative set of feedback from a single vendor representative, and one more balanced feedback (ie, only two people are communicating with me on the context chapter). I am not seeing widespread debate over the context chapter. This MUST happen now, between the meetings, since we have our best current draft available. We will not be successful if we are debating it all again at the next meeting without careful thought now (eg, on the threads issue). From owner-mpi-pt2pt@CS.UTK.EDU Sun Jul 18 11:29:22 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA08277; Sun, 18 Jul 93 11:29:22 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03618; Sun, 18 Jul 93 11:28:53 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 18 Jul 1993 11:28:52 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03600; Sun, 18 Jul 93 11:28:50 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19772; Sun, 18 Jul 93 10:28:48 CDT Date: Sun, 18 Jul 93 10:28:48 CDT From: Tony Skjellum Message-Id: <9307181528.AA19772@Aurora.CS.MsState.Edu> To: mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, mpi-core@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Heterogeneous communication proposal Dear colleagues: In order to make inter-vendor MPI implementations and cluster computing with MPI even a reasonable possibility, I suggest that we need to adopt the requirement that data formats follow IEEE Std 1596.5-1993 Data Transfer Formats Optimized for SCI. I propose that debate be started on this topic, and that a presentation be made at MPI in which the features of 1596.5-1993 are discussed and elaborated. Currently, there is little hope for standardization between vendors (or home-brew heterogeneous MPI) implementations. We recognize that XDR is inefficient, so this IEEE standard seems the logical alternative. If we say nothing, implementations will surely become incompatible. I volunteer to champion this effort, but only after the context chapter issues are resolved (so for September meeting or later). It is very important, to my mind, that we embrace other reasonable standards in creating MPI, such as this data standard. - Tony Skjellum Enclosure: From dbg@SLAC.Stanford.EDU Thu Jul 15 15:37:11 1993 Date: Thu, 15 Jul 1993 12:56:25 -0800 From: dbg@SLAC.Stanford.EDU (Dave Gustavson) Subject: SCI Data Transfer Formats standard approved To: sci_announce@hplsci.hpl.hp.com X-Envelope-To: sci_announce@hplsci.hpl.hp.com Content-Transfer-Encoding: 7BIT X-Sender: dbg@scs.slac.stanford.edu Content-Length: 4329 X-Lines: 86 Status: RO In its June 1993 meeting, the IEEE Standards Board approved: IEEE Std 1596.5-1993 Data Transfer Formats Optimized for SCI. (The approved document was Draft 1.0 8Dec92, but with significant edits to clarify the vendor-dependent formats listed in the appendix.) Congratulations to the working group, and especially to working group chairman David James! This new standard defines a set of data types and formats that will work efficiently on SCI for transferring data among heterogeneous processors in a multiprocessor SCI system. This work has attracted much interest, even beyond the SCI community. It solves a difficult problem that must be faced in heterogeneous systems. Over the years a great amount of effort has been invested in translating data among dissimilar computers. Computer-bus bridges have incorporated byte swappers to try to handle the big-endian/little-endian conversion. Software and hardware have been used to convert floating point formats. It was always tempting to have the hardware swap byte addresses to preserve full-bus-width integers, which seem to look the same on big- and little-endian machines, and then not swap bytes when passing character strings. But finally we understood that this problem cannot be solved by the hardware (at least until some far-future day when we all use standardized fully tagged self-describing data structures!). The magnitude of the problem became clearer during work on Futurebus+, where we had to deal with multiple bus widths and their interfaces with other standards like VME and SCI. When you observe data flowing along paths of various widths through a connected system, you see how hardware byte-swappers can arbitrarily scramble the data bytes of various number formats such as long integer or floating point. Furthermore, the scrambling may depend on the particular path used and on the state of the bridge hardware at the time the data passed through! Finally the solution became clear: first, keep the relative byte address of each component of a data item fixed as it flows through the complex system. (This is now referred to as the "address invariance" principle.) Thus, character strings arrive unchanged, but other data items may have been created with their bytes in inconvenient (but well-defined) places. Then provide the descriptive tools needed to tell the compiler what the original format of the data was. (That is what this standard does.) The compiler knows the properties of the machine for which it is compiling, and thus now has enough information to allow it to generate code to perform the needed conversions before trying to do arithmetic on the foreign data. For example, when the compiler loads a long integer into a register it may swap bytes to convert from little-endian to big-endian significance, so that the register will contain the correct arithmetic value for use in calculations. Similarly, when an arithmetic result is stored back into a structure that is declared with foreign data types the compiler ensures that the conversions are done appropriately before the data are stored. This capability is critical for work in heterogeneous multiprocessors, but it is also useful for interpreting data tapes or disk files that were written on a different machine as well. The IEEE Std 1596.5-defined descriptors include type (character, integer, floating), sizes, alignment, endian-ness, and atomic properties (can I be certain this long integer is always changed as a unit, never by a series of narrower loads and stores that might allow inconsistent data to be momentarily visible to a sharing machine). The standard also includes a C-code test suite that can be used to check the degree of compliance of a given implementation. The chairman is Dr. David V. James, MS 301-4G, Apple Computer, 20525 Mariani Avenue, Cupertino, CA 95014, 408-974-1321, fax 408-974-9793, dvj@apple.com. Again, my hearty congratulations on a job well done! David Gustavson, SCI (IEEE Std 1596-1992 Scalable Coherent Interface) chair David B. Gustavson phone 415/961-3539 SCI (ANSI/IEEE Std 1596 Scalable Coherent Interface) chairman SLAC Computation Research Group, Stanford University fax 415/961-3530 POB 4349, MS 88, Stanford, CA 94309 dbg@slac.stanford.edu From owner-mpi-pt2pt@CS.UTK.EDU Sun Jul 18 14:37:43 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA08447; Sun, 18 Jul 93 14:37:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12900; Sun, 18 Jul 93 14:36:50 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 18 Jul 1993 14:36:49 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12892; Sun, 18 Jul 93 14:36:46 -0400 Via: uk.ac.southampton.ecs; Sun, 18 Jul 1993 19:36:23 +0100 Via: brewery.ecs.soton.ac.uk; Sun, 18 Jul 93 19:28:00 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Sun, 18 Jul 93 19:38:12 BST Date: Sun, 18 Jul 93 19:38:15 BST Message-Id: <10706.9307181838@holt.ecs.soton.ac.uk> To: mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: All about threads > From: Tony Skjellum > Does anyone have real experience with multi-threaded message passing > (has it been done in an application setting, like we assume for MPI)? Yes. Most code written for transputer-based machines is multithreaded. Until quite recently (a couple of years ago) almost all the machines we had at Southampton were transputer based, so we developed quite a number of multi-threaded applications. We are not unique in this respect. Transputers have been very popular in Europe, and there are a large number of programmers on this side of the atlantic who are used to programming parallel machines in a multi-threaded style, using synchronous unbuffered message-passing primitives. > I need immediate guidance (specific guidance) about what multi-threaded > programming MEANS in MPI, if this was in fact a reasonable requirement Hmmm - I fear I'm sailing into dangerous waters here, but here goes... My understanding of what I thought was a consensus on threads is that in a multi-threaded environment, each thread can use the usual MPI primitives in the normal way, to communicate with other tasks, oblivious (almost) to any other threads within its own process. I say `almost' obvlivious, because obviously if two threads each try to receive a message with the same source, tag and context, then only one of them should succeed if a matching message arrives. I would suggest that MPI stipulate that it be unspecified which thread would get the message in such a situation. As far as I can see, this is the only place where extra semantics creep in. In particular, I don't see why contexts should introduce any further complication. A message with a particular context value should match a receive specifying the same context value (and source, tag), regardless of which thread posted it (the receive). Different contexts are distinct regardless of any considerations to do with threads. A good use for contexts in a multi-threaded environment would be to enable messages to be directed at particular threads within a process. Of course, in such a scheme the threads need somehow to find out which contexts they are supposed to be using, since contexts are allocated on a per-process (rather than per-thread) basis. As far as I'm concerned, this is outside the scope of MPI as currently defined, and would depend on the details of the particular threads implementation. However, an example of how things might proceed would be if there was initially one thread which allocated a number of contexts, then created some threads, and passed them each some contexts to use, in a manner specific to the thread creation scheme. > for MPI in the first place, and how multi-threading impacts point-to-point > and collective communication (that is, real programs with examples). I don't think we need examples - just some words along the above lines. > For instance, do we assume named threads or unnamed threads (and would > this help). Is there an examplar threads package ??? I don't think this is an issue, since MPI will not contain any operations that explicitly affect threads. > Here is one problem in a nutshell. We discussed "statically > initialized libraries" from time to time. Well, if there are multiple > overlapping threads, then one would need to have separate contexts > statically initialized for each concurrent thread. Such threads have > group scope. Hence, groups would have to cache contexts for each > concurrent thread (notice: groups cacheing contexts). Not a problem if contexts are allocated on a per-process basis, as I presumed they would be. What do other people think about this? By the way though Tony, I never did really understand what you meant by "statically initialized libraries". I hate to show my ignorance, but this is not a term I have encountered before - would you care to explain please? > I propose that we have a serious discussion on what thread safety > really means for MPI1. I need for there to be well-formulated guidelines and Well, I just had my $0.02 worth... > To summarize, I seriously propose that in absence of an in-depth > debate and specification of what thread safety means in MPI1, that we > abandon this requirement altogether (analogous to the abandonment of Well let's have some debate then. I think it would be disastrous not to address the issue of thread safety. > and collective examples) illustrating same. If this is to be an added > task of my subcommittee [which makes reasonablee sense to me] then I > am eager for assistance nonetheless. I would want to see what people think If my world view is seen as acceptable, then there is no implied extra work for your subcommittee. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Sun Jul 18 17:19:08 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10761; Sun, 18 Jul 93 17:19:08 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20907; Sun, 18 Jul 93 17:17:30 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 18 Jul 1993 17:17:29 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20899; Sun, 18 Jul 93 17:17:27 -0400 Via: uk.ac.southampton.ecs; Sun, 18 Jul 1993 22:17:20 +0100 Via: brewery.ecs.soton.ac.uk; Sun, 18 Jul 93 22:08:56 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Sun, 18 Jul 93 22:19:08 BST Date: Sun, 18 Jul 93 22:19:12 BST Message-Id: <10742.9307182119@holt.ecs.soton.ac.uk> To: mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: more on threads > From: Tony Skjellum > Thanks for your responses. I am trying to figure out how to deal with > the creation of communicators and contexts in a way that is "thread safe." > I will respond specifically to your two mails as soon as I can. For now... Hmmm, I guess my message dealt mainly with using contexts once they'd been created. Also, having thought a bit more about it, maybe a bit more needs to be said about what "thread safe" means, than I suggested. Lets take mpi_contexts_alloc() as an example. How I'd expect this to be used normally would be for it to be called by one thread per process in the group doing the allocating. But what happens if, say, if a different thread in one of those processes `simultaneously' makes a call to mpi_contexts_alloc()? This could happen if for example one of the processes in the group also belonged to another group, and wanted to allocate some contexts for use in it, using a different thread to make the call to mpi_contexts_alloc(). Presumably in such a case "thread safe" means that the behaviour of the system is as if the calls in the two threads were executed sequentially in some order. In other words they have to be treated as atomic operations. I guess that since two events are rarely simultaneous, the situation we are considering is when one thread calls mpi_contexts_alloc and begins to execute it, but then another thread also calls it, before the first invocation is complete. What is required is that there is some form of locking, so that the first invocation executes completely before the second one starts. Of course all of this does not prevent the example with the overlapping groups from being erroneous, since although the allocations are atomic, they may well occur in the wrong order. It would be up to the programmer to make sure one thread didn't make the call till the other had finished. How best to do that without introducing special thread support such as barriers to synchronize them? I'm not sure. A use for sending messages to yourself perhaps? The same issues apply to all of the collective operations, so for example I'd expect calls to mpi_barrier() to be made by one thread on each process in the group to be synchronized. > Ian asks about "contexts allocated on a per process basis." I have no > idea what this means. :-) Now I come to think of it, neither do I :-) In the light of the above discussion about making calls to mpi_contexts_alloc() an atomic operation, what I said in my previous message about contexts being passed to threads by their creators is not the only way of doing things. The threads could indeed create their own contexts, although there is that problem of interleaving... I suspect I oughtn't to mention this, but I suppose that it could be resolved if groups were objects with globally unique identifiers that were passed as part of the message header. Nope? Thought not... > Know that I am personally in favor of having threads, I am just trying I'm pleased to hear that. > to reveal to everyone the complexity that is being incorporated by > supporting the multiple threads, and that we agree specifically on what > that complexity shall be. Otherwise, I keep missing the mark with the > semantics of the context chapter :-) Yes, we should come to some agreement. As I say though, I don't think this is specifically a problem that affects the work of the context subcommittee, but rather that of point to point and collective communications. Ian From owner-mpi-pt2pt@CS.UTK.EDU Mon Jul 19 11:46:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA15533; Mon, 19 Jul 93 11:46:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29029; Mon, 19 Jul 93 11:46:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 19 Jul 1993 11:46:30 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29020; Mon, 19 Jul 93 11:46:25 -0400 Via: uk.ac.southampton.ecs; Mon, 19 Jul 1993 16:46:15 +0100 Via: brewery.ecs.soton.ac.uk; Mon, 19 Jul 93 16:37:49 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Mon, 19 Jul 93 16:48:02 BST Date: Mon, 19 Jul 93 16:48:04 BST Message-Id: <11100.9307191548@holt.ecs.soton.ac.uk> To: mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: more on threads Please excuse me, but, I feel I'm going to have to indulge in a bit of commenting on my own comments ;-) > From: Ian Glendinning > invocation executes completely before the second one starts. Of course all > of this does not prevent the example with the overlapping groups from being > erroneous, since although the allocations are atomic, they may well occur in > the wrong order. It would be up to the programmer to make sure one thread > didn't make the call till the other had finished. How best to do that > without introducing special thread support such as barriers to synchronize > them? I'm not sure. A use for sending messages to yourself perhaps? The It now occurs to me that the obvious method of doing this is for each thread to use a separate context to allocate the new contexts (or to perform whatever other collective operation it wants to). That is, go back to using a communicator as an argument to mpi_contexts_alloc() rather than just a group. Now there must have been good reasons for the change to just a group in the last draft, but I confess that they evade me. Could someone explain what they were please? In fact thinking about it, the whole point of having contexts in the first place is really to stop parallel threads treading on each other's toes, in the sense that message transfers invoked by `parallel procedures' (a la contexts chapter) continue to move around the system asynchronously after the procedure returns. In this sense I'm regarding the message-passing system as a thread running in parallel with the user's code here, which in fact it is. However, when we move over to user-defined threads, the same requirement exists to use contexts to protect them from each other. Ian From owner-mpi-pt2pt@CS.UTK.EDU Mon Jul 19 11:59:00 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA15643; Mon, 19 Jul 93 11:59:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29838; Mon, 19 Jul 93 11:59:15 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 19 Jul 1993 11:59:14 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29830; Mon, 19 Jul 93 11:59:13 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA26672; Mon, 19 Jul 93 10:59:05 CDT Date: Mon, 19 Jul 93 10:59:05 CDT From: Tony Skjellum Message-Id: <9307191559.AA26672@Aurora.CS.MsState.Edu> To: igl@ecs.soton.ac.uk, mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: more on threads >From owner-mpi-context@CS.UTK.EDU Mon Jul 19 10:47:49 1993 >Received: from Walt.CS.MsState.Edu by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); > id AA26451; Mon, 19 Jul 93 10:47:48 CDT >Received: from CS.UTK.EDU by Walt.CS.MsState.Edu (4.1/6.0s-FWP); > id AA15070; Mon, 19 Jul 93 10:47:47 CDT >Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) > id AA29074; Mon, 19 Jul 93 11:47:28 -0400 >X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Jul 1993 11:47:27 EDT >Errors-To: owner-mpi-context@CS.UTK.EDU >Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) > id AA29020; Mon, 19 Jul 93 11:46:25 -0400 >Via: uk.ac.southampton.ecs; Mon, 19 Jul 1993 16:46:15 +0100 >Via: brewery.ecs.soton.ac.uk; Mon, 19 Jul 93 16:37:49 BST >From: Ian Glendinning >Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; > Mon, 19 Jul 93 16:48:02 BST >Date: Mon, 19 Jul 93 16:48:04 BST >Message-Id: <11100.9307191548@holt.ecs.soton.ac.uk> >To: mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu >Subject: Re: more on threads >Status: R > >Please excuse me, but, I feel I'm going to have to indulge in a bit of >commenting on my own comments ;-) > >> From: Ian Glendinning > >> invocation executes completely before the second one starts. Of course all >> of this does not prevent the example with the overlapping groups from being >> erroneous, since although the allocations are atomic, they may well occur in >> the wrong order. It would be up to the programmer to make sure one thread >> didn't make the call till the other had finished. How best to do that >> without introducing special thread support such as barriers to synchronize >> them? I'm not sure. A use for sending messages to yourself perhaps? The > >It now occurs to me that the obvious method of doing this is for each thread >to use a separate context to allocate the new contexts (or to perform >whatever other collective operation it wants to). That is, go back to using a >communicator as an argument to mpi_contexts_alloc() rather than just a group. >Now there must have been good reasons for the change to just a group in the >last draft, but I confess that they evade me. Could someone explain what they >were please? In fact thinking about it, the whole point of having contexts in >the first place is really to stop parallel threads treading on each other's >toes, in the sense that message transfers invoked by `parallel procedures' >(a la contexts chapter) continue to move around the system asynchronously >after the procedure returns. In this sense I'm regarding the message-passing >system as a thread running in parallel with the user's code here, which in >fact it is. However, when we move over to user-defined threads, the same >requirement exists to use contexts to protect them from each other. > Ian > Quiescence as a requirement for library safety was the original mechanism we proposed. This meant that: i) no pending operations were present on a context ii) none would occur out of band while a library was using that context iii) mpi_contexts_alloc() & mpi_comm_make relied on quiescence to operate properly At the last meeting, there was pummeling regarding the quiescence requirement, in as much as, libraries could not guarantee their own safety (analogous to a stack-based language giving new, automatic variables to an invocation of a function). Changing the semantics to group instead of communicator in these two calls was meant to relieve the user of this worry. We also thereby removed the "you need a safe context to get a safe context." If multi-threading needs this rule, then we have to go back to a more restrictive model of computing for the non-threaded case; ie, that quiescence property. I believe that for the single-threaded case, the semantics currently posed are good. However, I need to understand how we manage the multi-threaded case... I need an explanation of how the following scenario will be handled (simple as possible) 1 group, 2 threads on the entire group, both threads call mpi_comm_make. Using contexts to disambiguate threads sounds good to me, off hand, but isn't that like giving threads "names." Do we want this association? Admittedly, I need to think more about Ian's comments. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Mon Jul 19 13:22:22 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA16287; Mon, 19 Jul 93 13:22:22 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05737; Mon, 19 Jul 93 13:22:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 19 Jul 1993 13:22:32 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05729; Mon, 19 Jul 93 13:22:30 -0400 Via: uk.ac.southampton.ecs; Mon, 19 Jul 1993 18:22:17 +0100 Via: brewery.ecs.soton.ac.uk; Mon, 19 Jul 93 18:13:46 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Mon, 19 Jul 93 18:23:59 BST Date: Mon, 19 Jul 93 18:24:01 BST Message-Id: <11148.9307191724@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: more on threads Oops, I mailed this to "pt2pt" instead of "mpi-pt2pt" by mistake. From now on I think I'll stick to "mpi-context", as I'm getting fed up getting two copies of everything, so please look there if you're interested. Ian ----- Begin Included Message ----- From igl Mon Jul 19 18:12:58 1993 To: mpi-context@edu.utk.cs, pt2pt@edu.utk.cs Subject: Re: more on threads Content-Length: 1681 > From: Tony Skjellum > At the last meeting, there was pummeling regarding the quiescence requirement, > in as much as, libraries could not guarantee their own safety (analogous > to a stack-based language giving new, automatic variables to an invocation > of a function). Changing the semantics to group instead of communicator > in these two calls was meant to relieve the user of this worry. We also > thereby removed the "you need a safe context to get a safe context." > If multi-threading needs this rule, then we have to go back to a more > restrictive model of computing for the non-threaded case; ie, that quiescence Thanks for that explanation. All is now clear... > property. I believe that for the single-threaded case, the semantics > currently posed are good. However, I need to understand how we manage > the multi-threaded case... I don't see any cleaner way than using contexts. If people are desperate to have the non-quiescent allocator, perhaps we could compromise and have one of each? People wanting to use threads could then use the one with the requirement for quiescence. > I need an explanation of how the following scenario will be handled > (simple as possible) > > 1 group, 2 threads on the entire group, both threads call mpi_comm_make. > > Using contexts to disambiguate threads sounds good to me, off hand, but > isn't that like giving threads "names." Do we want this association? Yes it is like that, and yes we do want it, if we want to make threads useable. And to do that, you've somehow got to identify them. This does it in a way that's not tied to any particular threads implementation. Ian ----- End Included Message ----- From owner-mpi-pt2pt@CS.UTK.EDU Fri Jul 23 18:42:43 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23597; Fri, 23 Jul 93 18:42:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15642; Fri, 23 Jul 93 18:41:51 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Jul 1993 18:41:47 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15628; Fri, 23 Jul 93 18:41:35 -0400 Message-Id: <9307232241.AA15628@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 2731; Fri, 23 Jul 93 18:41:34 EDT Date: Fri, 23 Jul 93 18:41:33 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V6.DVI.* %%Pages: 38 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 1 106 df<1204A21200A4123812FC12D81218A212321233123E123C080F7E8E0B>105 D E /Fc 2 81 df<1470EB01F0EB03E0EB0780EB0F00131E5B5BA25BB3AD485AA25B1203485A90 C7FC120E123C12705A1270123C120E7E7F6C7E12017FA26C7EB3AD1378A27F7F7FEB0780EB03E0 EB01F0EB007014637B811F>26 D80 D E /Fd 1 1 df0 D E /Fe 1 111 df110 D E /Ff 2 50 df<121FEA3F80EA60C0EA4040EAC060A8EA60C0A2EA3F80EA1F 000B107F8F0F>48 D<1218127812F812981218AAEAFF80A209107E8F0F>I E /Fg 4 111 df<126012F0A2126004047D830B>58 D<126012F0A212701230A31260A212C012 40040B7D830B>I<127EA2120EA25AA45AA2EA3BC0EA3FE0EA7C701278EA7030A2EAE070A4EAC0 E013C012E1EAE380EA7F00123C0C1A7E9910>98 D110 D E /Fh 6 116 df<120313801300C7FCA4121C123E126612C612CC120CA25AEA19801231EA33 00123F121C091480930C>105 D<1360137013601300A4EA0380EA07C0EA0CE0EA18C0A21200A2 EA0180A4EA0300A4126612EE12FC12780C1A81930E>I<123EA2120CA45A137013F0EA19B8EA32 70EA3C301300123E1267EA6318A21330EAC3F0EAC1C00D147E9312>I110 D114 D<1207EA1F80EA19C0EA3180EA3000121E121FEA0380124112E1EAC30012FE127C0A0D7E8C10> I E /Fi 3 62 df48 D<1206121E12FE12EE120EADEAFFE0A20B137D9211>I<387FFFE0B512 F0C8FCA6B512F06C13E0140A7E8B19>61 D E /Fj 22 123 df<127012F8A3127005057D840C> 58 D<127012F012F8A212781218A31230A2127012601240050D7D840C>I<12E01278121EEA0780 EA01E0EA0078131EEB0780EB01E0EB0078141EEC0780A2EC1E001478EB01E0EB0780011EC7FC13 78EA01E0EA0780001EC8FC127812E019187D9520>62 D<48B512804814E039003C01F0EC007815 3C151C5B151EA35BA44848133CA3153848481378157015F0EC01E039078003C0EC0780EC0F0014 7CB55A14C01F1C7E9B22>68 D<001FB512F0A2383C07800038146012701260EB0F0012C0A3D800 1E1300A45BA45BA45BA4387FFF80A21C1C7F9B18>84 D97 D<123F5A120F120EA45AA4EA39E0EA3FF8EA3E38EA3C1C1278EA701EA3EAE03CA3133813781370 13E0EA71C0EA3F80EA1E000F1D7E9C12>I100 D<13F8EA03FEEA0F06121E123C1238EA781CEA7FF8EAFFE0EAF000A4EA7004130EEA 383CEA1FF0EA0FC00F127F9113>II<13E0 1201A2EA00C01300A6120EEA1F8012331263124312C3EA0700A2120EA3EA1C201360A2EA38E013 C0EA1F80EA0F000B1C7F9B0E>105 D<1307130FA213061300A613F0EA01FCEA031C1206A2120C 1200A21338A41370A413E0A4EA01C01261EAF380EAF70012FE127C1024809B11>I108 D<381C0F80383F3FC0386770E013C0EAC780A2EA0700A2380E01C0A3EB0384001C138CA2EB071C 1418383803F0381801E016127F9119>110 D<13FCEA03FF38078780380E03C0EA1C0112381278 A2EAF003A3EB0780A2EB0F00EA701E6C5AEA1FF0EA0FC012127F9114>I<380707C0380FCFE038 19D870EBF038EA31E013C00001133CA238038078A31470380700F0EB80E0EB81C0EBC380380EFF 00137C90C7FCA25AA4B47EA2161A819115>I115 D<13C01201A3EA0380A4EAFFF0A2EA0700A2120EA45AA31310EA3830A2136013C0EA1F80EA0F00 0C1A80990F>I<000E1360381F80E012331263384381C012C3EA0701A2380E0380A31488EB0718 A2130FEB1F303807F3F03803E1E015127F9118>I 120 D<000E1360381F80E012331263384381C012C3EA0701A2380E0380A4EB0700A25B5BEA07FE EA03EEEA000E5B1230EA78381330EA70E0EA3FC06CC7FC131A7F9114>I<3801E180EA07F1380F FF00EA0C1EEA180CC65A5B5B5B485A48C7FCEA0601485A1218EA3F0EEA7FFCEA61F8EAC0F01112 7F9113>I E /Fk 2 4 df0 D<12041206A2EAC660EAF7E0EA3F80EA1F 00EA3F80EAF7E0EAC660EA0600A212040B0D7E8D11>3 D E /Fl 18 118 df70 D<133F3801FFE0487F3807C0F8 380F807C381E001E003E131F003C7F48EB0780A348EB03C0A86C130700781480A2007C130F003C 1400003E5B6C133E6C6C5A6C6C5A6CB45A6C5BD8003FC7FC1A1F7E9D1F>79 DI82 D97 D99 D<130EABEA0F8EEA1FEEEA3FFEEA7C3EEA700EA212E0A612F0 EA701EEA7C3EEA3FFEEA1FEEEA0F8E0F1D7E9C15>II<3803C3C0EA0FFF 5A381C3800487EA56C5AEA1FF85BEA3BC00038C7FCA2EA1FFC13FF481380EA700738E001C0A3EA F003387C0F80383FFF006C5AEA07F8121B7F9115>103 D<12F0A41200A71270B2041D7E9C0A> 105 D<12E0B3AB031D7D9C0A>108 D<38E3F03F39EFF8FF80D8FFFD13C039F81F81E038F00F00 EAE00EAD1B127D9124>III114 DI<121CA6EAFFE0A3EA1C00AB1320EA1FF0120FEA07C00C187F970F> II E /Fm 42 122 df<1238127C127EA212 3E121EA2127C12F81260070A7A8414>44 D50 D<13F8120113B81203EA0738A2120F120E121C123C1238127812F0B51280A338003800A43803FF 80A311177F9614>52 D<127012F8A312701200A6127012F8A312700510798F14>58 D65 DIIII71 D<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE3F80A311177F9614>I< EAFFFEA3EA0380B1EAFFFEA30F177E9614>I76 D<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A21303A538FE0FE0A313178096 14>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136EA4EAFE3EA311177F9614>I< EA1FF0EA7FFCA2EA701CEAF01EEAE00EADEAF01EEA783CEA7FFCA2EA1FF00F177E9614>II< EAFFE07F7FEA383C7F130EA3131E5BEA3FF85B7FEA383C131CA41480EB1DC0EAFE1F130FEB0700 12177F9614>82 DII<38FF8FF8A3381C01C0AEEA1E03000E1380EA0F0738 07FF006C5AEA00F81517819614>I<38FE3F80A338380E00A26C5AA56C5AA4EA0630EA0770A3EA 0360A213E0A26C5A11177F9614>I88 D<38FE3F80A3383C1E00EA 1C1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>I 97 D<12FCA3121CA413FCEA1FFE7FEB0780EA1E03001C13C01301A4EA1E03EB0780EA1F0FEBFF 005BEA1CF81217809614>II<137EA3130EA4EA07CEEA1FFE123FEA7C3EEA781EEAF00E 12E0A4EAF01EEA703EEA787E383FFFC0EA1FEFEA078F12177F9614>II< 133C13FE1201EA03DE138C1380A2EAFFFEA3EA0380AAEA7FFCA30F177F9614>II<12FCA3121CA413F8 EA1FFE7F130FEA1E07121CA838FF9FE0A31317809614>I<1203EA0780A2EA0300C7FCA4EAFF80 A31203AAEAFFFEA30F187D9714>I108 D110 DII<38FF1F80EB7FC013FF3807E080EBC0005BA290 C7FCA6EAFFFCA31210808F14>114 DI<1207A5EAFFFCA3EA0700A613 0EA3139E13FCEA03F8EA00E00F157F9414>II<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C5AA35BA3EA7B80127F90C7 FC127E123C11187F8F14>121 D E /Fn 6 104 df0 D<127012F8A3127005057D8C0C>I15 D<00C01306B36C130E0060130C0070131C6C1338381F01F03807FF C03800FE00171A7E981C>91 D<131F137F13F0EA01E013C0ADEA0380EA070012FE12F812FE1207 EA0380EA01C0AD13E0EA00F0137F131F10297E9E15>102 D<12F8B4FCEA0F80EA03C01201ADEA 00E013F0133F131F133F13F013E0EA01C0AD1203EA0F80EAFF0012F810297E9E15>I E /Fo 35 122 df<127012F812FCA3127C120CA21218A2123012701220060D7D850C>44 D<127812FCA4127806067D850C>46 D<127812FCA412781200A5127812FCA4127806117D900C> 58 D<1303497EA3497EA2497EA3EB33F0A2EB63F81361EBE1FC13C0A248487E13FF487FEB003F A20006EB1F80A2000E14C039FFC0FFFCA21E1A7F9921>65 D<90381FE0209038FFF8E03803F80F 3807C003381F800190C7FC123E007E1460127C12FC1500A61560127C127E003E14C07E90388001 803907C003003803F80E3800FFFCEB1FE01B1A7E9920>67 D III73 D76 DI<39FFE0 0FFCA2390FF000C07FEA0DFCEA0CFE137E7F1480EB1FC0EB0FE01307EB03F014F8EB01FCEB00FE 147E143FA2141F140F14071403A238FFC00114001E1A7E9923>I80 D<007FB51280A238783F07 007013030060130100E014C000C01300A300001400AF380FFFFCA21A1A7E991F>84 D97 DIII I<137E48B4FC3803CF80EA078F120FEB87001380A3EAFFF0A2EA0F80ADEA7FF0A2111A80990E> I<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C3CEA3FF8EA37E00070C7FCA2EA 3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF803807FC0013197F9016>II<121E123FA4121EC7FCA4 127FA2121FADEAFFC0A20A1B809A0C>I108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7FE7FE0A223117F9026> IIII114 DI<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03 E00D187F9711>I<38FF1FE0A2EA1F03AB1307130F380FFFFCEA03F316117F9019>I<38FF83F0A2 381F0180A21383000F130013C7EA07C6A2EA03ECA2EA01F8A36C5AA2136014117F9017>I<39FF 9FF1F8A2393F07C0E0001F14C014E0EB8DE1000F148014F13907D8F30014FBEBF8FF3803F07EA2 3801E03CA33800C0181D117F9020>I<38FF83F0A2381F0180A21383000F130013C7EA07C6A2EA 03ECA2EA01F8A36C5AA21360A2EA70C012F8EAC18012C3007FC7FC123C14187F9017>121 D E /Fp 17 122 df97 D<123F5A120F120EA45AA4EA39C0EA3FE0EA3E70EA38 30EA78381270A3EAE070A3136013E013C012E1EA6380EA7F00123C0D1D7B9C13>I101 DII<13C0EA01E013C01380C7FCA6120E123FEA33801263EAC700A21207120EA35A13 40EA38C0A3EA3980EA3F00121E0B1C7D9B0D>105 D107 DI<391E1F07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380 E0A3EC81C2391C0701C6A2EC038CA239380E01F839180600F01F127D9122>III114 DI<13C01201A3EA0380 A4EAFFE0A2EA0700A2120EA45AA31320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I118 D120 D<380F0180EA1F8312331263EB870012C3EA0707 A2EA0E0EA4EA0C1C121C120CEA0E3CEA0FF81203EA0038A2EA3070EA786013E0EA71C0EA3F8000 1EC7FC111A7D9114>I E /Fq 13 86 df<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA01C0A6EA0380A61207EAFFF8A20D1C7C9B15> 49 D<14301470A214F0801301A2EB0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFEA2 3801801EA23803001F801206120F397FC0FFF012FF1C1D7F9C1F>65 D<903807F030EB1FFC9038 7C0E603901F003E0EA03C038078001EA0F00000E1300001E14C05AA25A1500A25AA5EC0180EC03 001270127814066C5B001C131C000F5B3807C0E06CB45AD8007EC7FC1C1E7C9C1E>67 D<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B13C113C0A215C0EC C18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>69 D<380FFF80A23800F000 A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>73 D77 D<390FF80FFEA23900FC 01E0EC00C0A213DEA239018F0180A214811387A2EB83C1390303C300EB01E3A3EB00F3A2000613 7EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F>II<380FFFFC14FF 3900F00F80140315C01401A23801E003A315801407EC0F003803C03EEBFFF814E001C0C7FCA348 5AA6120FEAFFF85B1A1C7E9B1C>I83 D<001FB512F05A383C07810038EB806012701260A2EB 0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E>I<39FFF8FFE0A2390F001E 00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA0E0EEA07FCEA01F01B1D7A 9B1F>I E /Fr 10 58 df<121EEA7F801261EAC0C0A7EA6180127FEA1E000A0D7E8C0E>48 D<121812F8A21218A812FFA2080D7D8C0E>I<123EEA7F80EAC1C012C01200A2EA0180EA030012 0EEA18C01270EAFF80A20A0D7E8C0E>I<123E127FEA618012011203EA1F00A2EA0180EA00C012 C012C1EA7F80EA3E000A0D7E8C0E>I<12075AA2121B1233127312E3EAFFC0A2EA0300A2EA1FC0 A20A0D7E8C0E>II<120FEA1F801231EA60005AEAFF80A2EAC0C0A31261EA3F80EA1E000A0D7E8C0E>I<12 60EA7FE0A2EAC0C0EAC180EA03001206A35AA50B0E7E8D0E>I<121E123FEA6180A21273EA3F00 A2EA6380EAC0C0A212E1EA7F80EA3F000A0D7E8C0E>I<121E127FEAE180EAC0C0A3127FA21200 EA01801263EA7F00123C0A0D7E8C0E>I E /Fs 46 122 df45 D<12F0A404047B830E>I<13C01201120712FFA212FB1203B3A7B5FCA310227CA118>49 DII<131FA2132FA2136F13EF13CF1201A2EA038FA21207130F120F120E121E123CA2 1278A212F0B512F8A338000F00A915217FA018>I<383FFF80A3003CC7FCA913F8EA3FFE7FEB0F 80EA3E07383C03C01238380001E0A7384003C0A212E038700780387C1F00EA3FFEEA0FFCEA03F0 13227EA018>I<137E48B4FC5AEA07C148C7FC121EA25AA25AA213F8EAF3FEEAF7FF38FF0F80EA FC0738F803C0A2130100F013E0A51270A21278EB03C01238003C1380EA1C07381F0F00EA0FFE6C 5AEA01F013237EA118>IIII<131F A2497EA2133BEB7BC013731371EBF1E0A213E000017FA213C000031378A2138000077FA2130048 7FA2EBFFFE487FA2381C000F003CEB0780A212380078EB03C0A2127000F0EB01E0A25A1B237EA2 20>65 DI III<00FC147E6C14FEA300F7EB01DEA3EB800300F3149EA2EBC007A200 F1141EA2EBE00F00F0130EA2EBF01EA2EB701CEB783CA2EB3838EB3C78A2EB1C70A2EB1EF0EB0E E0A2130FEB07C0A390C7FC1F237BA22A>77 D<00FC131E7EA312F7A2EAF380A213C012F1A213E0 12F013F0A21378A21338133CA2131EA2130E130F1307A2149E1303A2EB01DEA2EB00FEA3147E17 237BA222>III82 D<13FF000313C04813F0EA0F83EA1E0048132014005AA5127C123C123FEA1FC0EA0FFC 6CB4FC0001138038003FC0EB07E0EB01F0130014F81478A5124000E013F07E38F801E0387F07C0 383FFF80000F1300EA01FC15257EA31B>I<00F0133CB3A900781378A36C13F0381E01E0EA1F87 380FFFC000031300EA00FC16247BA221>85 D97 D<12F0ADEAF1F0EAF7FCB47EEAFC1F487E38F00780A2EB03C0A8EB0780A238F80F00EAFC3EEAFF FEEAF7F8EAF3F012237CA219>IIII<133F13FF5AEA03C113801207A8EAFFF8A3EA0780B3 102380A20F>I<3801F0F8EA07FF5A381F1F80381E0F00383C0780A5381E0F00EA1F1F13FE5BEA 39F00038C7FCA2123CEA3FFE381FFFC014E04813F0EA780138F800F8481378A4007813F0383E03 E0381FFFC06C13803801FC0015217F9518>I<12F0ADEAF1F8EAF7FCEAFFFEEAFE1EEAF80FA212 F0B010237CA219>I<12F0A41200A912F0B3A404237DA20B>I<13F0A41300A913F0B3A91280EAE3 E012FFEA7FC0EA1F000C2D83A20D>I<12F0ADEB1F80EB3F00133E5B5BEAF1F0EAF3E0EAF7C012 FFA27FA2EAFDF0EAF8F8EAF078137C7F131E131F7F1480EB07C012237CA218>I<12F0B3B10423 7DA20B>I<39F0F807C039F3FE1FF0B5EA7FF839FE0FF07800FCEBE07C39F807C03CA200F01380 AF1E167C9529>IIII114 DI<120FA6EAFFF8A3EA0F00AE1308131CEA07FC13F8EA03E00E1C7F9B12>II<38F001E0A2387803C0A3383C0780A300 1E13005BA2EA0F0E131EA2EA071C139C13BCEA03B8A213F86C5AA213167F9516>I<387801F000 7C13E0383E03C0EA1E07000F138038078F0013DEEA03FC12016C5A137013F8487EEA03DC139EEA 078F380F0780121EEB03C0383C01E0387800F000F813F81516809516>120 D<38F001E0A2387803C0A2127C383C0780A2121E14005B120F130EEA071E139EEA039CA3EA0198 13D8EA00F85BA35BA212015BA2EA4380127F90C7FC127E13207F9516>I E /Ft 68 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4B5FCA2EA0E07AC 387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07AC387F9FE0A213 1A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E138190380380C01500A4B612 E0A2380E0380AC397F8FE3FCA21E1A809920>I34 D<126012F012F812781218A31230127012E01240050B7D99 0B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0AA1260A21270A2 12301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E121C120C7E1207 A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A267E9B0F>I<12 6012F0A212701230A31260A212C01240040B7D830B>44 DI<126012F0 A2126004047D830B>I<130CA2131C1318A213381330A213701360A213E013C0A212011380A212 0313005A1206A2120E120CA2121C1218A212381230A212701260A212E05AA20E257E9B13>II<12035AB4FCA21207B3EAFFF0A20C197D9813>III<1370A213F0A21201120313701206A2120C121C12181230A2126012E0B5FCA2EA 0070A5EA07FFA210197F9813>I<126012F0A212601200A8126012F0A2126004107D8F0B>58 D<126012F0A212601200A8126012F0A212701230A31260A212C0124004177D8F0B>I61 D63 D<130C131EA3133FA3497E13 67A3EBC3C0A3380181E0A348B47EA2130000061378A3487F121E39FF81FFC0A21A1B7F9A1D>65 DIIIIII<39FF F3FFC0A2390F003C00A9EBFFFCA2EB003CAB39FFF3FFC0A21A1A7F991D>II<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3EB7818A3EB 3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>77 DI<137F3801FFC03807C1 F0380F0078001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C13 1C001E133C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II82 DI<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FCAF3803FFF0A2191A 7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A3801E0E038007FC0011FC7FC1A 1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC5DD807801306EB81 9EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE13F0EBFC03017C5B A2EB7801A201385BEB3000241B7F9927>87 D<39FFE01FF0A2390F800F00000713066C6C5A13E0 00015BEBF038000013306D5A137CEB3CC0133F6D5AA26DC7FCA9EBFFF0A21C1A80991D>89 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A2381F0F00EA 1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A61270131EEA 3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA03BCEA073CEA0E181300A5EAFF C0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870EA3FE0EA 7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F9013>I< 12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA21218C7 FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A21200B012 6012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B121DEA 1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80A2091A 80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D107F8F20> IIIIIII< 120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>II<38FF3F80A2381C0E00130CA26C5AA21338EA0730 A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E5B13 631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA03B8 13B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80A238 1C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE12781117 7F8F14>III E /Fu 78 126 df<126012F0AF12601200 A4126012F0A212600419779816>33 D38 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312 781238123C7E7E7E13C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313 C01201A313E01200A7120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>I< EA01C0A4EA71C738F9CF80387FFF00EA1FFCEA07F0A2EA1FFCEA7FFF38F9CF803871C700EA01C0 A411127E9516>II<1238127C127EA2123E120E12 1E123C127C12F81260070B798416>II<127012F8A312700505788416> IIIIII<137C13FC13DC1201EA039CA2EA071C120F120E121E123C1238127812F0B512E0A338001C 00A53801FFC0A313197F9816>II<13F8EA03FEEA0FFFEA1F 0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2EAF80F38F00780A2EAE00312F0A21270EA78 07EB0F006C5AEA1FFEEA0FF8EA03E011197E9816>I<12E0B51280A338E00F00131EC65A133813 78137013F05B12015B12035BA3120790C7FCA7111A7E9916>I57 D<127012F8A312701200A8127012F8A3127005127891 16>I<1238127CA312381200A81238127CA3123C121C123C123812F812F012600618799116>III<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC0 00FF13E0007F13C013197F9816>65 DI<3801F180 EA07FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0F FE6C5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7 FCA2131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>III<387F1FC038FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C 07A9387F1FC038FFBFE0387F1FC013197F9816>II< 387F0FE038FF8FF0387F0FE0381C0780EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E3C 131CEA1C1E7F7F14801303387F07E038FF8FF0387F07E01419809816>75 DI<38FC07E0EAFE0FA2383A0B80EA3B 1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313197F9816>I<387E1FC038FF3FE038 7F1FC0381D07001387A313C7A2121CA213E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F 13197F9816>IIIIII<387FFFE0 B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F038 1C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FE0FE0EAFF1FEAFE 0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813B8A3EA01B013F0A26C5A13197F98 16>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803A313E3EA39F3A213B300191300A6 1313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F131F380E1E00131CEA073C1338EA 03B813F012015B120012017F120313B81207131CA2EA0E0EA2487E387F1FC000FF13E0007F13C0 13197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C5A A8EA03F8487E6C5A13197F9816>I<387FFF80B5FCA238E007005B131E131CEA003C5B137013F0 485A5B1203485A90C7FC5A381E0380121C123C12781270B5FCA311197E9816>II93 D95 D97 D<127E12FE127E120EA4 133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF000E1300 EA067C1419809816>II<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F487E1270 EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA3FFE487E 6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5AEA1FFC48 5AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C07C0383F FF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF148013871303A2120E A9387FC7F038FFE7F8387FC7F01519809816>II<13301378A213301300A4EA1FF8A3EA0038B3EA6078EAF0F0EA FFE0EA7FC0EA3F800D237E9916>I<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5BEA0FF8 A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>II<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870AA38FE7CF8A315128091 16>IIII<38FF0F80EB3FE0 13FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713873803FF005BEA00F811 177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C13F83803F3F015128091 16>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4EA03B8A3EA01F0A26C5A 13127F9116>I<38FF1FE013BF131F38380380A413E33819F300A213B3EA1DB7A4EA0F1EA31312 7F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F012016C5A12017FEA03B8EA073C13 1CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<387F1FC038FF9FE0387F1FC0381C0700 120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279127BEA7F806CC7FC123C 131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A13F8485A485A485A48C7FC381E01C0 123C1278B5FCA312127F9116>II<12FCB4FC13C012031201A97F6CB4FCEB7F80 A2EBFF00EA01E05BA9120312FF90C7FC12FC11207E9C16>125 D E /Fv 29 122 df<90381FE1F890B512FC3903F03F3E3807C07EEB807C000F141C1500A5B612C0A2390F 807C00AE397FE1FFC0A21F1D809C1C>11 D45 D63 D65 D<90381FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A2 12FE1500A7007E1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D 9B22>67 D70 D<3807F860381FFEE0EA3C07EA7801EA700012F01460A26C13 0012FEEAFFE0EA7FFE6C7E1480000F13C06C13E0EA007FEB03F01301130012C0A214E07E38F001 C0EAFC0338EFFF00EAC3FC141C7D9B1B>83 D97 DIIII<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E1FEA1FFCEA33F00030 C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F0383E03E0381FFFC03803 FE00151B7F9118>103 DI<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>I107 DI<39FF1FC0FE90387FE3FF3A1FE1 F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE0381FE1F0 EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFF E0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFFE0A217 1A7F911B>I114 DI<1203A35AA25AA2123FEAFFFCA2EA1F00A9130CA4EA0F 98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03F818127F911B> I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5AA213 78A2133016127F9119>I<39FF8FF8FEA2391F03E030A201831370000FEBF0601386D807C613C0 EBCEF8EBEC790003EB7D80EBF83D0001EB3F00A2497E0000131EEBE00EA21F127F9122>I<38FF C7FCA2381F8180EA0F833807C700EA03EEEA01FC5B1200137C13FEEA01DFEA039F38070F803806 07C0380C03E038FF07FCA216127F9119>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C1 80A23803E300A2EA01F6A213FE6C5AA21378A21330A25B1270EAF8E0EAC0C0EAE380007FC7FC12 3E161A7F9119>I E /Fw 44 122 df45 D<12F8A505057A8411>I<137F 3801FFC0000713F0487F13C1381F007C48137E003E133E003C131E007C131FA300787F00F81480 AF00781400007C5BA36C133EA2003F137E6C6C5A380FC1F813FF6C5B000113C06C6CC7FC19297E A71E>48 D<1318133813F8120712FFA312F81200B3AA387FFFF0A414287CA71E>I<13FE3803FF C04813E04813F0381F03F8383C00FC0038137E12780070133E00F0133FA20060131F1220C7FC14 3FA2143EA2147CA214F8EB01F0EB03E0130714C0EB0F80EB1E005B5B5B485A485A485A48C7FC12 1E5A007FB5FCA418287EA71E>I<137F3801FFC0000713F0487FEA1F81383E00FC003C137E5A00 30133E1210C7127EA2147CA214FC495AEB07F0EBFFE05C8080EB01F8EB007C147E80A2EC1F80A5 12400060EB3F0012F06C5B007C137E383F81FC6CB45A6C5B000313C0C66CC7FC19297EA71E>I< EB03F013071305130DA2131D1339A21379A213F1A2EA01E11203A2EA07C1A2EA0F81A2EA1F01A2 123E127CA212F8B61280A4390001F000AA19277EA61E>I<383FFFFCA4003EC7FCAA133FEBFFC0 003F13E014F0EBC1F8EB80FCEB007C003E137EC7123E143FA712200060137E127000F813FCEA7C 01383F03F8EBFFF0000F13E06C13803800FE0018287EA61E>III<137F3801FFC0000713F0487F 381FC1FC383F007E003E133E007E133F007C7FA56C133EA26C5B380FC1F86CB45A6C5BA2000F13 F8381FC1FC383F007E003E133E487FA248EB0F80A6007CEB1F00A2007E5B6C137E381FC1FC6CB4 5A6C5B6C5BC66CC7FC19297EA71E>I<137F3801FFC0487F000F7FEBC1F8381F007C123E487F14 1E141F5AA2801580A4141F127CA2007E133F003E137F003F13FF381FFFEF6C13CF6C138F3903FE 1F00C7FCA2141E143EA25CA25C381801F0381E07E0383FFFC06C5B6C48C7FCEA03F819297EA71E >I66 D<49B4FC010F13E0013F13F85B EBFF013901F800304848131048481300485A485AA248C8FCA2127EA45AAA127EA47EA26C7EA26C 7E6C7E6C6C13046C6C131C6CB413FCEB7FFF6D13F8010F13E0010113001E2C7CAA26>II<12FCB3B3A606 2A7BA911>73 D77 DI80 D83 DI<00FC147CB3AE007E14F8A26C130115F0391F8003E0EBC00F390FF03FC06CB5 12800001EBFE006C6C5AEB1FE01E2B7BA929>I97 D<12F8AF133F38F9FFC000FB13E0B512F0130738FC01F838F800FC147CA2 147E143EA7147CA36C13F8130138FF07F0EBFFE000FB13C000F9138038F87E00172A7BA91F>I< EB7FC03801FFF0000713FC5A381FC07CEB0008003E13005AA35AA7127CA2127E123E6C130CEBC0 7C380FFFFC7E000113F038007F80161B7E9A1B>I<143EAF13FCEA03FF4813FE5AEA1FC1383F00 7E003E133E5AA212FC5AA77E127CA26C137E003F13FEEA1FC1EA0FFF6C13BE6C133EEA00FC172A 7EA91F>I<137E3803FF804813C04813E0381F83F0EA3F00003E13785A14380078133CB512FCA4 00F8C7FCA31278127CA27E003F130C381FC07C380FFFFC7E000113F038007F80161B7E9A1B>I< EB1FC0137F13FF5A3803F0005B485AA9EAFFFEA4EA07C0B3A5122A7FA912>I<9038F807803903 FE7FC0000FB5FC5A903807C000383E03E0A2486C7EA66C485AA26C485A13FF485BD83BFEC7FCEA 38F80078C8FCA27E383FFFC014F86C7F80487F387C007F48EB1F80140FA36C131F007EEB3F0038 3F80FE13FF000F13F86C5BC613801A287E9A1E>I<12F8AF133FEBFF8000FB13C0B512E0130738 FE03F0EAFC01A312F8B2142A7BA91F>I<12FCA61200A9127CB3A9062A7DA90E>I<13FCA61300A9 137CB3AFEA40FCEAE1F812FF13F0EA7FE0EA0F800E3683A910>I<12F8AFEB01F8EB03F0EB07E0 14C0EB0F80EB1F00133E5B5B12F9EAFBF012FF7F7FA2EAFE7E487E12F8EB1F80130F14C0EB07E0 A2EB03F0130114F8EB00FC162A7BA91D>I<12F8B3B3A6052A7CA90E>III<137F3801FFC000 0713F0487F381FC1FC383F007E003E133E487FA200787F00F81480A7007CEB1F00A2007E5B003E 133E003F137E381FC1FC6CB45A6C5B000113C06C6CC7FC191B7E9A1E>II114 DII<38F801F0B3A21303A2EAFC0FB5FCEA7FFDEA3FF9EA1FC1141B7B9A1F>I<00 F8EB0F807E007CEB1F00A2127E003E133EA26C133C147CA26C6C5AA36C6C5A13C1A23803E1E013 E300015BA213F33800F780A3017FC7FCA3191B7F9A1C>I<00F8EB0F807E007CEB1F00127E003E 133EA27E5CA2EA0F805CEA07C05C13C1EA03E15C120113E3EBF3C012005C137313770137C7FC13 3F133E131EA25BA35BA25BA2EA7FE0A25B90C8FC19277F9A1C>121 D E /Fx 78 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512 C0A2EA0E01AE387F87F8A2151D809C17>II<90383F03F09038FFCFF83903C0FC1C390701F03CA2390E00E0 1892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120C A31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA03001206120E120C121C1218123812 30A21270A21260A212E0AC1260A21270A21230A212381218121C120C120E12067EEA0180EA00C0 0A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380A21201A213C0AC1380A21203 A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>II<1306 ADB612E0A2D80006C7FCAD1B1C7E9720>I<127012F012F8A212781218A31230A2127012601240 050D7D840C>II<127012F8A3127005057D840C>I<1303A213071306A2 130E130C131C1318A213381330A213701360A213E013C0A21201138012031300A25A1206A2120E 120CA2121C1218A21238123012701260A212E05AA210297E9E15>II<12035A123F B4FC12C71207B3A3EAFFF8A20D1C7C9B15>III<131CA2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012E0 B512C0A238001C00A63801FFC0A2121C7F9B15>II<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC 1CEAF81E130E12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<126038 7FFF80A21400EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15 >III<127012F8A312701200A8127012 F8A3127005127D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012 601240051A7D910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61 D<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3 001F131F39FFC0FFF0A21C1D7F9C1F>65 DI<90381F80 80EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070 EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>IIII<90381F8080 EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA5ECFFF0 A20070EB07801278A27E121C121E7E3807800F3803F0393800FFF090381FC0001C1E7E9C21>I< 39FFF3FFC0A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>II76 DIIII82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA 1FFEEA07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F812 1E7E9C17>I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B038 03FFFCA21B1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0 EB7FC0011FC7FC1A1D7E9B1F>I<39FFE01FF0A2390F000780EC0300A238078006A2EBC00E0003 130CA2EBE01C00011318A26C6C5AA3EB7860A36D5AA3EB1F80A36DC7FCA313061C1D7F9B1F>I< 3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060A32603C0E713C0ECC3 C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E137EA3013C133CA3011C 133801181318281D7F9B2B>I<397FF0FFC0A23907C03E0000031338EBE03000011370EBF06000 005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371EB61F0EBC0F800011378EB807C38 03003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I<12FEA212C0B3B312FEA207297C9E0C>91 DI<12 FEA21206B3B312FEA20729809E0C>I97 D<12FCA2121CA9137EEA 1DFF381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C14 1D7F9C17>IIII<1378EA01FCEA039E EA071EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFF EA1F07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2 121CAEEAFF80A2091D7F9C0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E 70EA1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F 9C0C>I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A22112 7F9124>IIII<3803E180EA0FF9EA1E1FEA3C07127813 0312F0A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA512 1CA2123CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03 AC1307EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2 131CEA0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E 1B061486A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F 078038070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038 FF8FF8A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C 5AA26C5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>II< B512F0A21402808B15>I E /Fy 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815 FE489039001FFF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC 91B6FC130F013FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA2 6D14FF6C6CEB01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03 C66CEBF0010107903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090 267FFC0113C0EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093 C7FC5BA212FFAA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C 01F8EB3F80903A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fz 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D83 D99 D<903801FFC0011F13F8017F13FE9038FFC1FF00039038007F80D807 FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03FCA212FFA290B6FCA301E0C8FCA5127F A27F003F153CA26C6C147C000F15786C6C14F86C6CEB01F06C6CEB07E06C9038E03FC0013FB512 00010F13FC010013E026267DA52D>101 D105 D<01FEEB7FC000FF903803FFF8 020F13FE91381F03FFEC3C010007017814800003497E6D4814C05CA25CA291C7FCB3A4B5D8FC3F 13FFA430267CA537>110 DI116 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 75 356 a Fz(Section)33 b(1)75 564 y Fy(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)884 780 y Fx(Marc)15 b(Snir)684 830 y(William)10 b(Gropp)j(and)h(Ewing)f(Lusk)75 964 y Fw(1.1)59 b(Intro)r(duction)75 1055 y Fx(This)10 b(c)o(hapter)i(is)e(a)g(draft)g(of)g(the)h(curren)o(t)h (prop)q(osal)e(for)g(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)14 b(Sending)c(and)g(receiv-)75 1105 y(ing)i(of)g(messages)h(b)o(y)g(pro)q (cesses)i(is)e(the)g(basic)g(MPI)g(comm)o(unication)d(mec)o(hanism.)15 b(All)d(other)h(comm)o(unication)75 1155 y(functions)18 b(can)g(b)q(e)h (implemen)o(ted)d(on)h(top)h(of)g(this)g(basic)g(comm)o(unication)d(la)o(y)o (er)i(\(although)g(a)h(more)f(direct)75 1205 y(implemen)o(tatio)o(n)11 b(ma)o(y)h(lead)i(to)g(greater)h(e\016ciency\).)158 1254 y(The)g(basic)g(p)q (oin)o(t)f(to)h(p)q(oin)o(t)f(comm)o(unication)d(op)q(erations)k(are)h Fv(send)d Fx(and)i Fv(receiv)o(e)p Fx(.)k(A)c Fv(send)e Fx(op)q(eration)75 1304 y(creates)18 b(and)d(sends)i(a)e(message.)23 b(The)17 b(op)q(eration)e(sp)q(eci\014es)j(a)d Fv(send)i(bu\013er)d Fx(in)h(the)h(sender)i(memory)13 b(where)75 1354 y(from)h(the)i(message)g (data)f(is)h(tak)o(en.)23 b(In)16 b(addition,)f(the)h(send)h(op)q(eration)e (asso)q(ciates)i(an)f Fv(en)o(v)o(elop)q(e)d Fx(with)i(the)75 1404 y(message.)26 b(This)17 b(en)o(v)o(elop)q(e)g(sp)q(eci\014es)i(the)e (message)g(destination)f(and)h(con)o(tains)f(distinguishing)g(information)75 1454 y(that)e(can)g(b)q(e)h(used)f(b)o(y)g(the)g Fv(receiv)o(e)f Fx(op)q(eration)h(to)f(select)j(a)d(particular)h(message.)158 1503 y(A)g Fv(receiv)o(e)f Fx(op)q(eration)h(consumes)g(a)f(message.)18 b(The)d(message)f(to)f(b)q(e)i(receiv)o(ed)g(is)f(selected)i(according)e(to) 75 1553 y(the)g(v)n(alue)g(on)f(its)h(en)o(v)o(elop)q(e,)g(and)g(the)g (message)g(data)f(is)h(put)g(in)o(to)g(the)g Fv(receiv)o(e)h(bu\013er)p Fx(.)158 1603 y(The)g(next)h(sections)g(describ)q(e)h(the)f(basic)f(\(blo)q (c)o(king\))f(send)i(and)f(receiv)o(e)h(op)q(erations.)22 b(W)m(e)14 b(discuss)j(send,)75 1653 y(receiv)o(e,)12 b(basic)e(comm)o(unicatio)o(n)e (seman)o(tics,)i(t)o(yp)q(e)g(matc)o(hing)e(requiremen)o(ts,)j(t)o(yp)q(e)g (con)o(v)o(ersion)f(in)g(hetrogenous)75 1703 y(en)o(vironmen)o(ts,)16 b(and)g(more)f(general)h(comm)o(unication)d(mo)q(des.)24 b(Non)o(blo)q(c)o (king)15 b(comm)o(unication)e(is)j(addressed)75 1753 y(next,)k(follo)o(w)o (ed)d(b)o(y)i(c)o(hannel-lik)o(e)f(constructs)i(and)f(send-receiv)o(e)i(op)q (erations.)32 b(W)m(e)19 b(then)g(consider)h(general)75 1802 y(datat)o(yp)q(es)13 b(that)f(allo)o(w)e(to)i(transfer)h(heterogenous)h(and)e (noncon)o(tiguous)f(data,)h(and)g(conclude)h(with)e(a)h(descrip-)75 1852 y(tion)f(of)h(an)g(implem)o(en)o(tation)d(of)j(MPI)g(p)q(oin)o(t)f(to)h (p)q(oin)o(t)g(comm)o(unicatio)o(n)d(using)j(a)g(small)e(n)o(um)o(b)q(er)h (of)g(primitiv)o(es.)75 1987 y Fw(1.2)59 b(Basic)19 b(send)h(op)r(eration)75 2078 y Fx(The)14 b(syn)o(tax)g(of)f(the)i(simplest)e(send)i(op)q(eration)e (is)h(giv)o(en)g(b)q(elo)o(w.)75 2175 y Fu(MPI)p 144 2175 14 2 v 15 w(SEND)21 b(\(start,)f(count,)h(datatype,)f(dest,)h(tag,)g(comm\))117 2225 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 2267 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er) 12 b(of)f(elemen)o(ts)h(in)g(send)g(bu\013er)g(\(nonnegativ)o(e)i(in)o (teger\))117 2310 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 2352 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2395 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2437 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))75 2577 y Fs(1.2.1)49 b(Message)16 b(data)75 2654 y Fx(The)e(send)g(bu\013er)g (sp)q(eci\014ed)h(b)o(y)e(the)h Fu(MPI)p 736 2654 V 15 w(SEND)e Fx(op)q(eration)h(consists)h(of)f Fu(count)f Fx(successiv)o(e)k(en)o(tries)e (of)f(the)g(t)o(yp)q(e)75 2704 y(indicated)g(b)o(y)g Fu(datatype)p Fx(,)e(starting)i(with)g(the)h(en)o(try)g(at)f(address)h Fu(start)p Fx(.)j(Note)c(that)g(w)o(e)h(sp)q(ecify)g(the)f(message)-32 46 y Fr(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-39 554 y(10)-39 611 y(11)-39 667 y(12)-39 724 y(13)-39 780 y(14)-39 836 y(15)-39 893 y(16)-39 949 y(17)-39 1006 y(18)-39 1062 y(19)-39 1119 y(20)-39 1175 y(21)-39 1232 y(22)-39 1288 y(23)-39 1345 y(24)-39 1401 y(25)-39 1457 y(26)-39 1514 y(27)-39 1570 y(28)-39 1627 y(29)-39 1683 y(30)-39 1740 y(31)-39 1796 y(32)-39 1853 y(33)-39 1909 y(34)-39 1966 y(35)-39 2022 y(36)-39 2078 y(37)-39 2135 y(38)-39 2191 y(39)-39 2248 y(40)-39 2304 y(41)-39 2361 y(42)-39 2417 y(43)-39 2474 y(44)-39 2530 y(45)-39 2587 y(46)-39 2643 y(47)-39 2699 y(48)p eop %%Page: 2 2 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(2)75 45 y(length)17 b(in)f(terms)g(of)g(n)o(um)o(b)q(er)g(of)g Fp(elements)p Fx(,)h(not)f(n)o(um)o(b)q(er)g(of)g Fp(bytes)p Fx(.)26 b(The)17 b(former)f(is)g(mac)o(hine)g(indep)q(enden)o(t)75 95 y(and)e(closer)g(to)g(the)h(application)d(lev)o(el.)158 145 y(The)h(data)g(part)g(of)g(the)g(message)g(consists)h(of)e(a)h(sequence)i (of)e Fu(count)f Fx(v)n(alues,)g(eac)o(h)h(of)g(the)g(t)o(yp)q(e)h(indicated) 75 195 y(b)o(y)h Fu(datatype)p Fx(.)k Fu(count)14 b Fx(ma)o(y)f(b)q(e)j (zero,)f(in)g(whic)o(h)f(case)i(the)g(data)e(part)i(of)e(the)h(message)g(is)g (empt)o(y)m(.)k(The)d(basic)75 244 y(datat)o(yp)q(es)h(that)g(can)h(b)q(e)f (sp)q(eci\014ed)i(for)d(message)h(data)f(v)n(alues)h(corresp)q(ond)h(to)f (the)h(basic)f(datat)o(yp)q(es)g(of)f(the)75 294 y(host)i(language.)29 b(The)18 b(p)q(ossible)g(v)n(alues)g(of)f(this)h(parameter)f(for)h(F)m (ortran)f(and)h(the)g(corresp)q(onding)h(F)m(ortran)75 344 y(t)o(yp)q(es)c(are)f(listed)g(b)q(elo)o(w)p 158 447 906 2 v 157 497 2 50 v 183 482 a(MPI)g(datat)o(yp)q(e)p 587 497 V 175 w(F)m(ortran)g(datat)o(yp)q(e)p 1063 497 V 158 499 906 2 v 157 549 2 50 v 183 534 a Fu(MPI)p 252 534 14 2 v 15 w(INTEGER)p 587 549 2 50 v 195 w Fx(INTEGER)p 1063 549 V 157 598 V 183 583 a Fu(MPI)p 252 583 14 2 v 15 w(REAL)p 587 598 2 50 v 261 w Fx(REAL)p 1063 598 V 157 648 V 183 633 a Fu(MPI)p 252 633 14 2 v 15 w(DOUBLE)p 587 648 2 50 v 217 w Fx(DOUBLE)g(PRECISION)p 1063 648 V 157 698 V 183 683 a Fu(MPI)p 252 683 14 2 v 15 w(COMPLEX)p 587 698 2 50 v 195 w Fx(COMPLEX)p 1063 698 V 157 748 V 183 733 a Fu(MPI)p 252 733 14 2 v 15 w(DOUBLE)p 399 733 V 14 w(COMPLEX)p 587 748 2 50 v 49 w Fx(DOUBLE)g(COMPLEX)p 1063 748 V 157 798 V 183 783 a Fu(MPI)p 252 783 14 2 v 15 w(LOGICAL)p 587 798 2 50 v 195 w Fx(LOGICAL)p 1063 798 V 157 847 V 183 832 a Fu(MPI)p 252 832 14 2 v 15 w(CHARACTER)p 587 847 2 50 v 151 w Fx(CHARA)o(CTER)p 1063 847 V 157 897 V 183 882 a Fu(MPI)p 252 882 14 2 v 15 w(BYTE)p 587 897 2 50 v 1063 897 V 158 899 906 2 v 75 1042 a Fx(The)g(p)q(ossible)g(v) n(alues)g(for)g(this)g(parameter)f(for)h(C)f(and)h(the)h(corresp)q(onding)f (C)g(t)o(yp)q(es)h(are)f(listed)g(b)q(elo)o(w)p 75 1187 560 2 v 74 1237 2 50 v 100 1222 a(MPI)g(datat)o(yp)q(e)p 379 1237 V 50 w(C)g(datat)o(yp)q(e)p 633 1237 V 75 1239 560 2 v 74 1288 2 50 v 100 1274 a Fu(MPI)p 169 1274 14 2 v 15 w(SHORT)p 379 1288 2 50 v 114 w Fx(short)p 633 1288 V 74 1338 V 100 1323 a Fu(MPI)p 169 1323 14 2 v 15 w(INT)p 379 1338 2 50 v 158 w Fx(in)o(t)p 633 1338 V 74 1388 V 100 1373 a Fu(MPI)p 169 1373 14 2 v 15 w(LONG)p 379 1388 2 50 v 136 w Fx(long)p 633 1388 V 74 1438 V 100 1423 a Fu(MPI)p 169 1423 14 2 v 15 w(UNSIGNED)p 379 1438 2 50 v 48 w Fx(unsigned)p 633 1438 V 74 1488 V 100 1473 a Fu(MPI)p 169 1473 14 2 v 15 w(FLOAT)p 379 1488 2 50 v 114 w Fx(\015oat)p 633 1488 V 74 1538 V 100 1523 a Fu(MPI)p 169 1523 14 2 v 15 w(DOUBLE)p 379 1538 2 50 v 92 w Fx(double)p 633 1538 V 74 1587 V 100 1572 a Fu(MPI)p 169 1572 14 2 v 15 w(CHAR)p 379 1587 2 50 v 136 w Fx(c)o(har)p 633 1587 V 74 1637 V 100 1622 a Fu(MPI)p 169 1622 14 2 v 15 w(BYTE)p 379 1637 2 50 v 633 1637 V 75 1639 560 2 v 158 1761 a Fx(The)g(datat)o(yp)q(e)g Fu(MPI)p 486 1761 14 2 v 15 w(BYTE)f Fx(do)q(es)h(not)g(corresp)q(ond)h(to)e (a)g(F)m(ortran)h(or)f(C)h(datat)o(yp)q(e.)k(A)c(v)n(alue)f(of)g(t)o(yp)q(e)h Fu(BYTE)75 1811 y Fx(consists)20 b(of)e(8)g(binary)g(digits.)32 b(A)19 b(b)o(yte)g(is)g(unin)o(terpreted)h(and)f(is)f(di\013eren)o(t)i(from)d (a)h(c)o(haracter.)34 b(Di\013eren)o(t)75 1861 y(mac)o(hines)19 b(ma)o(y)g(ha)o(v)o(e)h(di\013eren)o(t)h(represen)o(tations)i(for)d(c)o (haracters,)j(or)d(ma)o(y)f(use)i(more)e(than)i(one)f(b)o(yte)h(to)75 1911 y(represen)o(t)16 b(c)o(haracters.)k(On)14 b(the)h(other)f(hand,)f(a)h (b)o(yte)g(has)g(the)h(same)e(binary)g(v)n(alues)h(on)g(all)e(mac)o(hines.) 158 2039 y Fo(Discussion:)158 2085 y Ft(Need)h(to)g(decide)h(whether)g(w)o(e) e(w)o(an)o(t)h(\(for)g(C\))158 2130 y(unsigned)i(c)o(har,)e(unsigned)i (short,)f(unsigned)h(long)158 2176 y(long)f(double)75 2373 y Fs(1.2.2)49 b(Message)16 b(envelop)q(e)75 2450 y Fx(In)c(addition)g(to)g (the)h(data)f(part,)g(messages)h(con)o(tain)f(information)d(that)k(can)f(b)q (e)h(used)h(to)e(distinguish)g(messages)75 2500 y(and)h(selectiv)o(ely)h (receiv)o(e)h(them.)i(This)c(information)d(is)k(con)o(tained)f(in)g(a)g (\014xed)h(n)o(um)o(b)q(er)e(of)h(\014xed-format)f(\014elds,)75 2549 y(whic)o(h)i(w)o(e)g(collectiv)o(ely)f(call)g(the)i Fv(message)g(en)o(v) o(elop)q(e)p Fx(.)h(These)f(\014elds)f(are)75 2625 y Fv(source)75 2704 y(destinatio)o(n)p eop %%Page: 3 3 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(3)75 45 y Fv(tag)75 128 y(con)o(text)158 211 y Fx(The)16 b(in)o(teger-v)n(alued)f(message)f(tag)h(is)g(sp)q(eci\014ed)i(b) o(y)e(the)h Fu(tag)e Fx(parameter)h(of)f(the)i(send)g(op)q(eration.)22 b(This)75 261 y(in)o(teger)16 b(can)g(b)q(e)h(used)f(b)o(y)g(the)g(program)e (to)i(distinguish)f(di\013eren)o(t)i(t)o(yp)q(es)f(of)f(messages.)24 b(The)16 b(range)g(of)g(v)n(alid)75 311 y(tag)d(v)n(alues)f(is)h(implemen)o (tatio)o(n)d(dep)q(enden)o(t)15 b(and)e(can)g(b)q(e)g(queried)h(using)f(an)f (en)o(vironmetal)g(inquiry)g(function,)75 361 y(as)i(describ)q(ed)i(in)d (Chapter)i Fv(??)p Fx(.)158 410 y(The)g(con)o(text)g(of)e(the)i(message)f (sen)o(t)h(is)g(sp)q(eci\014ed)g(b)o(y)f(the)h Fu(comm)f Fx(parameter)g(in)g (the)g(send)i(op)q(eration:)i(the)75 460 y(message)c(carries)g(the)h(con)o (text)g(asso)q(ciated)f(with)g(this)g(comm)o(unicator)d(\(see)k(Chapter)g Fv(??)p Fx(\).)158 510 y(The)i(message)f(destination)g(is)h(sp)q(eci\014ed)g (as)g(a)f(rank)g(within)g(the)h(pro)q(cess)h(group)f(asso)q(ciated)g(with)f (that)75 560 y(same)e(comm)o(unicator.)j(The)e(range)g(of)f(v)n(alid)f(v)n (alues)i(for)f(the)h Fu(dest)f Fx(parameter)g(is)h Fu(0,)21 b(...)43 b(,)22 b(n-1)p Fx(,)13 b(where)j Fu(n)75 610 y Fx(is)e(the)h(n)o(um) o(b)q(er)e(of)g(pro)q(cesses)k(in)c(this)i(group.)j(Th)o(us,)c(p)q(oin)o (t-to-p)q(oin)o(t)e(comm)o(unications)f(do)j(not)g(use)h(absolute)75 659 y(addresses,)h(but)e(only)f(relativ)o(e)g(ranks)i(within)e(a)g(group.)18 b(This)c(pro)o(vides)g(imp)q(ortan)o(t)e(mo)q(dularit)o(y)m(.)158 709 y(The)i(message)g(source)h(is)f(implicitly)d(sp)q(eci\014ed)k(b)o(y)f (the)g(iden)o(tit)o(y)f(of)h(the)g(message)g(sender.)158 759 y(The)g(message)g(en)o(v)o(elop)q(e)g(w)o(ould)f(normally)e(b)q(e)j(enco)q (ded)h(b)o(y)e(a)h(\014xed-length)g(message)f(header.)19 b(Ho)o(w)o(ev)o(er,) 75 809 y(the)d(actual)f(mec)o(hanism)e(used)j(to)f(asso)q(ciate)i(an)e(en)o (v)o(elop)q(e)g(with)h(a)f(message)g(is)g(implemen)o(tatio)o(n)e(dep)q(enden) o(t;)75 859 y(some)g(of)h(the)h(information)c(\(e.g.,)j Fv(source)f Fx(or)h Fv(destination)o Fx(\))e(ma)o(y)g(b)q(e)j(implicit,)c(and)j(need)i (not)e(b)q(e)h(explicitly)75 909 y(carried)g(b)o(y)e(a)h(message.)75 1046 y Fw(1.3)59 b(Basic)19 b(receive)g(op)r(eration)75 1137 y Fx(The)14 b(syn)o(tax)g(of)f(the)i(simplest)e(receiv)o(e)i(op)q(eration)f (is)g(giv)o(en)f(b)q(elo)o(w.)75 1234 y Fu(MPI)p 144 1234 14 2 v 15 w(RECV)21 b(\(start,)f(count,)h(datatype,)f(source,)g(tag,)h(comm,)g (status\))117 1284 y Ft(out)14 b Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i(bu\013er)g(\(c)o(hoice\))117 1329 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (receiv)o(e)g(bu\013er)g(\(in)o(teger\))117 1375 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)g (\(state\))117 1421 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o (teger\))117 1466 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\)) 117 1512 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))117 1558 y(out)g Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1634 y Fx(The)18 b(receiv)o(e)h(bu\013er)g(consists)g(of)e(the)h(storage)g(con)o (taining)f Fu(count)f Fx(consecutiv)o(e)j(elemen)o(ts)f(of)f(the)h(t)o(yp)q (e)75 1684 y(sp)q(eci\014ed)e(b)o(y)e Fu(datatype)p Fx(,)f(starting)h(at)h (address)g Fu(start)p Fx(.)k(The)c(length)g(of)f(the)h(receiv)o(ed)g(message) g(m)o(ust)e(b)q(e)i(less)75 1733 y(then)h(or)f(equal)g(the)g(length)g(of)g (the)h(receiv)o(e)g(bu\013er.)23 b(I.e.,)14 b(all)g(incoming)f(data)h(m)o (ust)g(\014t,)i(without)e(truncation,)75 1783 y(in)o(to)f(the)i(receiv)o(e)g (bu\013er.)158 1833 y(The)e(selection)g(of)f(a)g(message)h(b)o(y)f(a)g (receiv)o(e)i(op)q(eration)e(is)h(done)g(uniquely)e(according)i(to)f(the)i(v) n(alue)d(of)h(the)75 1883 y(message)g(en)o(v)o(elop)q(e.)18 b(The)13 b(receiv)o(e)h(op)q(eration)e(sp)q(eci\014es)i(an)e Fv(en)o(v)o(elop)q(e)g(pattern)p Fx(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o (ed)75 1933 y(b)o(y)e(that)h(receiv)o(e)g(op)q(eration)g(only)e(if)h(its)g (en)o(v)o(elop)q(e)h(matc)o(hes)f(that)h(pattern.)18 b(A)12 b(pattern)g(sp)q(eci\014es)h(v)n(alues)e(for)g(the)75 1983 y Fu(source)p Fx(,)f Fu(tag)h Fx(and)h Fu(context)e Fx(\014elds)i(of)f(the)h (message)f(en)o(v)o(elop)q(e.)18 b(In)11 b(addition,)g(the)h(v)n(alue)f(for)g (the)h Fu(destination)75 2032 y Fx(\014eld)j(is)g(set,)h(implicitl)o(y)m(,)c (to)i(b)q(e)i(equal)f(to)g(the)g(receiving)g(pro)q(cess)i(id.)k(The)16 b(receiv)o(er)g(ma)o(y)d(sp)q(ecify)j(a)e(wildcard)75 2082 y Fu(MPI)p 144 2082 V 15 w(ANY)p 225 2082 V 15 w(SOURCE)d Fx(v)n(alue)g(for)g Fu(source)p Fx(,)g(and/or)g(a)h(wildcard)f Fu(MPI)p 1105 2082 V 15 w(ANY)p 1186 2082 V 15 w(TAG)g Fx(v)n(alue)g(for)h Fu(tag)p Fx(,)f(indicating)g(that)g(an)o(y)75 2132 y(source)k(and/or)f(tag)g(are)g (acceptable.)19 b(It)c(cannot)f(sp)q(ecify)g(a)g(wildcard)g(v)n(alue)f(for)h Fu(context)p Fx(.)j(Th)o(us,)d(a)f(message)75 2182 y(can)i(b)q(e)h(receiv)o (ed)g(b)o(y)f(a)f(receiv)o(e)j(op)q(eration)d(only)g(if)h(it)f(is)h (addressed)i(to)d(the)i(receiving)f(task,)g(has)g(a)g(matc)o(hing)75 2232 y(con)o(text,)e(has)f(matc)o(hing)e(source)j(unless)g(source=)p Fu(MPI)p 943 2232 V 17 w(ANY)p 1026 2232 V 15 w(SOURCE)e Fx(in)g(the)i (pattern,)f(and)g(has)g(a)g(matc)o(hing)e(tag)75 2281 y(unless)15 b(tag=)p Fu(MPI)p 357 2281 V 15 w(ANY)p 438 2281 V 15 w(TAG)e Fx(in)g(the)i(pattern.)158 2331 y(The)d(receiving)f(tag)g(v)n(alue)f(is)h(sp) q(eci\014ed)i(b)o(y)e(the)g Fu(tag)g Fx(parameter)g(of)f(the)i(receiv)o(e)g (op)q(eration.)17 b(The)12 b(receiving)75 2381 y Fu(context)g Fx(v)n(alue)h(is)g(the)h(con)o(text)g(attac)o(hed)g(with)f(the)h(comm)o (uniator)c(sp)q(eci\014ed)15 b(b)o(y)e(the)h(parameter)g Fu(comm)p Fx(.)j(The)75 2431 y(message)g(source,)h(if)e(di\013eren)o(t)i(from)d Fu(MPI)p 758 2431 V 15 w(ANY)p 839 2431 V 16 w(SOURCE)p Fx(,)g(is)i(sp)q (eci\014ed)h(as)f(a)g(rank)g(within)f(the)h(pro)q(cess)i(group)75 2481 y(asso)q(ciated)14 b(with)f(that)g(same)f(comm)o(unicator.)j(Th)o(us,)e (the)h(range)f(of)g(v)n(alid)e(v)n(alues)i(for)g(the)g Fu(source)f Fx(parameter)75 2531 y(is)i Fn(f)21 b Fu(0,)h(...)43 b(,)21 b(n-1)g Fn(g)h([f)f Fu(MPI)p 646 2531 V 15 w(ANY)p 727 2531 V 15 w(SOURCE)g Fn(g)p Fx(,)13 b(where)i Fu(n)f Fx(is)f(the)i(n)o(um)o(b)q (er)e(of)g(pro)q(cesses)k(in)c(this)h(group.)158 2580 y(Source)h(=)f (destination)g(is)f(allo)o(w)o(ed:)k(a)d(pro)q(cess)i(can)e(send)g(a)g (message)g(to)f(itself.)p eop %%Page: 4 4 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(4)75 45 y Fs(1.3.1)49 b(Return)16 b(status)75 204 y Fo(Discussion:)i(This)d(is)g(a)g(c)o(hange)g(from)e(previous)j(draft) 158 337 y Fx(The)d(source)h(or)f(the)g(tag)f(of)h(a)f(receiv)o(ed)i(message)e (ma)o(y)f(not)i(b)q(e)g(kno)o(wn)f(if)g(wildcard)h(v)n(alues)f(where)i(used)f (in)75 387 y(the)g(receiv)o(e)g(op)q(eration.)k(Also,)12 b(the)g(actual)g (length)g(of)f(the)i(message)f(receiv)o(ed)h(ma)o(y)d(not)i(b)q(e)h(kno)o (wn.)k(Th)o(us,)12 b(this)75 437 y(information)d(need)k(b)q(e)g(returned)g(b) o(y)f(the)h(receiv)o(e)g(op)q(eration.)k(F)m(or)12 b(reasons)h(that)f(will)e (b)q(ecome)i(clear)h(later,)e(it)h(is)75 486 y(preferable)j(to)g(return)g (this)g(information)c(in)j(a)h(separate)g Fu(out)f Fx(v)n(ariable,)f(rather)i (then)h(using)e(the)h Fu(count)p Fx(,)e Fu(dest)75 536 y Fx(and)k Fu(tag)f Fx(parameters)i(b)q(oth)f(for)g(input)f(and)h(output.)28 b(This)17 b(return)i(status)e(information)e(is)i(carried)g(b)o(y)g(the)75 586 y Fu(status)d Fx(parameter)h(of)g Fu(MPI)p 539 586 14 2 v 15 w(RECEIVE)p Fx(.)e(This)i(is)g(a)g(parameter)g(of)g(a)g(sp)q(ecial)g (MPI-de\014ned)i(t)o(yp)q(e.)22 b(The)16 b(status)75 636 y(v)n(ariable)f(can) i(b)q(e)g(\\deco)q(ded")g(to)g(retriev)o(e)g(the)g(information)d(returned)k (in)e(it)g(b)o(y)g(the)h(last)f(receiv)o(e)i(op)q(eration)75 686 y(that)c(up)q(dated)h(it,)e(using)g(the)i(query)f(functions)g(listed)g(b) q(elo)o(w.)75 783 y Fu(MPI)p 144 783 V 15 w(GET)p 225 783 V 15 w(SOURCE\(status,)19 b(source\))117 832 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i(op)q(eration)h(\(Status\))117 878 y(out)f Fu(source)629 b Ft(source)13 b(rank)h(\(in)o(teger\))158 954 y Fx(Returns)i(the)g(rank)g(of)f(the)h(message)f(source)h(\(in)g(the)g (group)f(asso)q(ciated)h(with)f(the)h(comm)o(unicator)d(that)75 1004 y(w)o(as)h(used)h(to)e(receiv)o(e\).)75 1101 y Fu(MPI)p 144 1101 V 15 w(GET)p 225 1101 V 15 w(TAG\(status,)20 b(tag\))117 1151 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i (op)q(eration)h(\(Status\))117 1197 y(out)f Fu(tag)695 b Ft(message)13 b(tag)h(\(in)o(teger\))158 1273 y Fx(Returns)h(the)f(tag)g(of)f(receiv)o(ed)i (message.)75 1370 y Fu(MPI)p 144 1370 V 15 w(GET)p 225 1370 V 15 w(LEN\(status,)20 b(len\))117 1420 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i(op)q(eration)h(\(Status\))117 1466 y(out)f Fu(len)695 b Ft(n)o(um)o(b)q(er)14 b(of)e(receiv)o(ed)j(elemen)o (ts)f(\(in)o(teger\))158 1542 y Fx(Returns)h(the)f(n)o(um)o(b)q(er)f(of)h (elemen)o(ts)f(receiv)o(ed.)20 b(\(Here,)15 b(again,)d(w)o(e)i(coun)o(t)g Fp(elements)p Fx(,)f(not)h Fp(bytes)p Fx(.\))158 1592 y(Note)h(that)g(it)f (is)g(not)g(mandatory)f(to)i(query)g(the)g(return)h(status)f(after)g(a)f (receiv)o(e.)21 b(The)15 b(receiv)o(er)h(will)d(use)75 1642 y(these)i(calls,)e(or)h(some)f(of)g(them,)g(only)g(when)i(the)f(information)d (they)j(return)h(is)f(needed.)158 1770 y Fo(Discussion:)158 1816 y Ft(The)f(use)g(of)g(a)g(separate)h(status)f(parameter)h(prev)o(en)o (ts)g(errors)f(that)g(are)g(often)g(attac)o(hed)h(with)f Fm(inout)e Ft(parameters)75 1861 y(\(e.g.,)h(passing)k(the)d Fm(MPI)p 435 1861 12 2 v 13 w(ANY)p 508 1861 V 13 w(TAG)f Ft(constan)o(t)i(as)g(the)f (actual)i(argumen)o(t)f(for)f Fm(tag)p Ft(\).)k(The)c(use)h(of)f(an)h (explicit)i(user)e(ob)r(ject)75 1907 y(b)q(ecomes)j(imp)q(ortan)o(t)g(for)e (non)o(blo)q(c)o(king)k(comm)o(unication)g(as)d(it)g(allo)o(ws)h(the)f (receiv)o(e)g(op)q(eration)i(to)e(b)q(e)g(stateless)g(and,)75 1953 y(hence,)d(reen)o(tran)o(t.)18 b(This)c(prev)o(en)o(ts)f(confusions)i (in)f(the)f(cast)g(where)g(m)o(ultiple)j(receiv)o(es)e(can)f(b)q(e)h(p)q (osted)g(b)o(y)f(a)g(pro)q(cess.)158 2164 y Fo(Implemen)o(tation)g(note:)158 2213 y Ft(One)g(exp)q(ects)h(deco)q(de)g(functions)g(to)f(b)q(e)h(inlined)h (in)f(man)o(y)g(implemen)o(tations.)158 2346 y Fx(All)f(send)h(and)g(receiv)o (e)h(op)q(erations)e(use)i(the)f Fu(start)p Fx(,)e Fu(count)p Fx(,)g Fu(datatype)p Fx(,)g Fu(source)p Fx(,)g Fu(dest)p Fx(,)g Fu(tag)p Fx(,)g Fu(comm)h Fx(and)75 2396 y Fu(status)d Fx(parameters)h(in)g (the)h(same)e(w)o(a)o(y)h(as)g(the)h(basic)f Fu(MPI)p 1014 2396 14 2 v 15 w(SEND)f Fx(and)h Fu(MPI)p 1271 2396 V 15 w(RECV)g Fx(op)q(erations)g(describ)q(ed)i(in)e(this)75 2445 y(section.)18 b(The)12 b(additional)d(op)q(erations)j(pro)o(vide)f(mec)o(hanisms)e(to)i (con)o(trol)g(more)f(precisely)j(the)e(comm)o(unication)75 2495 y(proto)q(col.)p eop %%Page: 5 5 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(5)75 45 y Fw(1.4)59 b(Semantic)o(s)17 b(of)j(p)r(oint-to-p)r (oint)e(comm)m(unicat)o(ion)75 136 y Fx(A)12 b(v)n(alid)e(MPI)i(implem)o(en)o (tation)d(guaran)o(tees)j(certain)h(general)e(prop)q(erties)j(of)d(p)q(oin)o (t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n,)75 186 y(whic)o(h)k(are)g(describ)q (ed)i(in)d(this)h(section.)75 294 y Fl(Order)42 b Fx(Messages)15 b(are)e Fp(nonovertaking)p Fx(,)g(within)f(eac)o(h)h(con)o(text:)19 b(if)12 b(t)o(w)o(o)g(messages)h(are)g(successiv)o(ely)h(sen)o(t)g(from)75 344 y(the)g(same)f(source,)i(to)f(the)g(same)f(destination,)g(with)h(the)g (same)f(con)o(text,)h(then)h(the)f(messages)g(are)g(receiv)o(ed)h(in)75 393 y(the)h(order)h(they)f(w)o(ere)h(sen)o(t.)25 b(I.e.,)15 b(if)g(the)i(receiv)o(er)g(p)q(osts)g(t)o(w)o(o)e(successiv)o(e)j(receiv)o (es)g(that)d(b)q(oth)h(matc)o(h)f(either)75 443 y(message,)k(then)h(the)f (\014rst)g(receiv)o(e)h(will)e(receiv)o(e)i(the)f(\014rst)h(message,)f(and)f (the)i(second)f(receiv)o(e)h(will)e(receiv)o(e)75 493 y(the)e(second)g (message.)21 b(This)14 b(requiremen)o(t)h(facilitates)g(matc)o(hing)e(of)h (sends)i(to)f(receiv)o(es.)23 b(It)15 b(guaran)o(tees)h(that)75 543 y(message-passing)11 b(co)q(de)i(is)e(deterministic,)g(if)g(pro)q(cesses) k(are)d(single-threaded)g(and)g(wildcard)f Fu(MPI)p 1651 543 14 2 v 15 w(ANY)p 1732 543 V 15 w(SOURCE)75 593 y Fx(is)j(not)g(used)g(in)g (receiv)o(es.)158 642 y(If)i(a)h(pro)q(cess)h(has)f(a)f(single)h(thread)g(of) f(execution,)i(then)f(an)o(y)f(t)o(w)o(o)g(comm)o(unications)e(executed)k(b)o (y)f(this)75 692 y(pro)q(cess)k(are)e(ordered.)35 b(On)20 b(the)f(other)h (hand,)g(if)e(the)i(pro)q(cess)h(is)e(m)o(ultithreaded,)f(then)i(the)g(seman) o(tics)f(of)75 742 y(thread)h(execution)g(ma)o(y)e(not)h(de\014ne)i(a)e (relativ)o(e)g(order)i(b)q(et)o(w)o(een)g(t)o(w)o(o)e(send)h(op)q(erations)g (executed)h(b)o(y)e(t)o(w)o(o)75 792 y(distinct)e(threads:)24 b(the)17 b(op)q(erations)g(are)g(logically)d(concurren)o(t,)k(ev)o(en)f (though)f(one)h(ph)o(ysically)e(precedes)k(the)75 842 y(other.)26 b(In)16 b(suc)o(h)h(case,)g(the)g(t)o(w)o(o)f(messages)g(sen)o(t)h(can)g(b)q (e)g(receiv)o(ed)g(in)f(an)o(y)g(order.)26 b(Similarly)l(,)13 b(if)j(t)o(w)o(o)g(receiv)o(e)75 891 y(op)q(erations)10 b(that)g(are)g (logically)e(concurren)o(t)j(receiv)o(e)h(t)o(w)o(o)d(successiv)o(ely)j(sen)o (t)e(messages,)g(then)h(the)f(t)o(w)o(o)g(messages)75 941 y(can)k(matc)o(h)f (the)h(t)o(w)o(o)g(receiv)o(es)h(in)f(either)g(order.)75 1049 y Fl(Progress)41 b Fx(If)12 b(a)h(pair)f(of)g(matc)o(hing)e(send)k(and)e (receiv)o(es)i(ha)o(v)o(e)f(b)q(een)g(initiated)f(on)g(t)o(w)o(o)g(pro)q (cesses,)j(then)e(at)g(least)75 1099 y(one)j(of)f(these)j(t)o(w)o(o)d(op)q (erations)h(will)f(complete,)g(indep)q(enden)o(tly)i(of)e(other)i(actions)f (in)f(the)i(system:)22 b(the)16 b(send)75 1149 y(op)q(eration)11 b(will)e(complete,)h(unless)i(the)f(receiv)o(e)h(is)f(satis\014ed)g(b)o(y)g (another)g(message,)g(and)f(completes;)h(the)h(receiv)o(e)75 1199 y(op)q(eration)h(will)f(complete,)g(unless)i(the)g(message)f(sen)o(t)i (is)e(consumed)g(b)o(y)g(another)h(matc)o(hing)d(receiv)o(e)k(that)e(w)o(as) 75 1248 y(p)q(osted)i(at)f(the)g(same)f(destination)h(pro)q(cess.)75 1356 y Fl(F)o(airness)42 b Fx(MPI)16 b(mak)o(es)f(no)h(guaran)o(tee)h(of)e Fp(fairness)h Fx(in)g(the)g(handling)f(of)h(comm)o(unicati)o(on.)22 b(Supp)q(ose)17 b(that)f(a)75 1406 y(send)h(w)o(as)f(p)q(osted.)25 b(Then)16 b(it)g(is)f(p)q(ossible)i(that)e(the)i(destination)f(rep)q(eatedly) h(p)q(osts)g(a)e(receiv)o(e)i(that)f(matc)o(hes)75 1456 y(this)e(send,)h(y)o (et)g(the)g(message)f(is)h(nev)o(er)g(receiv)o(ed,)h(b)q(ecause)g(it)e(is)g (eac)o(h)h(time)e(o)o(v)o(ertak)o(en)i(b)o(y)f(another)h(message,)75 1506 y(sen)o(t)j(from)e(another)h(source.)30 b(Similarly)l(,)15 b(supp)q(ose)j(that)g(a)f(receiv)o(e)h(w)o(as)f(p)q(osted.)30 b(Then)17 b(it)g(is)g(p)q(ossible)h(that)75 1556 y(messages)12 b(that)f(matc)o(h)g(this)g(receiv)o(e)i(are)f(rep)q(eatedly)h(receiv)o(ed,)g (y)o(et)f(the)g(receiv)o(e)h(is)e(nev)o(er)i(satis\014ed,)f(b)q(ecause)h(it) 75 1605 y(is)e(o)o(v)o(ertak)o(en)g(b)o(y)g(other)h(receiv)o(es)h(p)q(osted)f (at)f(this)g(no)q(de)h(\(b)o(y)f(other)h(executing)g(threads\).)18 b(It)11 b(is)g(the)h(programmer)75 1655 y(resp)q(onsibilit)o(y)h(to)h(prev)o (en)o(t)h(starv)n(ation)e(in)h(suc)o(h)g(situations.)158 1784 y Fo(Discussion:)158 1829 y(Need)g(to)h(b)q(e)g(discussed)g(b)o(y)g(MPIF)158 1875 y Ft(Alternativ)o(e)g(p)q(ositions:)158 1921 y(1.)25 b(Sends)17 b(are)f(handled)i(fairly;)g(e.g.)25 b(a)16 b(p)q(osted)g(send)h(cannot)f(b)q (e)g(rep)q(eatedly)i(o)o(v)o(ertak)o(en)f(b)o(y)f(other)g(sends.)26 b(An)75 1966 y(example)16 b(of)f(an)g(implemen)o(tation)k(that)c(violates)i (this:)k(Assume)16 b(that)f(arriving)i(messages)f(are)f(k)o(ept)g(in)h(a)f (hash)h(table)75 2012 y(organized)d(b)o(y)e(source,)h(and)f(that)h(a)f (receiv)o(e)h(with)f(source=don)o(tcare)i(is)e(handled)i(b)o(y)e(searc)o (hing)i(this)f(table)g(sequen)o(tially)n(.)75 2058 y(Then)j(a)f(message)h(p)q (osted)h(at)e(the)h(end)g(of)f(this)h(table)h(ma)o(y)e(b)q(e)h(rep)q(eatedly) h(o)o(v)o(ertak)o(en)g(b)o(y)f(messages)g(that)f(get)h(p)q(osted)75 2103 y(at)e(the)g(head)h(of)f(the)g(table.)18 b(\(Easy)13 b(\014x:)k(rotate)d (starting)g(p)q(oin)o(t)g(of)f(searc)o(h.\))158 2149 y(2.)18 b(Receiv)o(es)d(are)e(handled)j(fairly;)e(e.g.)k(a)13 b(p)q(osted)i(receiv)o (e)f(cannot)g(b)q(e)g(rep)q(eatedly)h(o)o(v)o(ertak)o(en)g(b)o(y)e(other)h (receiv)o(es.)75 2195 y(An)j(example)i(of)d(an)i(implemen)o(tation)i(that)e (violates)h(this:)26 b(assume)18 b(that)f(there)g(are)g(t)o(w)o(o)g (di\013eren)o(t)i(p)q(osting)g(tables)75 2240 y(for)e(receiv)o(es)h(with)f(a) g(sp)q(eci\014c)i(source)f(and)f(for)g(receiv)o(es)h(with)g(source=don)o (tcare;)i(assume)d(further)h(that)f(these)g(t)o(w)o(o)75 2286 y(structures)i(are)f(searc)o(hed)h(in)g(a)f(\014xed)h(order)g(when)f(a)h (message)f(arriv)o(es.)34 b(Then)18 b(a)g(receiv)o(e)i(with)e(source=don)o (tcare)75 2332 y(ma)o(y)e(b)q(e)f(rep)q(eatedly)i(o)o(v)o(ertak)o(en)g(b)o(y) e(receiv)o(es)i(with)f(source)g(sp)q(eci\014ed.)26 b(\(Fix:)c(alternate)17 b(searc)o(h)f(order;)g(but)g(this)g(has)75 2377 y(p)q(erformance)e(problems,) g(if)f(w)o(e)g(assume)h(that)f(one)g(t)o(yp)q(e)h(of)e(receiv)o(e)i(is)g(m)o (uc)o(h)f(more)g(frequen)o(t)h(than)g(the)f(other)g(t)o(yp)q(e.\))158 2423 y(3.)k(Both)c(send)h(and)g(receiv)o(es)g(are)f(handled)i(fairly)m(.)158 2473 y(Oppinions?)p eop %%Page: 6 6 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(6)75 45 y Fl(Resource)18 b(lim)o(itations)39 b Fx(There)17 b(is)f(no)g(requiremen)o(t)g(that)h(MPI)f(bu\013ers)i(message)e (data.)24 b(Th)o(us,)17 b(a)f(v)n(alid)f(MPI)75 95 y(implemen)o(tatio)o(n)10 b(of)h Fu(MPI)p 484 95 14 2 v 15 w(SEND)h Fx(is)g(to)g(blo)q(c)o(k)g(the)h (sender)h(un)o(til)d(a)h(matc)o(hing)e(receiv)o(e)k(has)e(b)q(een)i (initiated.)i(Ev)o(en)75 145 y(if)c(messages)g(are)h(bu\013ered,)h(bu\013er)g (space)f(can)g(b)q(e)g(exhausted.)19 b(It)12 b(is)h(v)n(alid)e(for)h(MPI)h (to)f(bu\013er)i(some)d(messages,)75 195 y(and)17 b(not)f(to)h(bu\013er)h (some)e(other,)h(blo)q(c)o(king)f(the)h(sender)i(un)o(til)d(a)g(matc)o(hing)f (reciev)o(e)j(is)f(p)q(osted.)27 b(In)17 b(general,)75 244 y(the)g(programmer)d(can)j(mak)o(e)e(no)h(assumptions)g(on)g(the)h(a)o(v)n (ailabilit)o(y)c(of)j(bu\013er)h(space,)h(and)e(ho)o(w)g(this)h(space)75 294 y(is)e(allo)q(cated.)23 b(Th)o(us,)15 b(p)q(ortable)h(\(safe\))g(MPI)g (co)q(de)g(should)f(w)o(ork)h(under)g(the)g(assumption)e(that)i(an)f (arbitrary)75 344 y(subset)21 b(of)f(the)g(send)h(op)q(erations)f(are)h (going)d(to)i(return)h(b)q(efore)g(a)f(matc)o(hing)e(receiv)o(e)j(is)f(p)q (osted,)i(and)d(the)75 394 y(remaining)12 b(will)g(blo)q(c)o(k)i(un)o(til)f (a)g(matc)o(hing)f(receiv)o(e)k(is)d(p)q(osted.)158 444 y(Ho)o(w)o(ev)o(er,)g (MPI)f(implemen)o(tation)d(will)i(pro)o(vide)h(information)d(on)j(the)h (amoun)o(t)e(of)h(a)o(v)n(ailable)e(bu\013er)j(space)75 493 y(and)h(on)f(the)i(bu\013ering)f(p)q(olicy)f(via)g(the)i(en)o(vironmen)o(tal) d(enquiries)i(describ)q(ed)i(in)e(Chapter)g Fv(??)p Fx(.)158 543 y(Examples)f(\(in)o(v)o(olving)f(t)o(w)o(o)h(pro)q(cessors)j(with)e (ranks)g(0)f(and)h(1\))158 593 y(The)g(follo)o(wing)e(program)g(is)i(safe,)f (and)h(should)g(alw)o(a)o(ys)f(succeed.)75 718 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 768 y(IF)g(\(rank.EQ.0\))f (THEN)140 818 y(CALL)h(MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(1,)h(tag,)g (comm,)g(ierr\))140 868 y(CALL)g(MPI_RECV\(recvbuf,)e(count,)h(MPI_REAL,)g (1,)h(tag,)g(comm,)g(ierr\))75 918 y(ELSE)86 b(!)22 b(rank.EQ.1)140 968 y(CALL)f(MPI_RECV\(recvbuf,)e(count,)h(MPI_REAL,)g(0,)h(tag,)g(comm,)g (ierr\))140 1017 y(CALL)g(MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(0,)h (tag,)g(comm,)g(context\))75 1067 y(END)g(IF)158 1143 y Fx(The)14 b(follo)o(wing)e(program)g(is)i(erroneous,)g(and)g(should)g(alw)o(a)o(ys)f (fail.)75 1268 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1318 y(IF)g(\(rank.EQ.0\))f(THEN)140 1368 y(CALL)h(MPI_RECV\(recvbuf,)e (count,)h(MPI_REAL,)g(1,)h(tag,)g(comm,)g(ierr\))140 1418 y(CALL)g (MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(1,)h(tag,)g(comm,)g(ierr\))75 1468 y(ELSE)86 b(!)22 b(rank.EQ.1)140 1517 y(CALL)f(MPI_RECV\(recvbuf,)e (count,)h(MPI_REAL,)g(0,)h(tag,)g(comm,)g(ierr\))140 1567 y(CALL)g (MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(0,)h(tag,)g(comm,)g(ierr\))75 1617 y(END)g(IF)158 1693 y Fx(The)13 b(receiv)o(e)h(op)q(eration)f(of)f(the)i (\014rst)f(pro)q(cess)i(m)o(ust)d(complete)g(b)q(efore)i(its)e(send,)i(and)f (can)g(complete)f(only)75 1742 y(if)g(the)h(matc)o(hing)d(send)k(of)e(the)h (second)g(pro)q(cessor)i(is)d(executed;)i(the)f(receiv)o(e)h(op)q(eration)f (of)e(the)j(second)f(pro)q(cess)75 1792 y(m)o(ust)j(complete)h(b)q(efore)h (its)g(send)g(and)f(can)h(complete)e(only)h(if)f(the)i(matc)o(hing)e(send)i (of)f(the)h(\014rst)g(pro)q(cess)h(is)75 1842 y(executed.)h(This)14 b(program)e(will)g(deadlo)q(c)o(k.)158 1892 y(The)i(follo)o(wing)e(program)g (is)i(unsafe,)f(and)h(ma)o(y)e(succeed)k(or)e(fail,)e(dep)q(ending)i(on)g (implemen)o(tatio)o(n.)75 2024 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e (rank,)i(ierr\))75 2074 y(IF)g(\(rank.EQ.0\))f(THEN)140 2123 y(CALL)h(MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(1,)h(tag,)g(comm,)g (ierr\))140 2173 y(CALL)g(MPI_RECV\(recvbuf,)e(count,)h(MPI_REAL,)g(1,)h (tag,)g(comm,)g(ierr\))75 2223 y(ELSE)86 b(!)22 b(rank.EQ.1)140 2273 y(CALL)f(MPI_SEND\(sendbuf,)e(count,)h(MPI_REAL,)g(0,)h(tag,)g(comm,)g (ierr\))140 2323 y(CALL)g(MPI_RECV\(recvbuf,)e(count,)h(MPI_REAL,)g(0,)h (tag,)g(comm,)g(ierr\))75 2372 y(END)g(IF)158 2504 y Fx(The)15 b(message)g(sen)o(t)h(b)o(y)f(eac)o(h)g(pro)q(cess)i(has)e(to)g(b)q(e)h (copied)f(out)g(b)q(efore)h(the)g(send)g(op)q(eration)f(returns)h(and)75 2554 y(the)h(receiv)o(e)g(op)q(eration)e(starts.)26 b(F)m(or)15 b(the)i(program)d(to)i(complete,)f(it)g(is)h(necessary)i(that)e(at)g(least)g (one)g(of)f(the)75 2604 y(t)o(w)o(o)d(messages)g(sen)o(t)h(is)f(bu\013ered)i (out)e(of)f(either)i(pro)q(cesses')h(address)g(space.)k(Th)o(us,)13 b(this)f(program)e(can)j(succeed)75 2654 y(only)g(if)f(the)i(comm)o(unicatio) o(n)d(system)i(has)g(su\016cien)o(t)h(bu\013er)g(space)h(to)e(bu\013er)h Fu(count)e Fx(w)o(ords)i(of)f(data,)f(and)h(the)75 2704 y(MPI)h(implemen)o (tation)c(bu\013ers)16 b(messages.)p eop %%Page: 7 7 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(7)75 45 y Fw(1.5)59 b(T)-5 b(yp)r(e)18 b(Matching)75 136 y Fx(One)d(can)f(think)f(of)h(message)f(transmission)g(as)h(consisting)g (of)f(three)i(phases:)126 207 y(1.)20 b(Data)13 b(is)h(pulled)f(out)h(of)f (the)i(send)g(bu\013er)f(and)g(a)g(message)f(is)h(assem)o(bled)126 283 y(2.)20 b(A)14 b(message)f(is)h(transferred)i(from)c(sender)j(to)f (receiv)o(er)126 360 y(3.)20 b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming) c(message)j(and)g(disassem)o(bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 431 y(T)o(yp)q(e)g(matc)o(hing)e(has)i(to)f(b)q(e)i(observ)o(ed)f(at)g(eac)o (h)g(of)f(these)i(three)g(phases:)21 b(The)15 b(t)o(yp)q(e)h(of)e(eac)o(h)h (v)n(ariable)e(in)75 481 y(the)18 b(sender)h(bu\013er)g(has)e(to)h(matc)o(h)e (the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(for)e(that)h(en)o(try)g(b)o(y)f(the)h (send)h(op)q(eration;)f(the)g(t)o(yp)q(e)75 530 y(sp)q(eci\014ed)g(b)o(y)e (the)h(send)g(op)q(eration)f(has)g(to)g(matc)o(h)g(the)g(t)o(yp)q(e)h(sp)q (eci\014ed)h(b)o(y)e(the)h(receiv)o(e)g(op)q(eration;)g(and)f(the)75 580 y(t)o(yp)q(e)j(of)e(eac)o(h)i(v)n(ariable)e(in)g(the)i(receiv)o(e)g (bu\013er)g(has)f(to)g(matc)o(h)f(the)i(t)o(yp)q(e)f(sp)q(eci\014ed)i(for)e (that)g(en)o(try)h(b)o(y)e(the)75 630 y(receiv)o(e)e(op)q(eration.)j(A)c (program)e(that)i(fails)f(to)h(observ)o(e)h(these)g(three)g(rules)g(is)e (erroneous.)158 680 y(T)m(o)j(de\014ne)j(t)o(yp)q(e)e(matc)o(hing)f(more)g (precisely)m(,)i(w)o(e)g(need)g(to)f(deal)g(with)g(t)o(w)o(o)f(issues:)26 b(matc)o(hing)15 b(of)i(t)o(yp)q(es)75 730 y(of)f(the)h(host)g(language)f (with)g(t)o(yp)q(es)h(sp)q(eci\014ed)i(in)d(comm)o(unicati)o(on)e(op)q (erations;)k(and)e(matc)o(hing)f(of)h(t)o(yp)q(es)h(at)75 779 y(sender)e(and)f(receiv)o(er.)158 829 y(A)k(t)o(yp)q(e)h(sp)q(eci\014ed)h (for)d(an)h(en)o(try)h(b)o(y)f(a)g(send)h(op)q(eration)f(matc)o(hes)f(the)i (t)o(yp)q(e)g(sp)q(eci\014ed)g(for)f(that)g(en)o(try)75 879 y(b)o(y)d(a)f(receiv)o(e)i(op)q(eration)f(if)f(b)q(oth)h(op)q(erations)g(use) h(iden)o(tical)e(names:)19 b Fu(MPI)p 1289 879 14 2 v 15 w(INTEGER)13 b Fx(matc)o(hes)i Fu(MPI)p 1699 879 V 15 w(INTEGER)p Fx(,)75 929 y Fu(MPI)p 144 929 V 15 w(REAL)e Fx(matc)o(hes)h Fu(MPI)p 487 929 V 15 w(REAL)p Fx(,)e(and)i(so)g(on.)158 979 y(The)19 b(t)o(yp)q(e)g(of)g(a)f(v)n(ariable)g(in)g(a)g(host)h(program)f(matc)o(hes)g (the)h(t)o(yp)q(e)h(sp)q(eci\014ed)g(in)e(the)h(comm)o(unication)75 1029 y(op)q(eration)e(if)f(the)i(datat)o(yp)q(e)f(name)e(used)j(b)o(y)f(that) g(op)q(eration)g(corresp)q(onds)i(to)e(the)g(basic)g(t)o(yp)q(e)h(of)e(the)i (host)75 1078 y(program)8 b(v)n(ariable:)15 b(an)9 b(en)o(try)i(with)e(t)o (yp)q(e)h(name)f Fu(MPI)p 920 1078 V 15 w(INTEGER)f Fx(matc)o(hes)h(a)h(F)m (ortran)f(v)n(ariable)g(of)g(t)o(yp)q(e)h Fu(INTEGER)p Fx(,)75 1128 y(an)i(en)o(try)h(with)e(t)o(yp)q(e)i(name)e Fu(MPI)p 600 1128 V 15 w(REAL)g Fx(matc)o(hes)h(a)g(F)m(ortran)f(v)n(ariable)g(of)h(t) o(yp)q(e)h Fu(REAL)p Fx(,)d(and)i(so)g(on.)18 b(There)13 b(is)f(one)75 1178 y(exception)17 b(to)f(this)g(last)f(rule:)23 b(An)16 b(en)o(try)g(with)g (t)o(yp)q(e)h(name)d Fu(MPI)p 1137 1178 V 15 w(BYTE)i Fx(can)g(b)q(e)g(used)h (to)f(matc)o(h)f(an)o(y)g(b)o(yte)i(of)75 1228 y(storage)f(\(on)f(a)g(b)o (yte-addressable)h(mac)o(hine\),)e(irresp)q(ectiv)o(e)j(of)d(the)i(datat)o (yp)q(e)f(of)g(the)h(v)n(ariable)e(that)h(con)o(tains)75 1278 y(this)j(b)o(yte.)30 b(The)18 b(v)n(alue)g(of)f(the)h(message)g(en)o(try)g (will)e(b)q(e)j(the)f(binary)g(v)n(alue)f(of)g(the)h(corresp)q(onding)h(b)o (yte)f(in)75 1327 y(memory)m(.)158 1377 y(W)m(e)c(th)o(us)g(ha)o(v)o(e)g(t)o (w)o(o)f(cases:)137 1448 y Fn(\017)21 b Fx(Comm)o(uni)o(cation)d(of)j(t)o(yp) q(ed)h(v)n(alues)g(\(e.g.,)g(with)f(datat)o(yp)q(e)h(di\013eren)o(t)g(from)e Fu(MPI)p 1541 1448 V 15 w(BYTE)p Fx(\),)h(where)h(the)179 1498 y(datat)o(yp)q(es)15 b(of)g(the)g(corresp)q(onding)h(en)o(tries)g(in)e(the)i (sender)g(program,)d(in)i(the)g(send)h(call,)e(in)g(the)i(receiv)o(e)179 1547 y(call)d(and)h(in)f(the)i(receiv)o(er)g(program)d(should)i(all)f(matc)o (h.)137 1624 y Fn(\017)21 b Fx(Comm)o(uni)o(cation)15 b(of)i(un)o(t)o(yp)q (ed)i(v)n(alues)f(\(e.g.,)g(of)f(datat)o(yp)q(e)i Fu(MPI)p 1237 1624 V 15 w(BYTE)p Fx(\),)d(where)k(b)q(oth)e(sender)i(and)d(re-)179 1674 y(ceiv)o(er)e(use)h(the)f(datat)o(yp)q(e)g Fu(MPI)p 686 1674 V 15 w(BYTE)p Fx(.)e(In)h(this)h(case,)g(there)h(are)f(no)g(requiremen)o (ts)f(on)h(the)g(t)o(yp)q(es)g(of)f(the)179 1724 y(corresp)q(onding)h(en)o (tries)g(in)e(the)i(sender)h(and)e(the)g(receiv)o(er)i(programs,)c(nor)i(is)g (it)g(required)h(that)f(they)g(b)q(e)179 1774 y(the)g(same.)158 1844 y(Examples:)158 1894 y(1st)g(program:)140 1965 y Fu(CALL)21 b(MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 2014 y(IF\(rank.EQ.0\))e(THEN)249 2064 y(CALL)i(MPI_SEND\(a\(1\),)e(10,)i(MPI_REAL,)f(1,)i(tag,)f(comm,)f (ierr\))445 2114 y(ELSE)249 2164 y(CALL)h(MPI_RECV\(a\(1\),)e(15,)i (MPI_REAL,)f(0,)i(tag,)f(comm,)f(ierr\))140 2214 y(END)i(IF)158 2284 y Fx(This)12 b(co)q(de)i(is)e(correct)i(if)e(b)q(oth)g(sender)j(and)d (receiv)o(er)i(programs)d(ha)o(v)o(e)h(allo)q(cated)g(consecutiv)o(e)i (storage)f(for)75 2334 y(ten)h(real)g(n)o(um)o(b)q(ers,)f(starting)h(from)e Fu(a\(1\))p Fx(.)75 2384 y(2nd)i(program:)140 2455 y Fu(CALL)21 b(MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 2504 y(IF\(rank.EQ.0\))e(THEN)249 2554 y(CALL)i(MPI_SEND\(a\(1\),)e(10,)i(MPI_REAL,)f(1,)i(tag,)f(comm,)f (ierr\))445 2604 y(ELSE)249 2654 y(CALL)h(MPI_RECV\(a\(1\),)e(40,)i (MPI_BYTE,)f(0,)i(tag,)f(comm,)f(ierr\))140 2704 y(END)i(IF)p eop %%Page: 8 8 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(8)158 45 y(This)12 b(co)q(de)h(is)e(erroneous,)j(since)e (sender)i(and)d(receiv)o(er)j(do)e(not)f(pro)o(vide)h(matc)o(hing)e(datat)o (yp)q(e)i(parameters.)75 95 y(3rd)i(program:)140 174 y Fu(CALL)21 b(MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 224 y(IF\(rank.EQ.0\))e(THEN)249 274 y(CALL)i(MPI_SEND\(a\(1\),)e(40,)i(MPI_BYTE,)f(1,)i(tag,)f(comm,)f (ierr\))445 324 y(ELSE)249 373 y(CALL)h(MPI_RECV\(a\(1\),)e(60,)i(MPI_BYTE,)f (0,)i(tag,)f(comm,)f(ierr\))140 423 y(END)i(IF)158 502 y Fx(This)14 b(co)q(de)h(is)e(correct,)i(irresp)q(ectiv)o(e)h(of)d(the)i(t)o(yp)q(e)f(of)g Fu(a\(1\))f Fx(and)g(follo)o(wing)f(v)n(ariables)h(in)g(store.)158 631 y Fo(Discussion:)158 676 y Ft(The)j(b)q(eha)o(vior)i(of)e(erroneous)h(co) q(de)f(is)h(implemen)o(tation-de)q(p)q(end)q(en)o(t.)29 b(It)15 b(is)i(desireable)h(but)f(not)f(required)i(that)75 722 y(errors)13 b(are)g(rep)q(orted)h(at)f(run-time.)k(An)c(MPI)g(implemen)o(tation)j(ma)o(y) d(decide)h(to)f(a)o(v)o(oid)h(t)o(yp)q(e-c)o(hec)o(king)h(for)e(the)g(sak)o (e)g(of)75 768 y(e\016ciency)m(,)i(in)g(whic)o(h)g(case)f(none)h(of)e(the)h (three)h(programs)g(will)g(rep)q(ort)f(an)h(error.)20 b(On)14 b(a)g(system)g(where)g(real)h(n)o(um)o(b)q(ers)75 813 y(o)q(ccup)o(y)g(four)f (b)o(ytes)g(and)g(b)q(oth)g(sender)h(and)f(receiv)o(er)h(execute)f(in)h(the)f (same)g(en)o(vironmen)o(t,)h(it)f(is)h(lik)o(ely)h(that)e(all)h(three)75 859 y(co)q(des)f(will)g(ha)o(v)o(e)g(the)f(same)g(b)q(eha)o(vior.)75 1078 y Fw(1.6)59 b(Data)19 b(conversion)75 1169 y Fx(One)g(of)e(the)h(goals)f (of)g(MPI)h(is)g(to)g(supp)q(ort)g(parallel)f(computations)g(across)h (heterogeneous)i(en)o(vironmen)o(ts.)75 1219 y(Comm)o(uni)o(cation)11 b(in)i(a)h(heterogeneous)i(en)o(vironmen)o(t)d(ma)o(y)f(require)i(data)g(con) o(v)o(ersions.)158 1269 y(W)m(e)g(use)g(the)h(follo)o(wing)c(terminology:)75 1348 y Fv(t)o(yp)q(e)k(con)o(v)o(ersion)j Fx(c)o(hanges)d(the)f(datat)o(yp)q (e)g(of)f(a)h(v)n(alue,)f(e.g.)18 b(b)o(y)13 b(rounding)h(a)f Fu(REAL)g Fx(to)h(an)g Fu(INTEGER)p Fx(.)75 1429 y Fv(represen)o(tati)o(on)e (con)o(v)o(ersion)18 b Fx(c)o(hanges)e(the)g(binary)f(represen)o(tation)i(of) e(a)g(v)n(alue,)f(e.g.)23 b(from)13 b(Hex)j(\015oating)179 1479 y(p)q(oin)o(t)d(to)h(IEEE)g(\015oating)f(p)q(oin)o(t.)158 1558 y(Represen)o(tation)20 b(con)o(v)o(ersion)f(is)g(needed)i(when)e(data)g (is)g(mo)o(v)o(ed)f(across)h(di\013eren)o(t)i(en)o(vironmen)o(ts,)e(e.g.)75 1608 y(di\013eren)o(t)h(mac)o(hines)d(that)i(use)g(di\013eren)o(t)h(binary)e (enco)q(dings)h(for)f(the)h(same)f(datat)o(yp)q(e,)h(or)g(co)q(des)h (compiled)75 1658 y(with)d(di\013eren)o(t)h(compiler)e(options)h(for)g(datat) o(yp)q(e)h(represen)o(tation.)30 b(Represen)o(tation)18 b(con)o(v)o(ersion)g (preserv)o(es,)75 1707 y(to)c(the)h(greatest)h(p)q(ossible)e(extend,)h(the)g Fv(v)m(alue)f Fx(represen)o(ted.)22 b(Ho)o(w)o(ev)o(er,)15 b(rounding)e(errors)j(and)e(o)o(v)o(er\015o)o(w)g(and)75 1757 y(under\015o)o(w)i(exceptions)h(ma)o(y)d(o)q(ccur)j(during)e(\015oating)g(p)q (oin)o(t)g(con)o(v)o(ersions;)i(con)o(v)o(ersion)f(of)f(in)o(tegers)i(ma)o(y) d(also)75 1807 y(lead)i(to)h(exceptions)g(when)g(a)f(v)n(alue)g(that)h(can)f (represen)o(ted)j(in)d(one)h(system)f(cannot)h(b)q(e)g(represen)o(ted)i(in)d (the)75 1857 y(other)f(system.)i(MPI)d(do)q(es)h(not)f(sp)q(ecify)g(rules)h (for)e(represen)o(tation)j(con)o(v)o(ersion.)158 1907 y(The)j(t)o(yp)q(e)f (matc)o(hing)e(rules)j(imply)d(that)i(MPI)g(comm)o(unication)d(nev)o(er)k(en) o(tails)e(t)o(yp)q(e)i(con)o(v)o(ersion.)31 b(On)75 1956 y(the)15 b(other)g(hand,)f(MPI)g(requires)i(that)e(a)g(represen)o(tation)i(con)o(v)o (ersion)e(b)q(e)h(p)q(erformed)f(when)h(a)f(t)o(yp)q(ed)g(v)n(alue)g(is)75 2006 y(transferred)g(across)g(en)o(vironmen)o(ts)d(that)i(use)g(di\013eren)o (t)g(represen)o(tations)i(for)d(the)h(datat)o(yp)q(e)g(of)e(this)i(v)n(alue.) k(An)75 2056 y(exception)f(o)q(ccurring)g(during)f(represen)o(tation)i(con)o (v)o(ersion)f(results)g(in)f(a)h(failure)e(of)h(the)h(comm)o(unicatio)o(n;)d (an)75 2106 y(error)i(o)q(ccurs)g(either)g(in)e(the)i(send)g(op)q(eration,)e (or)h(the)g(receiv)o(e)h(op)q(eration,)f(or)g(b)q(oth.)158 2156 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o(t)o(yp)q (ed)i(\(i.e.,)e(of)g(t)o(yp)q(e)i Fu(MPI)p 1180 2156 14 2 v 15 w(BYTE)p Fx(\),)e(then)h(the)h(binary)e(represen)o(tation)75 2205 y(of)h(the)g(b)o(yte)h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o (tical)f(to)h(the)h(binary)f(represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded) e(at)h(the)75 2255 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender) g(and)e(receiv)o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o (vironmen)o(ts;)75 2305 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g (required.)158 2355 y(Note)f(that)g(no)g(con)o(v)o(ersion)g(ev)o(er)h(o)q (ccurs)g(when)g(an)e(MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f (system,)75 2405 y(where)j(all)e(pro)q(cesses)j(run)f(in)e(the)i(same)e(en)o (vironmen)o(t.)k(Also)c(note)i(the)f(di\013eren)o(t)h(b)q(eha)o(vior)f(of)f Fu(MPI)p 1695 2405 V 15 w(BYTE)g Fx(and)75 2455 y(of)18 b Fu(MPI)p 196 2455 V 15 w(CHARACTER)p Fx(.)f(A)h(bu\013er)i(descriptor)g(en)o(try)f (with)g(datat)o(yp)q(e)g(of)f Fu(MPI)p 1320 2455 V 15 w(CHARACTER)f Fx(can)i(only)f(matc)o(h)f(a)75 2504 y(F)m(ortran)e(v)n(ariable)f(of)h(t)o (yp)q(e)g Fu(CHARACTER)p Fx(;)e(and)i(represen)o(tation)i(con)o(v)o(ersion)e (ma)o(y)e(o)q(ccur)j(when)g(v)n(alues)f(of)f(t)o(yp)q(e)75 2554 y Fu(MPI)p 144 2554 V 15 w(CHARACTER)e Fx(are)i(transferred.,)h(e.g.,)e (from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q(ding.)158 2604 y(Consider)g(the)h(previous)f(three)h(examples.)158 2654 y(The)h(\014rst)h(program)d(is)i(correct,)h(assuming)d(b)q(oth)i(sender)i (and)d(receiv)o(er)j(declared)e(ten)h(consecutiv)o(e)g(real)75 2704 y(v)n(ariables)e(in)g(storage)g(starting)h(at)f Fu(a\(1\))p Fx(.)22 b(If)15 b(the)h(sender)h(and)e(receiv)o(er)i(execute)g(in)e (di\013eren)o(t)h(en)o(vironmen)o(ts,)p eop %%Page: 9 9 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(9)75 45 y(then)13 b(the)f(ten)h(real)f(v)n(alues)g(that)g(are) h(fetc)o(hed)g(from)d(the)j(send)g(bu\013er)g(will)e(b)q(e)h(con)o(v)o(erted) i(to)e(the)g(represen)o(tation)75 95 y(for)j(reals)g(on)f(the)i(receiv)o(er)g (site)g(b)q(efore)f(they)h(are)f(stored)h(in)e(the)i(receiv)o(e)g(bu\013er.) 22 b(While)14 b(the)i(n)o(um)o(b)q(er)e(of)g(real)75 145 y(elemen)o(ts)e (fetc)o(hed)i(from)c(the)j(send)h(bu\013er)f(equal)f(the)h(n)o(um)o(b)q(er)f (of)g(real)g(elemen)o(ts)g(stored)i(in)e(the)h(receiv)o(e)g(bu\013er,)75 195 y(the)j(n)o(um)o(b)q(er)e(of)h(b)o(ytes)h(stored)g(need)g(not)f(equal)g (the)g(n)o(um)o(b)q(er)g(of)f(b)o(ytes)i(loaded:)k(e.g.)i(the)15 b(sender)i(ma)o(y)c(use)j(a)75 244 y(four)11 b(b)o(yte)h(represen)o(tation)h (and)e(the)i(receiv)o(er)g(an)e(eigh)o(t)g(b)o(yte)h(represen)o(tation)h(for) e(reals.)18 b(If)11 b(the)h(send)g(or)g(receiv)o(e)75 294 y(bu\013er)i(do)e (not)g(con)o(tain)g(ten)i(consecutiv)o(e)g(real)e(v)n(ariables,)g(then)h(the) g(program)e(is)i(erroneous,)g(and)g(its)f(b)q(eha)o(vior)75 344 y(is)i(unde\014ned.)158 394 y(The)g(second)h(program)e(is)g(erroneous,)i (and)f(its)g(b)q(eha)o(vior)f(is)h(unde\014ned.)158 444 y(The)h(third)g (program)e(is)h(correct.)22 b(The)16 b(exact)f(same)f(sequence)i(of)f(fourt)o (y)f(b)o(ytes)h(that)g(w)o(ere)g(loaded)g(from)75 493 y(the)j(send)g (bu\013er)h(will)d(b)q(e)i(stored)g(in)f(the)h(receiv)o(e)h(bu\013er,)f(ev)o (en)g(if)f(sender)i(and)e(receiv)o(er)i(run)f(in)f(a)g(di\013eren)o(t)75 543 y(en)o(vironmen)o(t.)i(The)c(message)f(sen)o(t)i(has)e(exactly)h(the)g (same)f(length)h(\(in)f(b)o(ytes\))h(and)g(the)g(same)f(binary)g(repre-)75 593 y(sen)o(tation)i(as)h(the)g(message)f(receiv)o(ed.)26 b(If)16 b(the)h(v)n(ariables)f(in)g(the)h(send)g(bu\013er)g(are)g(of)e(di\013eren)o (t)j(t)o(yp)q(es)f(as)f(the)75 643 y(v)n(ariables)f(in)h(the)g(receiv)o(e)i (bu\013er,)f(or)f(they)g(are)g(of)g(the)g(same)f(t)o(yp)q(e)i(but)f (di\013eren)o(t)h(data)f(represen)o(tations)i(are)75 693 y(used,)c(then)g (the)g(bits)g(stored)g(in)f(the)i(receiv)o(e)f(bu\013er)h(ma)o(y)c(enco)q(de) k(v)n(alues)e(that)h(are)g(di\013eren)o(t)g(from)e(the)i(v)n(alues)75 742 y(they)g(enco)q(ded)i(in)d(the)i(send)f(bu\013er.)158 792 y(Data)i(represen)o(tation)h(con)o(v)o(ersion)g(also)e(applies)h(to)g(the)h (en)o(v)o(elop)q(e)g(of)e(a)h(message:)23 b(source,)17 b(destination)75 842 y(and)d(tag)f(are)i(all)d(in)o(tegers)j(that)f(ma)o(y)e(need)j(to)e(b)q (e)i(con)o(v)o(erted.)158 892 y(The)i(curren)o(t)h(draft)f(do)q(es)g(not)g (pro)o(vide)f(a)g(mec)o(hanism)f(for)h(in)o(terlanguage)g(comm)o(unicatio)o (n:)21 b(messages)75 942 y(sen)o(t)d(b)o(y)f(F)m(ortran)f(calls)h(should)g(b) q(e)g(receiv)o(ed)i(b)o(y)d(F)m(ortran)h(calls)g(and)g(messages)g(sen)o(t)g (b)o(y)g(C)g(calls)g(should)f(b)q(e)75 992 y(receiv)o(ed)g(b)o(y)f(C)f(calls) h(\(this)g(follo)o(ws)e(from)g(the)i(requiremen)o(ts)g(of)f(t)o(yp)q(e)i (matc)o(hing)d(and)h(the)i(fact)e(that)h(F)m(ortran)75 1041 y(and)g(C)g(datat)o(yp)q(es)g(are)g(distinct\).)22 b(If)14 b(in)o(terlanguage)h(comm)o(uni)o(cation)d(is)j(needed)h(then)g(one)f(needs)h (to)f(in)o(v)o(ok)o(e)75 1091 y(C)i(comm)o(unication)d(routines)k(from)e(a)h (F)m(ortran)g(program)e(or)j(F)m(ortran)f(comm)o(unicatio)o(n)e(routines)i (from)f(a)h(C)75 1141 y(program.)158 1269 y Fo(Implemen)o(tation)c(note:)158 1315 y Ft(The)d(curren)o(t)g(de\014nition)j(do)q(es)d(not)h(require)g (messages)g(to)e(carry)i(t)o(yp)q(e)f(information.)18 b(A)9 b(message)i(can)f(b)q(e)h(comp)q(osed)75 1361 y(and)h(sen)o(t)f(using)i(only) g(the)e(information)i(pro)o(vided)h(b)o(y)d(the)h(send)f(call,)i(and)f(can)g (b)q(e)f(receiv)o(ed)i(and)f(stored)f(using)i(only)g(the)75 1406 y(information)g(pro)o(vided)g(b)o(y)e(the)g(receiv)o(e)h(call.)18 b(If)10 b(messages)i(are)f(sen)o(t)g(b)q(et)o(w)o(een)g(di\013eren)o(t)h(mac) o(hines)h(then)e(one)h(can)f(either)75 1452 y(use)h(a)g(\\univ)o(ersal")j (data)d(enco)q(ding)i(for)d(messages,)i(use)f(kno)o(wledge)i(of)d(the)h (receiv)o(er)h(en)o(vironmen)o(t)h(in)e(order)g(to)g(con)o(v)o(ert)75 1498 y(data)h(at)g(the)h(sender,)f(or)g(use)g(kno)o(wledge)i(of)e(the)g (sender)h(en)o(vironmen)o(t)h(in)e(order)h(to)f(con)o(v)o(ert)g(data)h(at)f (the)g(receiv)o(er.)18 b(In)75 1543 y(either)12 b(case)g(the)g(datat)o(yp)q (e)g(parameter)g(in)h(the)e(lo)q(cal)i(call)g(can)f(b)q(e)g(used)g(to)g (deriv)o(e)g(the)g(t)o(yp)q(es)g(of)f(the)h(v)n(alues)g(transferred.)75 1589 y(Ho)o(w)o(ev)o(er,)g(additional)17 b(t)o(yp)q(e)c(information)i (carried)f(b)o(y)g(messages)g(can)f(pro)o(vide)i(b)q(etter)e(error)g (detection.)158 1800 y Fo(Discussion:)158 1846 y Ft(MPI)h(do)q(es)h(not)f (handle)i(in)o(terlanguage)h(comm)o(unication)g(b)q(ecause)e(there)f(seem)g (to)g(b)q(e)g(no)h(agreed)g(standards)g(for)75 1891 y(in)o(terlanguage)i(t)o (yp)q(e)e(con)o(v)o(ersions.)24 b(The)14 b(curren)o(t)h(\(non\)solution)i (means)e(that)g(in)o(terlanguage)i(comm)o(unication)g(relies)75 1937 y(on)c(the)g(v)o(endor)h(sp)q(eci\014c)h(rules)f(for)f(in)o(terlanguage) j(calls)e(at)f(eac)o(h)h(site.)75 2157 y Fw(1.7)59 b(Comm)n(unicat)o(ion)17 b(Mo)r(des)75 2248 y Fx(The)i(basic)f(send)i(op)q(eration)e(describ)q(ed)i (in)e(Section)h(1.2)e(used)i(the)g Fv(standard)e Fx(comm)o(uni)o(cation)f(mo) q(de.)30 b(In)75 2298 y(suc)o(h)15 b(comm)o(unicatio)o(n)c(mo)q(de,)i(a)h (send)h(op)q(eration)f(can)g(b)q(e)h(started)g(whether)h(or)e(not)g(a)f(matc) o(hing)g(receiv)o(e)i(w)o(as)75 2347 y(p)q(osted.)j(The)13 b(completion)d(of)i(the)g(send)h(op)q(eration)f(indicates)g(that)g(the)h (message)f(and)f(its)i(en)o(v)o(elop)q(e)f(ha)o(v)o(e)g(b)q(een)75 2397 y(safely)17 b(stored)h(a)o(w)o(a)o(y)d(and)i(that)g(the)h(sender)g(is)f (free)h(to)e(access)j(and)e(mo)q(dify)e(the)i(sender)i(bu\013er.)28 b(Th)o(us,)17 b(the)75 2447 y(op)q(eration)f(is)g Fv(blo)q(c)o(king)p Fx(:)k(it)c(do)q(es)g(not)g(return)h(un)o(til)f(the)g(send)h(op)q(eration)f (has)g(completed)g Fv(lo)q(cally)p Fx(,)f(on)g(the)75 2497 y(sender)g(side.)158 2547 y(The)h(completion)e(of)i(a)f(send)i(op)q(eration)e (giv)o(es)h(no)g(indication)e(that)i(the)g(message)g(w)o(as)g(receiv)o(ed)h (on)e(the)75 2597 y(receiv)o(er)k(side.)29 b(A)17 b(blo)q(c)o(king)g(send)h (ma)o(y)d(b)q(e)j(implemen)o(ted)d(so)j(that)f(a)g(blo)q(c)o(king)g(send)h (returns)h(only)e(after)g(a)75 2646 y(matc)o(hing)e(receiv)o(e)k(has)e(b)q (een)h(executed)h(on)e(the)g(receiv)o(er)i(side.)28 b(This)17 b(a)o(v)o(oids)f(the)h(need)h(to)f(bu\013er)h(message)75 2696 y(data)f(out)h(of)f(sender)i(or)f(receiv)o(er)h(memory)m(.)26 b(In)18 b(this)f(case)i(the)f(send)h(op)q(eration)e(completes)h(only)e(after) i(the)p eop %%Page: 10 10 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(10)75 45 y(matc)o(hing)16 b(receiv)o(e)i(has)g(started)g (executing.)30 b(On)17 b(the)h(other)h(hand,)e(it)g(is)h(also)e(p)q(ossible)i (for)f(MPI)h(to)f(bu\013er)75 95 y(messages,)12 b(so)g(as)g(to)g(allo)o(w)f (the)h(sender)i(to)e(pro)q(ceed)h(ahead)g(of)e(the)i(receiv)o(er.)19 b(In)12 b(suc)o(h)g(case)h(the)g(send)g(op)q(eration)75 145 y(ma)o(y)d(complete)i(sucessfully)g(b)q(efore)h(the)g(message)f(w)o(as)f (receiv)o(ed.)19 b(Th)o(us,)12 b(the)h(basic)f(send)h(op)q(eration)f(describ) q(ed)75 195 y(in)g(the)g(previous)h(section)g(is)f Fv(async)o(hronous)p Fx(,)d(since)k(its)f(return)i(do)q(es)f(not)f(imply)e(a)h(sync)o(hronization) i(with)e(the)75 244 y(\(remote\))j(receiv)o(e)h(op)q(eration,)e(and)h(do)q (es)h(not)e(imply)f Fv(global)g Fx(termination)g(of)h(the)i(comm)o(uni)o (cation.)158 294 y(There)g(are)f(t)o(w)o(o)g(additional)e(comm)o(unicatio)o (n)f(mo)q(des:)158 344 y(A)j(send)h(that)f(uses)h(the)f Fv(ready)g Fx(comm)o(uni)o(cation)d(mo)q(de)i(ma)o(y)f(b)q(e)i(started)h(only)e(if)g(a)h (matc)o(hing)e(receiv)o(e)j(is)75 394 y(already)d(p)q(osted;)h(otherwise)g (the)g(op)q(eration)f(is)g(erroneous)i(and)e(its)g(outcome)g(is)g (unde\014ned.)19 b(In)12 b(some)f(systems,)75 444 y(this)h(allo)o(ws)e(the)i (remo)o(v)n(al)d(of)i(a)g(hand-shak)o(e)h(op)q(eration)g(that)f(is)h (otherwise)g(required,)h(and)e(results)i(in)e(impro)o(v)o(ed)75 493 y(p)q(erformance.)24 b(The)17 b(completion)d(of)i(the)g(send)h(op)q (eration)f(do)q(es)h(not)f(dep)q(end)h(on)f(the)h(status)g(of)e(a)h(matc)o (hing)75 543 y(receiv)o(e,)i(and)e(merely)g(indicates)h(that)g(the)g(send)g (bu\013er)h(can)f(b)q(e)g(reused.)27 b(A)17 b(send)h(op)q(eration)e(that)h (uses)g(the)75 593 y(ready)i(mo)q(de)e(has)i(the)g(same)e(seman)o(tics)h(as)h (a)f(standard)h(send)g(op)q(eration.)32 b(It)18 b(is)g(merely)g(that)g(the)h (sender)75 643 y(pro)o(vides)14 b(addtional)f(information)e(to)i(the)i(comm)o (unicatio)o(n)c(subsystem,)j(when)g(using)g(the)g(ready)h(mo)q(de.)158 693 y(A)d(send)h(that)f(uses)h(the)g Fv(sync)o(hronous)c Fx(mo)q(de)i(can)i (b)q(e)f(started)h(whether)h(or)e(not)g(a)g(matc)o(hing)e(receiv)o(e)j(w)o (as)75 742 y(p)q(osted.)23 b(Ho)o(w)o(ev)o(er,)16 b(the)g(send)g(will)e (complete)g(successfully)j(only)d(if)h(a)g(matc)o(hing)e(receiv)o(e)k(is)e(p) q(osted,)h(and)f(the)75 792 y(receiv)o(e)g(op)q(eration)e(has)g(started)h(to) f(receiv)o(e)i(the)f(message)f(sen)o(t)h(b)o(y)f(the)h(sync)o(hronous)g (send.)k(\(I.e.,)13 b(the)h(receiv)o(e)75 842 y(has)j(b)q(een)g(p)q(osted,)h (and)f(the)g(incoming)d(message)j(has)f(b)q(een)i(matc)o(hed)e(to)g(this)h(p) q(osted)h(receiv)o(e.\))27 b(Th)o(us,)17 b(the)75 892 y(completion)e(of)h(a)g (sync)o(hronous)h(send)h(not)e(only)g(indicates)h(that)f(the)i(send)f (bu\013er)h(can)e(b)q(e)h(reused,)i(but)e(also)75 942 y(indicates)f(that)f (the)h(receiv)o(er)h(has)f(reac)o(hed)h(a)e(certain)h(p)q(oin)o(t)f(in)g(its) h(execution,)g(namely)e(that)h(it)g(has)h(started)75 992 y(executing)g(the)f (matc)o(hing)e(receiv)o(e.)23 b(If)14 b(b)q(oth)h(sends)i(and)d(receiv)o(es)j (are)e(blo)q(c)o(king)f(op)q(erations)i(then)f(the)h(use)g(of)75 1041 y(the)f(sync)o(hronous)h(mo)q(de)e(pro)o(vides)h(sync)o(hronous)g(comm)o (unication)d(seman)o(tics:)19 b(a)14 b(comm)o(unication)e(do)q(es)j(not)75 1091 y(complete)f(at)h(either)g(ends)h(b)q(efore)f(b)q(oth)g(pro)q(cesses)i (\\attend")e(to)g(the)g(comm)o(unicatio)o(n;)d(the)j(completion)e(of)h(a)75 1141 y(sync)o(hronous)h(send)g(is)e(a)h Fv(global)e Fx(ev)o(en)o(t.)158 1191 y(Tw)o(o)g(new)i(send)g(functions)f(are)g(pro)o(vided)g(for)g(the)g(t)o (w)o(o)g(new)g(comm)o(unication)d(mo)q(des.)17 b(The)c(comm)o(unica-)75 1241 y(tion)g(mo)q(de)g(is)h(indicated)g(b)o(y)f(a)h(one)g(letter)h (pre\014x:)k Fu(R)13 b Fx(for)h(ready)g(and)g Fu(S)f Fx(for)h(sync)o(hronous) 158 1340 y(Send)g(in)g(ready)g(mo)q(de)75 1437 y Fu(MPI)p 144 1437 14 2 v 15 w(RSEND)21 b(\(start,)f(count,)h(datatype,)f(dest,)g(tag,)h (comm\))117 1487 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 1533 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g (bu\013er)g(\(in)o(teger\))117 1578 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 1624 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1670 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1715 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 1891 y Fx(Send)g(in)g(sync)o(hronous)h(mo)q(de)75 1988 y Fu(MPI)p 144 1988 V 15 w(SSEND)21 b(\(start,)f(count,)h(datatype,)f(dest,)g(tag,)h (comm\))117 2038 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 2084 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g (bu\013er)g(\(in)o(teger\))117 2130 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 2175 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2221 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2267 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 2343 y Fx(There)g(is)e(only)g(one)g(receiv)o(e)i(mo)q(de,)d(whic)o(h)h(can)h (matc)o(h)e(an)o(y)h(of)g(the)h(send)h(mo)q(des.)i(The)d(receiv)o(e)h(op)q (eration)75 2393 y(describ)q(ed)h(in)e(the)h(last)g(section)g(is)f Fv(blo)q(c)o(king)p Fx(:)j(it)d(returns)i(only)d(after)i(the)g(receiv)o(e)h (bu\013er)g(con)o(tains)e(the)h(newly)75 2442 y(receiv)o(ed)20 b(message.)34 b(It)19 b(is)g Fv(async)o(hronous)p Fx(:)25 b(the)20 b(completion)d(of)i(a)f(receiv)o(e)j(is)e(a)f(lo)q(cal)h(op)q(eration,)g(and) g(a)75 2492 y(receiv)o(e)c(can)e(complete)g(b)q(efore)i(the)f(matc)o(hing)d (send)k(has)e(completed)g(\(or)h(course,)g(it)f(can)h(complete)f(only)g (after)75 2542 y(the)h(matc)o(hing)f(send)h(has)g(started\).)158 2592 y(Comm)o(uni)o(cation)f(imp)q(oses)j(an)g(order)h(on)g(the)g(ev)o(en)o (ts)g(o)q(ccurring)g(at)f(the)h(comm)o(unicating)c(no)q(des.)27 b(It)16 b(is)75 2642 y(alw)o(a)o(ys)d(the)i(case)h(that)e(the)h(completion)d (of)i(a)g(receiv)o(e)i(o)q(ccurs)g(after)e(the)h(start)g(of)f(the)h(matc)o (hing)d(send.)20 b(If)14 b(the)75 2692 y(sync)o(hronous)i(send)g(mo)q(de)f (is)g(used,)h(then)g(it)f(is)g(also)g(the)h(case)g(that)f(the)h(completion)e (of)h(a)g(send)h(o)q(ccurs)h(after)p eop %%Page: 11 11 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(11)75 45 y(the)16 b(start)g(of)g(the)g(matc)o(hing)e(receiv)o (e.)24 b(Of)16 b(course,)h(on)e(eac)o(h)h(pro)q(cess,)i(a)d(comm)o(unicatio)o (n)e(completes)i(after)h(it)75 95 y(is)h(started.)28 b(No)17 b(other)g(order)h(is)f(imp)q(osed)f(b)o(y)g(MPI.)h(E.g.,)f(if)g(the)i (standard)f(send)h(mo)q(de)e(is)g(used,)j(the)e(send)75 145 y(op)q(eration)d(ma)o(y)e(complete)h(b)q(efore)i(the)f(matc)o(hing)e(receiv)o (e)j(w)o(as)f(started.)158 195 y(In)e(a)f(m)o(ultithreaded)f(implemen)o (tation)e(of)j(MPI,)h(a)f(blo)q(c)o(king)g(send)h(or)g(receiv)o(e)g(op)q (eration)g(ma)o(y)e(only)g(blo)q(c)o(k)75 244 y(the)18 b(executing)h(thread,) f(not)g(the)g(en)o(tire)g(pro)q(cess.)31 b(Th)o(us,)19 b(while)e(a)g(thread)h (is)g(blo)q(c)o(k)o(ed)f(w)o(aiting)g(for)g(a)g(send)75 294 y(or)f(receiv)o(e)h(to)f(complete,)g(other)g(threads)h(ma)o(y)e(execute)i (within)f(the)g(same)g(address)h(space.)25 b(In)17 b(suc)o(h)f(case)h(it)75 344 y(the)e(user)h(resp)q(onsibilit)o(y)f(not)f(to)h(mo)q(dify)d(the)k(send)f (bu\013er)h(un)o(til)e(the)h(send)h(completes)f(and)f(not)h(to)f(access)j(or) 75 394 y(mo)q(dify)12 b(the)k(receiv)o(e)g(bu\013er)g(un)o(til)e(the)h (receiv)o(e)h(completes;)f(otherwise)h(the)f(outcome)f(of)h(the)g (computation)e(is)75 444 y(unde\014ned.)158 572 y Fo(Discussion:)158 618 y Ft(Do)g(w)o(e)g(w)o(an)o(t)g(to)g(sa)o(y)g(that)g(a)g(blo)q(c)o(king)j (op)q(eration)f(MA)m(Y)d(or)h(MUST)g(only)h(blo)q(c)o(k)g(the)g(executing)g (thread?)158 829 y Fo(Implemen)o(tation)h(note:)45 b Ft(A)15 b(ready)h(send)g(can)g(b)q(e)f(implemen)o(ted)j(as)e(a)f(standard)i(send;)g (in)f(suc)o(h)g(case)g(there)75 874 y(will)f(b)q(e)e(no)g(p)q(erformance)h (adv)n(an)o(tage)g(\(or)f(disadv)n(an)o(tage\))j(for)d(the)g(use)g(of)g (ready)g(send.)158 920 y(A)f(standard)h(send)g(can)f(b)q(e)h(implemen)o(ted)h (as)e(a)g(sync)o(hronous)i(send.)j(In)c(suc)o(h)f(case,)g(no)h(data)f (bu\013ering)i(is)f(needed.)158 966 y(A)g(p)q(ossible)i(comm)o(unication)h (proto)q(col)e(for)f(the)g(v)n(arious)i(comm)o(unication)g(mo)q(des)f(is)g (outlined)h(b)q(elo)o(w:)158 1011 y Fm(ready)j(send)p Ft(:)d(The)e(message)h (is)g(sen)o(t)f(asap;)h(an)f(ac)o(kno)o(wledge)i(is)f(returned)g(b)o(y)g (receiv)o(er.)k(The)13 b(proto)q(col)i(is)f(\(send,)75 1057 y(ac)o(k\).)158 1103 y Fm(synchronou)o(s)i(send:)25 b Ft(The)18 b(sender)g(sends)h(a)f(request-to-send)h(message.)31 b(The)18 b(receiv)o(er)h(stores)f(this)g(request.)75 1148 y(When)e(a)e(matc)o(hing)j (receiv)o(e)e(is)h(p)q(osted,)f(the)g(receiv)o(er)h(sends)f(bac)o(k)h(a)f(p)q (ermission-to-send)i(message,)f(and)f(the)g(sender)75 1194 y(no)o(w)e(sends)h(the)f(message.)18 b(The)12 b(proto)q(col)j(is)f(\(req,)e (ac)o(k-req,)h(send,)h(ac)o(k\).)158 1240 y Fm(standard)j(send:)d Ft(First)e(proto)q(col)g(ma)o(y)g(b)q(e)f(used)h(for)f(short)g(messages,)h (and)g(second)g(proto)q(col)h(for)e(long)h(messages.)158 1289 y(Of)g(course,)i(there)f(are)g(man)o(y)h(other)f(p)q(ossible)i(c)o(hoices.)75 1509 y Fw(1.8)59 b(Nonblo)r(cking)20 b(comm)m(unicat)o(ion)75 1600 y Fx(One)f(can)g(impro)o(v)o(e)d(p)q(erformance)i(on)g(man)o(y)f (systems)h(b)o(y)g(o)o(v)o(erlaping)f(comm)o(unication)e(and)j(computation.) 75 1650 y(This)13 b(is)g(esp)q(ecially)g(true)h(on)f(systems)g(where)h(comm)o (unicatio)o(n)d(can)i(b)q(e)g(executed)i(autonomously)c(b)o(y)i(an)g(in)o (tel-)75 1700 y(ligen)o(t)d(comm)o(unicatio)o(n)e(con)o(troller.)17 b(Ligh)o(t-w)o(eigh)o(t)9 b(threads)i(are)g(one)g(mec)o(hanism)d(for)i(ac)o (hieving)g(suc)o(h)h(o)o(v)o(erlap.)75 1750 y(An)j(alternativ)o(e)g(mec)o (hanism)e(that)j(often)f(leads)g(to)g(b)q(etter)i(p)q(erformance)e(is)g(to)h (use)g Fv(non)o(blo)q(c)o(kin)o(g)e(comm)o(u-)75 1800 y(nication)p Fx(.)i(A)f(non)o(blo)q(c)o(king)e(send)i(call)e(initiates)h(the)h(send)g(op)q (eration,)f(but)g(do)q(es)h(not)f(complete)g(it.)k(The)d(send)75 1849 y(call)h(will)e(return)k(b)q(efore)f(the)g(message)f(w)o(as)g(copied)h (out)f(of)g(the)h(send)g(bu\013er.)24 b(A)15 b(separate)i(call)d(is)h(needed) i(to)75 1899 y(complete)f(the)i(comm)o(unicati)o(on,)c(i.e.)27 b(to)16 b(v)o(erify)h(that)f(the)i(data)e(has)h(b)q(een)h(copied)f(out)g(of)f (the)i(send)f(bu\013er.)75 1949 y(With)e(suitable)h(hardw)o(are,)g(the)g (transfer)h(of)e(data)g(out)h(of)f(the)h(sender)h(memory)d(ma)o(y)f(pro)q (ceed)18 b(concurren)o(tly)75 1999 y(with)10 b(computations)g(done)h(at)f (the)h(sender)i(after)d(the)i(send)f(w)o(as)g(initiated)f(and)g(b)q(efore)i (it)e(completed.)16 b(Similarly)l(,)75 2049 y(a)e(non)o(blo)q(c)o(king)f (receiv)o(e)i(initiates)f(the)h(receiv)o(e)g(op)q(eration,)f(but)g(do)q(es)h (not)f(complete)g(it.)k(The)d(call)e(will)g(return)75 2098 y(b)q(efore)i(a)f(message)f(is)h(stored)h(in)o(to)e(the)i(receiv)o(e)g (bu\013er.)20 b(A)14 b(separate)h(call)e(is)h(needed)i(to)e(complete)f(the)i (receiv)o(e)75 2148 y(op)q(eration)c(and)g(v)o(erify)g(that)h(the)f(data)g (has)h(b)q(een)g(receiv)o(ed)h(in)o(to)e(the)g(receiv)o(e)i(bu\013er.)18 b(With)11 b(suitable)g(hardw)o(are,)75 2198 y(the)18 b(transfer)g(of)e(data)h (in)o(to)g(the)g(receiv)o(er)i(memory)c(ma)o(y)g(pro)q(ceed)k(concurren)o (tly)f(with)f(computations)f(done)75 2248 y(after)e(the)h(receiv)o(e)g(w)o (as)f(initiated)f(and)g(b)q(efore)i(it)f(completed.)158 2298 y(Non)o(blo)q(c)o(king)k(sends)j(can)e(use)h(the)g(same)f(three)h(mo)q(des)f (as)g(blo)q(c)o(king)f(sends:)30 b Fu(standard)p Fx(,)19 b Fu(ready)f Fx(and)75 2347 y Fu(asynchronous)p Fx(.)j(These)16 b(carry)h(the)f(same)f(meaning.)21 b(The)16 b(initiation)d(and)j(completion)e (of)h(a)g(standard)h(send)75 2397 y(do)c(not)h(dep)q(end)g(on)g(the)g(status) g(of)f(a)g(matc)o(hing)e(receiv)o(e.)19 b(A)13 b(ready)g(send)g(can)g(b)q(e)g (initiated)e(only)h(if)g(a)g(matc)o(hing)75 2447 y(receiv)o(e)h(has)f (already)f(b)q(een)i(initiated,)e(otherwise)i(the)f(call)f(is)h(erroneous;)h (its)f(completion)e(do)q(es)j(not)e(dep)q(end)i(on)75 2497 y(the)g(status)h(of)d(a)i(matc)o(hing)d(receiv)o(e.)19 b(A)13 b(sync)o(hronous)g(send)h(can)f(b)q(e)g(initiated)f(b)q(efore)h(a)f(matc)o (hing)f(receiv)o(e)j(has)75 2547 y(b)q(een)i(initiated,)f(but)g(will)f (complete)h(only)f(after)i(a)f(matc)o(hing)e(receiv)o(e)k(has)e(b)q(een)i (initiated,)d(and)h(has)h(started)75 2597 y(receiving)e(the)h(message)e (generated)j(b)o(y)d(the)i(send)f(op)q(eration.)158 2646 y(In)k(all)e(cases)j (the)f(op)q(eration)f(that)h(initiates)f(the)h(comm)o(unicati)o(on)d(\(send)j (or)g(receiv)o(e\))h(is)e(a)g(lo)q(cal,)g(non-)75 2696 y(blo)q(c)o(king)d (call)g(that)h(returns)h(as)f(so)q(on)g(as)g(p)q(ossible.)20 b(A)15 b(non)o(blo)q(c)o(king)f(comm)o(unicatio)o(n)e(call)i(ma)o(y)f(fail)g (b)q(ecause)p eop %%Page: 12 12 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(12)75 45 y(b)q(ecause)13 b(the)g(system)e(has)h(exhausted)h(a) o(v)n(ailable)d(resources)k(\(e.g.,)d(exceeded)j(the)e(limit)d(on)j(n)o(um)o (b)q(er)f(of)g(p)q(ending)75 95 y(comm)o(unications)f(p)q(er)15 b(no)q(de\).)j(Go)q(o)q(d)13 b(qualit)o(y)f(implemen)o(tatio)o(ns)f(of)i(MPI) h(will)e(set)i(these)h(limits)c(high)i(enough)75 145 y(so)h(that)g(a)f(non)o (blo)q(c)o(king)g(comm)o(unication)e(call)i(will)f(fail)g(only)h(in)h (\\pathological")d(cases.)158 195 y(Non)o(blo)q(c)o(king)i(sends)i(can)f(b)q (e)g(matc)o(hed)g(with)f(blo)q(c)o(king)g(receiv)o(es,)i(and)f(vice-v)o (ersa.)75 311 y Fs(1.8.1)49 b(Comm)o(unication)14 b(Objects)75 387 y Fx(Non)o(blo)q(c)o(king)f(comm)o(unications)f(use)j(opaque)f(comm)o (unication)d(ob)r(jects)16 b(to)e(iden)o(tify)g(comm)o(unicatio)o(n)e(op)q (era-)75 437 y(tions)h(and)g(matc)o(h)f(the)i(op)q(eration)f(that)g (initiates)g(the)h(comm)o(uni)o(cation)c(with)j(the)h(op)q(eration)f(that)g (terminates)75 487 y(it.)32 b(These)20 b(are)f(system)f(ob)r(jects)i(that)f (are)g(accessed)i(via)d(a)g(handle.)33 b(An)18 b(opaque)h(comm)o(unicatio)o (n)d(ob)r(ject)75 537 y(iden)o(ti\014es)e(v)n(arious)f(prop)q(erties)i(of)e (a)g(comm)o(unicati)o(on)e(op)q(eration,)i(suc)o(h)h(as)f(the)i(\(send)f(or)g (receiv)o(e\))g(bu\013er)h(that)75 587 y(is)g(asso)q(ciated)h(with)f(it,)g (its)h(con)o(text,)g(the)g(tag)f(and)g(destination)g(parameters)g(to)h(b)q(e) g(used)g(for)f(a)g(send,)h(or)f(the)75 636 y(tag)d(and)f(source)j(parameters) e(to)g(b)q(e)g(used)h(for)f(a)f(receiv)o(e.)19 b(In)12 b(addition,)f(this)h (ob)r(ject)h(stores)g(information)c(ab)q(out)75 686 y(the)14 b(status)h(of)e(the)i(p)q(ending)f(comm)o(unicatio)o(n)d(op)q(eration)j(that) g(is)g(p)q(erformed)f(with)h(this)g(ob)r(ject.)75 802 y Fs(1.8.2)49 b(Comm)o(unication)14 b(initiation)75 879 y Fx(W)m(e)i(use)g(the)h(same)e (naming)e(con)o(v)o(en)o(tions)j(as)g(for)g(blo)q(c)o(king)f(comm)o(unicatio) o(n:)20 b(a)15 b(pre\014x)i(of)e Fu(R)h Fx(\()p Fu(S)p Fx(\))g(is)f(used)i (for)75 929 y Fu(READY)g Fx(\()p Fu(SYNCHRONOUS)p Fx(\))f(mo)q(de.)29 b(In)18 b(addition)f(a)g(pre\014x)i(of)e Fu(I)h Fx(\(for)g Fu(IMMEDIATE)p Fx(\))e(indicates)i(that)g(the)h(call)e(is)75 979 y(non)o(blo)q(c)o(king.)158 1078 y(Initiate)c(a)h(standard)g(mo)q(de)f (non)o(blo)q(c)o(king)g(comm)o(unicatio)o(n.)75 1175 y Fu(MPI)p 144 1175 14 2 v 15 w(ISEND\(handle,)19 b(start,)i(count,)f(datatype,)g(dest,) h(tag,)g(comm\))117 1225 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1271 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 1317 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (send)g(bu\013er)g(\(in)o(teger\))117 1362 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 1408 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1454 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1499 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 1675 y Fx(Initiate)f(a)h(ready)g(mo)q(de)f(non)o(blo)q(c)o(king)g(comm)o (unicatio)o(n.)75 1772 y Fu(MPI)p 144 1772 V 15 w(IRSEND\(handle,)19 b(start,)i(count,)f(datatype,)g(dest,)h(tag,)g(comm\))117 1822 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 1868 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 1913 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er) 14 b(of)e(elemen)o(ts)i(in)g(send)g(bu\013er)g(\(in)o(teger\))117 1959 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h (bu\013er)g(elemen)o(t)117 2005 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2050 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2096 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 2272 y Fx(Initiate)f(a)h(sync)o (hronous)h(mo)q(de)e(non)o(blo)q(c)o(king)f(comm)o(unication.)75 2369 y Fu(MPI)p 144 2369 V 15 w(ISSEND\(handle,)19 b(start,)i(count,)f (datatype,)g(dest,)h(tag,)g(comm\))117 2419 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 2464 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o (hoice\))117 2510 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(bu\013er)g(\(in)o(teger\))117 2556 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h (bu\013er)g(elemen)o(t)117 2601 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2647 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2693 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))p eop %%Page: 13 13 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(13)158 145 y(Initiate)13 b(a)h(non)o(blo)q(c)o(king)f(receiv)o (e.)75 242 y Fu(MPI)p 144 242 14 2 v 15 w(IRECV)21 b(\(handle,)f(start,)h (count,)f(datatype,)g(source,)g(tag,)h(comm\))117 292 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 337 y(out)e Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i (bu\013er)g(\(c)o(hoice\))117 383 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(receiv)o(e)g(bu\013er)g(\(in)o(teger\))117 429 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(receiv)o (e)h(bu\013er)g(elemen)o(t)117 474 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 520 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 566 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 642 y Fx(The)h(handle)f(parameter)h (should)f(not)h(b)q(e)g(asso)q(ciated)g(with)f(a)h(comm)o(unicati)o(on)d(ob)r (ject)j(when)g(passed)h(in.)75 692 y(The)g(call)g(allo)q(cates)f(suc)o(h)i (ob)r(ject)g(and)f(asso)q(ciates)h(it)f(with)f(the)i(handle.)24 b(The)16 b(handle)g(can)h(b)q(e)f(used)h(later)f(to)75 741 y(query)e(ab)q(out)g(the)h(status)f(of)g(the)g(comm)o(unication)d(or)i(w)o (ait)h(for)f(its)h(completion.)158 791 y(A)21 b(sender)h(should)f(not)g(up)q (date)g(an)o(y)f(part)h(of)f(a)h(send)h(bu\013er)f(after)g(a)g(non)o(blo)q(c) o(king)e(send)j(op)q(eration)75 841 y(returned,)14 b(and)e(un)o(til)g(the)h (send)h(completes.)j(A)c(receiv)o(er)h(should)e(not)h(access)h(or)e(up)q (date)i(an)o(y)e(part)g(of)g(a)h(receiv)o(e)75 891 y(bu\013er)i(after)f(a)g (non)o(blo)q(c)o(king)e(receiv)o(e)j(op)q(eration)f(returned,)h(and)f(un)o (til)f(the)h(receiv)o(e)i(completes.)75 1007 y Fs(1.8.3)49 b(Comm)o(unication)14 b(Completion)75 1084 y Fx(The)j(completion)e(of)h(a)h (send)g(op)q(eration)g(indicates)g(that)g(the)g(sender)i(is)d(no)o(w)h(free)g (to)g(up)q(date)g(the)g(lo)q(cations)75 1134 y(in)d(the)h(send)h(bu\013er,)f (or)g(an)o(y)f(other)h(lo)q(cation)f(that)h(can)f(b)q(e)i(referenced)h(b)o(y) d(the)h(send)h(op)q(eration.)k(It)15 b(do)q(es)g(not)75 1183 y(indicate)c(that)f(the)h(message)g(has)g(b)q(een)g(receiv)o(ed;)i(rather,)f (it)e(ma)o(y)e(ha)o(v)o(e)j(b)q(een)h(bu\013ered)g(b)o(y)e(the)h(comm)o (unication)75 1233 y(subsystem.)21 b(Ho)o(w)o(ev)o(er,)15 b(if)g(a)f Fu(synchronous)f Fx(mo)q(de)h(send)h(w)o(as)g(used,)h(the)f(completion)e(of)i (the)g(send)h(op)q(eration)75 1283 y(indicates)d(that)h(a)f(matc)o(hing)e (receiv)o(e)j(w)o(as)g(initiated,)e(and)h(that)g(the)h(message)e(will)g(ev)o (en)o(tually)h(b)q(e)h(receiv)o(ed)g(b)o(y)75 1333 y(this)g(matc)o(hing)e (receiv)o(e.)158 1383 y(The)k(completion)e(of)h(a)h(receiv)o(e)h(op)q (eration)e(indicates)i(that)e(the)i(receiv)o(er)g(is)f(no)o(w)f(free)h(to)g (access)i(the)e(lo-)75 1432 y(cations)i(in)f(the)h(receiv)o(e)h(bu\013er,)g (whic)o(h)e(con)o(tain)h(the)g(receiv)o(ed)h(message,)e(or)h(an)o(y)f(other)h (lo)q(cation)f(that)h(can)75 1482 y(b)q(e)e(referenced)i(b)o(y)d(the)h (receiv)o(e)h(op)q(eration.)23 b(It)15 b(do)q(es)h(not)g(indicate)f(that)h (the)g(matc)o(hing)d(send)j(op)q(eration)g(has)75 1532 y(completed)d(\(but)i (indicates,)e(of)h(course,)g(that)g(the)h(send)g(w)o(as)e(initiated\).)75 1679 y Fu(MPI)p 144 1679 V 15 w(WAIT)21 b(\(handle,)f(status\))117 1729 y Ft(in)14 b Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 1774 y(out)e Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1851 y Fx(A)k(call)e(to)i Fu(MPI)p 407 1851 V 15 w(WAIT)e Fx(returns)j(when)f (the)g(send)h(op)q(eration)e(iden)o(ti\014ed)g(b)o(y)h Fu(handle)e Fx(is)h(complete.)25 b(If)16 b(the)75 1901 y(comm)o(unication)9 b(ob)r(ject)j(asso)q(ciated)h(with)f(this)g(handle)g(w)o(as)g(created)i(b)o (y)d(a)h(non)o(blo)q(c)o(king)f(send)i(or)f(receiv)o(e)h(call,)75 1950 y(then)i(the)f(ob)r(ject)h(is)e(deallo)q(cated)h(b)o(y)g(the)h(call)e (to)g Fu(MPI)p 951 1950 V 15 w(WAIT)g Fx(and)h(the)h(handle)e(b)q(ecomes)i(n) o(ull.)158 2000 y(The)i(call)f(returns)j(in)d Fu(status)g Fx(information)d (on)k(the)g(completed)g(op)q(eration.)26 b(The)18 b(status)f(ob)r(ject)h(for) e(a)75 2050 y(receiv)o(e)c(op)q(eration)e(can)h(b)q(e)g(queried)g(using)f (the)h(functions)g(describ)q(ed)h(in)e(Section)h(1.3.1.)k(The)c(status)g (parameter)75 2100 y(is)g(not)h(used)g(or)f(up)q(dated)h(for)f(a)h(send)g(op) q(eration,)f(and)h(an)f(arbitrary)g(v)n(alue)g(ma)o(y)e(b)q(e)j(passed)h(for) e(this)g(parameter)75 2150 y(in)i(the)i(call.)158 2278 y Fo(Discussion:)158 2324 y Ft(Do)e(w)o(e)g(agree)g(on)h(this?)k(\(this)c(implies)h(some)e (restriction)i(on)e(the)h(implemen)o(tation\))75 2503 y Fu(MPI)p 144 2503 V 15 w(TEST)21 b(\(handle,)f(flag,)h(status\))117 2553 y Ft(in)14 b Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 2599 y(out)36 b Fu(flag)651 b Ft(logical)117 2644 y(out)14 b Fu(status)629 b Ft(status)13 b(ob)r(ject)p eop %%Page: 14 14 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(14)158 45 y(A)12 b(call)e(to)i Fu(MPI)p 392 45 14 2 v 15 w(TEST)e Fx(returns)k Fu(flag=true)9 b Fx(if)i(the)h(op)q (eration)f(iden)o(ti\014ed)h(b)o(y)f Fu(handle)g Fx(is)g(complete.)17 b(In)11 b(suc)o(h)75 95 y(case,)g(the)g(status)f(ob)r(ject)h(is)f(set)h(to)e (con)o(tain)h(information)d(on)i(the)i(completed)e(op)q(eration;)i(if)e(the)h (comm)o(unication)75 145 y(ob)r(ject)15 b(w)o(as)e(created)j(b)o(y)e(a)f(non) o(blo)q(c)o(king)g(send)h(or)g(receiv)o(e,)h(then)g(it)e(is)h(deallo)q(cated) g(and)f(the)i(handle)f(b)q(ecomes)75 195 y(n)o(ull.)38 b(The)21 b(call)f(returns)i Fu(flag=false)p Fx(,)e(otherwise.)40 b(In)20 b(suc)o(h)i(case,)h(the)e(v)n(alue)f(of)h(the)g(status)h(ob)r(ject)f(is)75 244 y(unde\014ned.)158 373 y Fo(Discussion:)158 423 y Ft(W)m(e)13 b(could)i(disp)q(ose)f(of)f(the)g(\015ag)h(parameter)f(and)h(add)g(an)f (additional)j(query)e(function)h(for)d(status)i(ob)r(jects.)158 555 y Fu(MPI)p 227 555 V 15 w(TEST)c Fx(is)i(a)f(lo)q(cal,)f(non)o(blo)q(c)o (king)g(op)q(eration.)17 b(If)11 b(rep)q(eatedly)h(called)f(for)g(an)g(op)q (eration)h(that)f(is)g(enabled,)75 605 y(it)j(m)o(ust)e(ev)o(en)o(tually)i (succeed.)158 655 y(The)g(return)h(status)g(ob)r(ject)f(for)g(a)f(receiv)o(e) i(op)q(eration)f(carries)g(information)d(that)j(can)g(b)q(e)g(accessed)i (using)75 704 y(the)i(functions)f(describ)q(ed)i(in)e(Section)h(1.3.1.)26 b(The)18 b(status)g(argumen)o(t)e(is)h(not)g(used)h(or)f(up)q(dated)h(for)f (a)g(send)75 754 y(op)q(eration)d(and)f(an)h(arbitrary)g(v)n(alue)f(can)h(b)q (e)h(passed)g(for)e(this)h(parameter)g(in)f(the)i(call.)158 804 y(In)i(m)o(ultithreaded)f(en)o(vironmen)o(t,)g(the)h(use)h(of)e(a)h(blo)q (c)o(king)e(receiv)o(e)k(op)q(eration)d(\()p Fu(MPI)p 1556 804 V 16 w(WAIT)p Fx(\))g(ma)o(y)f(allo)o(w)75 854 y(the)e(op)q(erating)f (system)h(to)f(desc)o(hedule)i(the)f(blo)q(c)o(k)o(ed)g(thread)g(and)f(sc)o (hedule)i(another)f(thread)g(for)f(execution,)h(if)75 904 y(suc)o(h)h(is)f(a) o(v)n(ailable.)i(The)e(use)h(of)f(a)f(non)o(blo)q(c)o(king)g(receiv)o(e)i(op) q(eration)f(\()p Fu(MPI)p 1265 904 V 15 w(TEST)p Fx(\))g(allo)o(ws)e(the)j (user)g(to)f(sc)o(hedule)75 954 y(alternativ)o(e)h(activities)f(within)g(a)h (single)g(thread)g(of)f(execution.)75 1053 y(Example:)140 1136 y Fu(CALL)21 b(MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 1186 y(IF\(rank.EQ.0\))e(THEN)249 1236 y(CALL)i(MPI_ISEND\(handle,)e(a\(1\),)h (10,)i(MPI_REAL,)d(1,)j(tag,)f(comm,)g(ierr\))249 1286 y(****)g(do)h(some)f (computation)e(to)j(mask)f(latency)f(****)249 1335 y(CALL)h (MPI_WAIT\(handle,)e(status,)h(ierr\))445 1385 y(ELSE)249 1435 y(CALL)h(MPI_IRECV\(handle,)e(a\(1\),)h(15,)i(MPI_REAL,)d(0,)j(tag,)f(comm,)g (ierr\))249 1485 y(****)g(do)h(some)f(computation)e(to)j(mask)f(latency)f (****)249 1535 y(CALL)h(MPI_WAIT\(handle,)e(status,)h(ierr\))119 1584 y(END)h(IF)158 1667 y Fx(The)16 b(functions)f Fu(MPI)p 494 1667 V 15 w(WAIT)g Fx(and)g Fu(MPI)p 760 1667 V 15 w(TEST)f Fx(can)i(b)q(e)g(used)g(to)f(complete)g(b)q(oth)g(sends)i(and)e(receiv)o(es;) i(they)75 1717 y(will)12 b(b)q(e)j(also)e(used)i(to)f(complete)f(an)o(y)g (other)i(non)o(blo)q(c)o(king)d(comm)o(unication)f(call)i(pro)o(vided)h(b)o (y)f(MPI.)158 1767 y(Non)o(blo)q(c)o(king)g(comm)o(unicatio)o(ns)f(cannot)i (b)q(e)g(implemen)o(ted)e(as)i(blo)q(c)o(king)f(comm)o(unicatio)o(ns,)e(ev)o (en)k(in)e(the)75 1817 y Fu(synchronous)f Fx(comm)o(uni)o(cation)f(mo)q(de.) 17 b(Consider)d(the)h(follo)o(wing)c(example:)140 1900 y Fu(CALL)21 b(MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 1950 y(IF\(rank.EQ.0\))e(THEN)249 2000 y(CALL)i(MPI_ISEND\(handle1,)d(a\(1\),)j(count,)g(MPI_REAL,)f(1,)h (tag1,)g(comm,)g(ierr\))249 2049 y(CALL)g(MPI_ISEND\(handle2,)d(a\(2\),)j (count,)g(MPI_REAL,)f(1,)h(tag2,)g(comm,)g(ierr\))249 2099 y(CALL)g(MPI_WAIT\(handle1,)e(status,)h(ierr\))249 2149 y(CALL)h (MPI_WAIT\(handle2,)e(status,)h(ierr\))424 2199 y(ELSE)249 2249 y(CALL)h(MPI_IRECV\(handle2,)d(a\(2\),)j(count,)g(MPI_REAL,)f(0,)h (tag2,)g(comm,)g(ierr\))249 2298 y(CALL)g(MPI_IRECV\(handle1,)d(a\(1\),)j (count,)g(MPI_REAL,)f(0,)h(tag1,)g(comm,)g(ierr\))249 2348 y(CALL)g(MPI_WAIT\(handle2,)e(status,)h(ierr\))249 2398 y(CALL)h (MPI_WAIT\(handle1,)e(status,)h(ierr\))119 2448 y(END)h(IF)158 2531 y Fx(The)c(co)q(de)h(is)f(guaran)o(teed)g(to)g(execute)h(correctly)m(,)g (ev)o(en)g(in)e(an)h(implem)o(en)o(tation)d(that)j(do)q(es)g(not)g(bu\013er) 75 2581 y(messages)f(\(the)h(only)e(resource)k(requiremen)o(t)d(is)g(the)g (abilit)o(y)f(to)h(ha)o(v)o(e)g(t)o(w)o(o)g(p)q(ending)g(comm)o(unicatio)o (ns\).)23 b(On)75 2631 y(the)14 b(other)h(hand,)e(the)i(co)q(de)g(will)d (deadlo)q(c)o(k)i(if)f(blo)q(c)o(king)g(send)h(and)g(receiv)o(es)i(are)e (used.)p eop %%Page: 15 15 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(15)75 45 y Fs(1.8.4)49 b(Multiple)17 b(Completions)75 122 y Fx(It)f(is)g(con)o(v)o(enien)o(t)h(to)f(b)q(e)h(able)e(to)h(w)o(ait)g (for)g(the)g(completion)f(of)g(an)o(y)h(or)g(all)f(the)i(op)q(erations)f(in)g (a)g(set,)h(rather)75 172 y(than)e(ha)o(ving)g(to)g(w)o(ait)f(for)h(a)g(sp)q (eci\014c)i(message.)22 b(A)16 b(call)e(to)h Fu(MPI)p 1125 172 14 2 v 15 w(WAITANY)f Fx(or)h Fu(MPI)p 1426 172 V 16 w(TESTANY)e Fx(can)j(b)q(e)g(used)g(to)75 221 y(w)o(ait)d(for)g(the)i(completion)d(of)h (one)h(out)f(of)h(sev)o(eral)g(op)q(erations;)f(a)h(call)f(to)g Fu(MPI)p 1336 221 V 15 w(WAITALL)g Fx(or)g Fu(MPI)p 1634 221 V 15 w(TESTALL)g Fx(can)75 271 y(b)q(e)i(used)f(to)g(w)o(ait)f(for)h(all)e(p) q(ending)i(op)q(erations)g(in)g(a)f(list.)158 400 y Fo(Discussion:)158 449 y Fm(MPI)p 220 449 12 2 v 13 w(TESTALL)f Fo(is)j(new.)75 629 y Fu(MPI)p 144 629 14 2 v 15 w(WAITANY)20 b(\(count,)h(array)p 618 629 V 14 w(of)p 676 629 V 16 w(handles,)f(index,)g(status\))117 679 y Ft(in)14 b Fu(count)675 b Ft(list)14 b(length)g(\(in)o(teger\))117 725 y(in)g Fu(array)p 275 725 V 15 w(of)p 334 725 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o(unication)i(ob)r(jects)117 770 y(out)f Fu(index)651 b Ft(index)14 b(of)f(handle)i(for)d(op)q(eration)j (that)e(completed)i(\(in)o(teger\))117 816 y(out)f Fu(status)629 b Ft(return)13 b(status)h(ob)r(ject)158 892 y Fx(Blo)q(c)o(ks)g(un)o(til)f (one)h(of)f(the)h(op)q(erations)g(asso)q(ciated)h(with)e(the)i(comm)o(uni)o (cation)c(handles)j(in)f(the)h(arra)o(y)g(has)75 942 y(completed.)19 b(Returns)c(the)g(index)g(of)e(that)i(handle)f(in)g(the)h(arra)o(y)m(,)e(and) i(returns)h(the)f(status)g(of)f(that)g(op)q(eration)75 992 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f(with)g(the)g(status.)158 1042 y(The)23 b(successful)h(execution)f(of)f Fu(MPI)p 768 1042 V 15 w(WAITANY\(count,)d(array)p 1220 1042 V 15 w(of)p 1279 1042 V 15 w(handles,)h(index,)h(status\))g Fx(has)75 1091 y(the)f(same)f(e\013ect)i(as)e(the)h(successful)h(execution)g(of)d Fu(MPI)p 1017 1091 V 16 w(WAIT\(array)p 1253 1091 V 13 w(of)p 1310 1091 V 15 w(handles[i],)i(status\))p Fx(,)f(where)75 1141 y Fu(i)c Fx(is)f(the)i(v)n(alue)e(returned)j(b)o(y)e Fu(index)p Fx(.)20 b(If)14 b(more)g(then)i(one)f(op)q(eration)g(is)g(enabled)g(and)g (can)g(terminate,)f(one)h(is)75 1191 y(arbitrarily)e(c)o(hosen.)19 b(There)c(is)f(no)f(requiremen)o(t)h(that)g(the)h(c)o(hoice)f(satisfy)g(an)o (y)f(fairness)h(criterion.)158 1319 y Fo(Discussion:)158 1369 y Ft(Need)f(to)g(revisit)i(fainess)f(here,)f(together)g(with)h(fairness)g(in) g(general.)75 1549 y Fu(MPI)p 144 1549 V 15 w(TESTANY\()20 b(count,)h(array)p 618 1549 V 14 w(of)p 676 1549 V 16 w(handles,)f(index,)g (status\))117 1599 y Ft(in)14 b Fu(count)675 b Ft(list)14 b(length)g(\(in)o (teger\))117 1644 y(in)g Fu(array)p 275 1644 V 15 w(of)p 334 1644 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o (unication)i(ob)r(jects)117 1690 y(out)f Fu(index)651 b Ft(index)18 b(of)f(handle)h(for)f(op)q(eration)h(that)f(completed,)i(or)e(-1)g(if)905 1740 y(none)d(completed)g(\(in)o(teger\))117 1786 y(out)g Fu(status)629 b Ft(return)13 b(status)h(ob)r(ject)158 1862 y Fx(Causes)20 b(either)h(one)f(or)f(none)h(of)f(the)h(op)q(erations)g(asso)q(ciated)g(with) f(the)h(comm)o(unication)c(handles)k(to)75 1912 y(return.)i(In)15 b(the)h(former)e(case,)h(it)g(returns)h(the)g(index)f(of)f(that)h(handle)g (in)f(the)h(arra)o(y)m(,)f(and)h(returns)i(the)e(status)75 1961 y(of)g(that)g(op)q(eration)h(in)f(the)h(ob)r(ject)g(asso)q(ciated)h (with)e(the)h(status.)24 b(In)15 b(the)h(latter)g(case,)g(it)f(returns)i(a)f (v)n(alue)f(of)75 2011 y(-1)e(in)g Fu(index)g Fx(and)h Fu(status)e Fx(is)i(unde\014ned.)19 b(Lik)o(e)13 b Fu(MPI)p 925 2011 V 15 w(TESTT)p Fx(,)f(this)i(is)g(a)f(non)o(blo)q(c)o(king)f(op)q(eration,)i (that)f(returns)75 2061 y(immediatl)o(y;)j(and)h(if)g(one)h(busy)f(w)o(aits)g (with)h Fu(MPI)p 896 2061 V 15 w(TESTANY)p Fx(,)d(and)i(some)g(op)q(eration)g (on)h(the)g(list)f(is)g(enabled,)75 2111 y(then)e Fu(MPI)p 239 2111 V 15 w(TESTANY)d Fx(will)g(ev)o(en)o(tually)i(return)h Fu(flag)21 b(=)g(true)p Fx(.)75 2208 y Fu(MPI)p 144 2208 V 15 w(WAITALL\()f(count,)h(array)p 618 2208 V 14 w(of)p 676 2208 V 16 w(handles,)f(array)p 998 2208 V 14 w(of)p 1056 2208 V 16 w(status\))117 2258 y Ft(in)14 b Fu(count)675 b Ft(lists)14 b(length)g(\(in)o(teger\))117 2303 y(in)g Fu(array)p 275 2303 V 15 w(of)p 334 2303 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to) e(comm)o(unication)i(ob)r(jects)117 2349 y(out)f Fu(array)p 299 2349 V 14 w(of)p 357 2349 V 16 w(status)445 b Ft(arra)o(y)13 b(of)g(status)g(ob)r(jects)158 2425 y Fx(Blo)q(c)o(ks)19 b(un)o(til)e(all)g (comm)o(unicati)o(on)e(op)q(erations)k(asso)q(ciated)g(with)f(handles)g(in)g (the)g(list)g(complete,)g(and)75 2475 y(return)f(the)f(status)g(of)f(all)f (these)j(op)q(erations.)22 b(Both)16 b(arra)o(ys)g(ha)o(v)o(e)f(the)h(same)f (n)o(um)o(b)q(er)f(of)h(v)n(alid)f(en)o(tries.)24 b(The)75 2525 y Fu(i)p Fx(-th)14 b(en)o(try)g(in)g Fu(array)p 434 2525 V 14 w(of)p 492 2525 V 16 w(status)e Fx(is)i(set)h(to)e(the)i(return)g (status)g(of)e(the)h Fu(i)p Fx(-th)g(op)q(eration.)75 2622 y Fu(MPI)p 144 2622 V 15 w(TESTALL\(count,)19 b(array)p 596 2622 V 15 w(of)p 655 2622 V 15 w(handles,)h(flag,)h(array)p 1107 2622 V 14 w(of)p 1165 2622 V 16 w(status\))117 2672 y Ft(in)14 b Fu(count)675 b Ft(lists)14 b(length)g(\(in)o(teger\))p eop %%Page: 16 16 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(16)117 45 y Ft(in)14 b Fu(array)p 275 45 14 2 v 15 w(of)p 334 45 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to) e(comm)o(unication)i(ob)r(jects)117 90 y(out)f Fu(flag)673 b Ft(\(logical\))117 135 y(out)14 b Fu(array)p 299 135 V 14 w(of)p 357 135 V 16 w(status)445 b Ft(arra)o(y)13 b(of)g(status)g(ob)r(jects) 158 211 y Fx(Returns)19 b Fu(flag)i(=)g(true)d Fx(if)f(all)g(comm)o(unicati)o (ons)f(asso)q(ciated)i(with)g(handles)g(in)g(the)h(arra)o(y)e(ha)o(v)o(e)h (com-)75 261 y(pleted.)32 b(In)19 b(this)f(case,)i(eac)o(h)f(status)g(en)o (try)h(is)e(set)h(to)f(the)h(status)h(of)d(the)j(corresp)q(onding)f(comm)o (unicatio)o(n.)75 311 y(Otherwise,)e Fu(flag)k(=)g(false)14 b Fx(is)i(returned)h(and)e(the)h(v)n(alues)f(of)f(the)i(status)g(en)o(tries)h (are)e(unde\014ned.)24 b(This)15 b(is)g(a)75 361 y(non)o(blo)q(c)o(king)d(op) q(eration)i(that)f(returns)i(immediatl)o(y;)10 b(if)j(one)h(busy)f(w)o(aits)g (with)h Fu(MPI)p 1434 361 V 15 w(TESTALL)p Fx(,)d(and)i(all)g(op)q(era-)75 411 y(tions)h(in)f(the)i(list)e(b)q(ecome)h(enabled,)g(then)g Fu(MPI)p 842 411 V 15 w(TESTALL)f Fx(will)f(ev)o(en)o(tually)h(return)i Fu(flag)21 b(=)h(true)p Fx(.)158 510 y(Example:)140 640 y Fu(CALL)f (MPI_COMM_RANK\(comm,)d(rank,)j(ierr\))140 690 y(IF\(rank)g(<)g(2\))206 740 y(THEN)249 790 y(CALL)g(MPI_ISEND\(handle,)e(a,)i(n,)g(MPI_REAL,)f(2,)i (tag,)f(comm,)f(ierr\))249 840 y(****)h(do)h(some)f(computation)e(to)j(mask)f (latency)f(****)249 889 y(CALL)h(MPI_WAIT\(handle,)e(status,)h(ierr\))206 939 y(ELSE)249 989 y(CALL)h(MPI_IRECV\(handle_li)o(st\(0\))o(,)e(a,)i(n,)h (MPI_REAL,)e(0,)h(tag,)g(comm,)g(ierr\))249 1039 y(CALL)g (MPI_IRECV\(handle_li)o(st\(1\))o(,)e(b,)i(n,)h(MPI_REAL,)e(1,)h(tag,)g (comm,)g(ierr\))249 1089 y(****)g(do)h(some)f(computation)e(to)j(mask)f (latency)f(****)249 1139 y(CALL)h(MPI_WAITANY\(2,)e(handle_list,)h(index,)g (status,)g(ierr\))249 1188 y(IF\(index.EQ.0\))293 1238 y(THEN)337 1288 y(****)g(handle)h(message)f(from)h(process)g(0)g(****)337 1338 y(CALL)f(MPI_WAIT\(handle_list)o(\(1\),)e(status,)j(ierr\))337 1388 y(****)f(handle)h(message)f(from)h(process)g(1)g(****)293 1437 y(ELSE)337 1487 y(****)f(handle)h(message)f(from)h(process)g(1)g(****) 337 1537 y(CALL)f(MPI_WAIT\(handle_list)o(\(0\),)e(status,)j(ierr\))337 1587 y(****)f(handle)h(message)f(from)h(process)g(0)g(****)271 1637 y(END)g(IF)140 1686 y(END)h(IF)158 1767 y Fx(The)c(calls)f(in)o(tro)q (duced)h(in)f(this)h(subsection)h(can)e(b)q(e)i(used)f(to)f(w)o(ait)g(or)h (test)g(for)f(the)h(completion)e(of)h(an)75 1817 y(arbitrary)f(mix)f(of)h (non)o(blo)q(c)o(king)f(comm)o(unicatio)o(n)f(calls,)i(including)g(a)g(mix)e (sends)k(and)e(receiv)o(es)i(and)f(of)e(an)o(y)75 1866 y(additional)d(non)o (blo)q(c)o(king)h(comm)o(unicati)o(on)e(calls)j(pro)o(vided)f(b)o(y)h(MPI.) 158 1916 y(The)e Fu(status)f Fx(argumen)o(t)f(in)h Fu(MPI)p 682 1916 V 16 w(WAITANY)f Fx(and)h Fu(MPI)p 1006 1916 V 15 w(WAITALL)g Fx(and)g(the)h Fu(array)p 1443 1916 V 15 w(of)p 1502 1916 V 15 w(status)f Fx(argumen)o(t)f(in)75 1966 y Fu(MPI)p 144 1966 V 15 w(WAITALL)i Fx(and)h Fu(MPI)p 471 1966 V 15 w(TESTALL)f Fx(are)i(not)f(used)i(or)e(up)q(dated)h(if)f(the)h(completing)d(comm)o (unication)f(is)j(a)h(send.)75 2016 y(If)h(all)e(op)q(erations)i(in)g(the)g (list)g(pro)o(vided)g(b)o(y)f(the)i(argumen)o(t)e Fu(array)p 1175 2016 V 14 w(of)p 1233 2016 V 16 w(handles)f Fx(are)i(send)h(op)q (erations,)f(then)75 2066 y(an)f(arbitrary)f(v)n(alue)h(can)g(b)q(e)g(passed) h(to)f(the)g Fu(status)f Fx(\(resp.)19 b Fu(array)p 1181 2066 V 15 w(of)p 1240 2066 V 15 w(status)p Fx(\))13 b(argumen)o(t.)158 2194 y Fo(Discussion:)158 2244 y Ft(Is)g(that)g(agreed?)75 2463 y Fw(1.9)59 b(Prob)r(e)20 b(and)f(Cancel)75 2554 y Fx(The)h Fu(MPI)p 235 2554 V 15 w(PROBE)e Fx(and)h Fu(MPI)p 530 2554 V 15 w(IPROBE)f Fx(op)q(eration)h(allo)o(ws)f(incoming)f(messages)j(to)f(b)q (e)h(c)o(hec)o(k)o(ed)g(for,)g(without)75 2604 y(actually)15 b(receiving)g(them.)23 b(The)15 b(user)i(can)f(then)g(decide)g(ho)o(w)g(to)f (receiv)o(e)i(them,)d(based)i(on)g(the)g(information)75 2654 y(returned)g(b)o(y)d(the)i(prob)q(e)g(\(basically)m(,)d(the)i(information)d (returned)16 b(b)o(y)e Fu(status)p Fx(\).)j(In)d(particular,)f(the)i(user)g (ma)o(y)75 2704 y(allo)q(cate)e(memory)f(for)h(the)i(receiv)o(e)g(bu\013er,)f (according)g(to)g(the)h(length)e(of)h(the)g(prob)q(ed)h(message.)p eop %%Page: 17 17 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(17)158 45 y(The)13 b Fu(MPI)p 311 45 14 2 v 15 w(CANCEL)e Fx(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o(unicatio)o (ns)d(to)i(b)q(e)h(cancelled.)19 b(This)12 b(is)g(required)h(for)75 95 y(clean)o(up.)20 b(P)o(osting)14 b(a)h(send)g(or)g(a)f(receiv)o(e)i(ties)f (user)g(resources)i(\(send)f(or)e(receiv)o(e)i(bu\013ers\),)g(and)e(a)h (cancel)g(ma)o(y)75 145 y(b)q(e)g(needed)g(to)f(free)g(these)i(resources)g (gracefully)m(.)158 273 y Fo(Discussion:)158 323 y Ft(Made)j(one)g(c)o(hange) h(from)e(appro)o(v)o(ed)i(draft)f(on)g(prob)q(e:)29 b(remo)o(v)o(ed)19 b(the)g(datat)o(yp)q(e)h(argumen)o(t)f(from)g(prob)q(e.)34 b(If)75 373 y(generalized)15 b(datat)o(yp)q(e)f(are)e(adopted,)i(then)f(it)g (seems)g(more)f(natural)i(to)f(pro)o(vide)h(the)e(argumen)o(t)i(in)f(the)g (status)g(deco)q(de)75 423 y(function.)75 602 y Fu(MPI)p 144 602 V 15 w(IPROBE\(source,)19 b(tag,)i(comm,)g(flag,)f(status\))117 652 y Ft(in)14 b Fu(source)653 b Ft(source)13 b(rank,)h(or)f Fm(MPI)p 1273 652 12 2 v 13 w(ANY)p 1346 652 V 13 w(SOURCE)d Ft(\(in)o(teger\))117 698 y(in)k Fu(tag)719 b Ft(tag)13 b(v)n(alue)h(or)f Fm(MPI)p 1221 698 V 13 w(ANY)p 1294 698 V 13 w(TAG)f Ft(\(in)o(teger\))117 743 y(in)i Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 789 y(out)h Fu(flag)673 b Ft(\(logical\))117 835 y(out)14 b Fu(status)629 b Ft(status)13 b(ob)r(ject)158 911 y Fu(MPI)p 227 911 14 2 v 15 w(IPROBE)h Fx(returns)i Fu(flag)21 b(=)h(true)14 b Fx(if)g(there)i(is)f(a)g(message)f(that)h(can)h(b)q(e)f(receiv)o(ed)i(and)d (that)h(matc)o(hes)75 961 y(the)d(pattern)g(sp)q(eci\014ed)g(b)o(y)f(the)h (parameters)f Fu(source)p Fx(,)f Fu(tag)p Fx(,)g(and)h Fu(comm)p Fx(.)16 b(It)11 b(returns)i Fu(flag)21 b(=)g(false)p Fx(,)10 b(otherwise.)75 1011 y(If)k Fu(MPI)p 186 1011 V 15 w(IPROBE)f Fx(returns)k Fu(flag)k(=)g(true)p Fx(,)13 b(then)j(the)f(source)h(and)e(tag)g (of)g(the)h(message)f(matc)o(hed)g(are)h(returned)75 1060 y(in)c(the)i (status)g(ob)r(ject.)18 b(These)13 b(are)g(the)f(same)f(v)n(alues)h(that)g(w) o(ould)f(ha)o(v)o(e)h(b)q(een)h(returned)g(b)o(y)f(a)g(call)f(to)h Fu(MPI)p 1776 1060 V 15 w(RECV)75 1110 y Fx(executed)k(at)e(the)g(same)f(p)q (oin)o(t)g(in)h(the)g(program.)158 1160 y(These)19 b(v)n(alues)e(can)h(b)q(e) g(deco)q(ded)h(from)d(the)i(return)h(status)f(ob)r(ject)h(using)e(the)h (status)h(query)f(functions)75 1210 y(describ)q(ed)e(in)d(Section)h(1.3.1.)j (The)d(return)h(status)g(ob)r(ject)f(is)g(unde\014ned)h(if)e Fu(flag=false)p Fx(.)158 1260 y(The)20 b(call)e(to)h Fu(MPI)p 455 1260 V 15 w(PROBE)f Fx(do)q(es)i(not)f(carry)h(a)f(datat)o(yp)q(e)g (argumen)o(t:)28 b(the)19 b(message)g(t)o(yp)q(e)h(ma)o(y)d(not)i(b)q(e)75 1310 y(kno)o(wn)13 b(when)h(the)f(prob)q(e)h(is)g(executed.)19 b(Accordingly)m(,)13 b(the)g(prob)q(e)h(call)f(do)q(es)h(not)f(return)h(in)f (the)h(status)g(ob)r(ject)75 1359 y(the)j(length)f(of)g(the)h(prob)q(ed)g (message,)f(and)g Fu(MPI)p 870 1359 V 15 w(GET)p 951 1359 V 15 w(LEN)g Fx(cannot)g(bb)q(e)h(used)h(after)e(a)g(call)g(to)g Fu(MPI)p 1701 1359 V 15 w(PROBE)f Fx(to)75 1409 y(query)g(on)f(the)i(length)e (of)g(the)h(prob)q(ed)h(message.)k(\(Remem)o(b)q(er)13 b(that)h(the)i(length) e(of)g(a)g(message,)g(in)h(elemen)o(ts,)75 1459 y(dep)q(end)g(on)f(the)h(t)o (yp)q(es)g(of)f(these)i(elemen)o(ts.\))i(Instead,)d(a)f(di\013eren)o(t)h (status)g(deco)q(ding)g(function)e(that)i(pro)o(vides)75 1509 y(the)f(datat)o(yp)q(e)h(as)f(argumen)o(t)e(can)i(b)q(e)h(used)g(in)e(this)h (case.)75 1606 y Fu(MPI)p 144 1606 V 15 w(PROBE)p 269 1606 V 15 w(LEN\(status,)19 b(datatype,)h(count\))117 1656 y Ft(in)14 b Fu(status)653 b Ft(status)13 b(ob)r(ject)117 1701 y(in)h Fu(datatype)609 b Ft(message)13 b(datat)o(yp)q(e)117 1747 y(out)h Fu(count)651 b Ft(n)o(um)o(b)q(er)14 b(of)e(en)o(tries)i(in)g(message)g(\(in) o(teger\))158 1823 y Fx(Returns)h(the)f(length)g(of)f(the)h(message)f(that)h (w)o(as)g(last)f(prob)q(ed)i(with)e Fu(status)f Fx(as)i(an)g(argumen)o(t.)j (The)d(call)75 1873 y(is)g(erroneous)h(if)e Fu(datatype)f Fx(do)q(es)j(not)f (matc)o(h)e(the)j(t)o(yp)q(e)f(of)g(the)g(message.)158 2001 y Fo(Discussion:)158 2047 y Ft(Message)19 b(passing)g(in)g(MPI)e(can)i(b)q(e) f(implemen)o(ted)i(without)e(app)q(ending)j(t)o(yp)q(e)d(information)i(to)d (messages.)32 b(A)75 2093 y(message)14 b(is)g(merely)h(a)f(string)g(of)g(b)o (ytes)g(and)g(the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h(a)e (sequence)i(of)e(t)o(yp)q(ed)i(elemen)o(ts)75 2138 y(is)g(done)g(using)h(the) f(datat)o(yp)q(e)h(information)g(pro)o(vided)h(b)o(y)e(the)f(lo)q(cal)i(comm) o(unication)i(call.)k(The)15 b(abilit)o(y)i(to)d(use)h(suc)o(h)75 2184 y(implemen)o(tation)21 b(strategy)e(is)g(deemed)g(to)f(b)q(e)g(an)h(imp) q(ortan)o(t)g(goal.)33 b(In)19 b(suc)o(h)f(implemen)o(tation)q(,)k(when)c(a)g (message)75 2230 y(arriv)o(es,)c(it)f(is)g(not)h(kno)o(wn)f(ho)o(w)g(man)o(y) g(elemen)o(ts)h(it)g(con)o(tains,)g(or)f(ev)o(en)g(ho)o(w)g(m)o(uc)o(h)g (storage)h(is)f(needed)h(to)f(receiv)o(e)h(that)75 2275 y(message)g(\(b)q (ecause)g(of)e(p)q(ossible)k(represen)o(tation)f(con)o(v)o(ersion)g(in)f(a)f (heterogeneous)i(en)o(vironmen)o(t\).)k(The)13 b(length)h(of)f(the)75 2321 y(incoming)18 b(message)e(can)g(b)q(e)g(computed)h(b)o(y)f Fm(MPI)p 818 2321 12 2 v 13 w(PROBE)p 931 2321 V 12 w(LEN)e Ft(from)i(the)f(b)o(yte)h(length)h(of)f(the)g(message)g(and)g(from)g(the)75 2367 y(datat)o(yp)q(e)e(parameter.)158 2412 y(The)h(curren)o(t)g(solution)i (sa)o(v)o(es)f(us)f(the)g(need)h(for)e(one)i(additional)i(w)o(ord)d(p)q(er)g (message)h(that)f(w)o(ould)h(otherwise)g(b)q(e)75 2458 y(needed)e(to)f (transfer)g(the)g(message)h(length)g(\(in)g(elemen)o(ts\))g(with)g(the)f (message.)158 2504 y Fo(Need)h(b)q(e)h(discussed)g(b)o(y)g(MPIF)158 2549 y Ft(W)m(e)f(ma)o(y)g(w)o(an)o(t)f(a)h(call)h(to)e Fm(MPI)p 616 2549 V 13 w(PROBE)p 729 2549 V 12 w(LEN)g Ft(to)g(b)q(e)h(signi\014can)o (t)i(not)e(just)g(after)f(prob)q(e,)i(but)f(also)g(after)g(receiv)o(e,)g(for) 75 2595 y(consistency)h(reasons.)p eop %%Page: 18 18 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(18)158 45 y Ft(It's)13 b(tempting)i(to)f(merge)g Fm(MPI)p 615 45 12 2 v 13 w(GET)p 688 45 V 13 w(LEN)e Ft(and)j Fm(MPI)p 909 45 V 13 w(PROBE)p 1022 45 V 12 w(LEN)d Ft(in)o(to)j(one)f(call.) 21 b(E.g.,)13 b(alw)o(a)o(ys)i(pro)o(vide)g(a)f(datat)o(yp)q(e)75 91 y(argumen)o(t,)f(ev)o(en)g(though)g(it)g(is)g(redundan)o(t)h(after)e(a)g (receiv)o(e.)18 b(But)13 b(then)f(w)o(e)g(imp)q(ose)i(an)f(additional)i (burden)f(on)e(regular)75 136 y(receiv)o(es.)158 269 y Fx(A)i(subsequen)o(t)j (receiv)o(e)e(executed)h(with)e(the)h(same)f(con)o(text,)g(and)h(the)g (source)g(and)f(tag)g(returned)i(b)o(y)e(the)75 319 y(call)f(to)h Fu(MPI)p 271 319 14 2 v 15 w(IPROBE)f Fx(will)f(receiv)o(e)k(the)e(message)g (that)g(w)o(as)g(matc)o(hed)f(b)o(y)h(the)g(prob)q(e,)h(if)e(no)g(other)i(in) o(terv)o(ening)75 369 y(receiv)o(e)d(o)q(ccurred)g(after)f(the)g(prob)q(e.)18 b(If)10 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o(ultithreaded,)g(it)g(is)h (the)g(user)g(resp)q(onsibilit)o(y)75 418 y(to)j(ensure)h(that)f(the)h(last)e (condition)g(holds.)158 547 y Fo(Discussion:)158 592 y Ft(MPI)d(guaran)o (tees)h(that)e(successiv)o(e)j(messages)e(sen)o(t)g(from)f(a)h(source)g(to)g (a)g(destination)i(within)f(the)f(same)g(con)o(text)g(are)75 638 y(receiv)o(ed)i(in)f(the)g(order)g(they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h(either)f(explicitly)j(or)c (implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75 684 y(to)i(manage)h(messages) g(b)q(et)o(w)o(een)f(eac)o(h)h(pair)g(of)f(pro)q(cesses,)h(for)f(eac)o(h)g (con)o(text.)17 b Fm(MPI)p 1309 684 12 2 v 13 w(PROBE)10 b Ft(returns)j(information)h(on)f(the)75 729 y(\014rst)i(matc)o(hing)g(message) g(in)h(this)f(FIF)o(O;)e(this)j(will)g(also)f(b)q(e)g(the)f(message)h(receiv) o(ed)h(b)o(y)f(the)f(\014rst)h(subsequen)o(t)h(receiv)o(e)75 775 y(with)d(the)h(same)f(source,)g(tag)g(and)h(con)o(text)f(as)h(the)f (message)g(matc)o(hed)h(b)o(y)f Fm(MPI)p 1261 775 V 13 w(PROBE)p Ft(.)75 955 y Fu(MPI)p 144 955 14 2 v 15 w(PROBE\()21 b(source,)f(tag,)h (comm,)g(datatype,)f(status\))117 1005 y Ft(in)14 b Fu(source)653 b Ft(source)13 b(rank,)h(or)f Fm(MPI)p 1273 1005 12 2 v 13 w(ANY)p 1346 1005 V 13 w(SOURCE)d Ft(\(in)o(teger\))117 1050 y(in)k Fu(tag)719 b Ft(tag)13 b(v)n(alue,)h(or)f Fm(MPI)p 1232 1050 V 13 w(ANY)p 1305 1050 V 13 w(TAG)e Ft(\(in)o(teger\))117 1096 y(in)j Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 1142 y(in)h Fu(datatype)609 b Ft(assumed)14 b(t)o(yp)q(e)f(of)g(data)g(in)h (message)g(\(status\))117 1187 y(out)g Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1264 y Fu(MPI)p 227 1264 14 2 v 15 w(PROBE)g Fx(b)q(eha)o(v)o(es)i(lik)o(e)f Fu(MPI)p 663 1264 V 15 w(IPROBE)f Fx(except)i(that)g(it)f(is)g(a)g(blo)q(c)o(king)f(call)g(whic)o(h)i(returns)g (only)f(after)g(a)75 1313 y(matc)o(hing)e(message)i(has)g(b)q(een)h(found.) 158 1442 y Fo(Discussion:)158 1487 y(CANCEL)f(has)h(b)q(een)g(c)o(hanged)f (since)h(last)h(draft)75 1667 y Fu(MPI)p 144 1667 V 15 w(CANCEL\(handle\))117 1717 y Ft(in)e Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)158 1793 y Fx(A)g(call)f(to)g Fu(MPI)p 404 1793 V 15 w(CANCEL)g Fx(marks)f(for)h(cancellation)g(a)h(p)q(ending)g(non)o(blo)q(c)o (king)e(comm)o(unicatio)o(n)f(op)q(eration)75 1843 y(\(send)i(or)f(receiv)o (e\).)21 b(The)14 b(cancel)h(call)e(is)h(non-blo)q(c)o(king,)f(and)h(lo)q (cal.)k(It)c(returns)i(immedia)o(tly)m(,)10 b(p)q(ossibly)k(b)q(efore)75 1893 y(the)g(comm)o(unication)d(is)j(actually)f(cancelled.)158 1943 y(It)f(is)g(still)f(necessary)j(to)e(complete)g(a)f(comm)o(unication)e (that)j(has)g(b)q(een)h(mark)o(ed)e(for)h(cancellation,)f(using)h(a)75 1992 y(call)f(to)h Fu(MPI)p 267 1992 V 15 w(WAIT)f Fx(or)g Fu(MPI)p 495 1992 V 16 w(TEST)f Fx(\(or)i(an)o(y)g(of)f(the)h(deriv)o(ed)h (op)q(erations\).)k(If)12 b(the)g(op)q(eration)g(has)g(b)q(een)h(cancelled,) 75 2042 y(then)f(information)d(ab)q(out)j(that)g(will)e(b)q(e)i(returned)i (in)d(the)h(status)h(argumen)o(t)d(of)h(the)i(op)q(eration)e(that)h (completes)75 2092 y(the)17 b(comm)o(unicatio)o(n.)22 b(If)16 b(a)g(comm)o(unication)d(is)j(mark)o(ed)f(for)h(cancellation,)f(then)i(a)f Fu(MPI)p 1539 2092 V 15 w(WAIT)g Fx(call)f(for)h(that)75 2142 y(comm)o(unication)f(is)j(guaran)o(teed)h(to)g(return,)h(irresp)q(ectiv)o(e)g (of)e(the)h(status)g(of)f(other)h(pro)q(cesses;)k(similarly)16 b(if)75 2192 y Fu(MPI)p 144 2192 V 15 w(TEST)11 b Fx(is)h(rep)q(eatedly)i (called)e(in)f(a)h(busy)h(w)o(ait)e(lo)q(op)g(for)h(a)g(cancelled)h(comm)o (unicati)o(on,)c(then)k Fu(MPI)p 1699 2192 V 15 w(TEST)e Fx(will)75 2241 y(ev)o(en)o(tually)i(b)q(e)i(successful.)158 2291 y(Either)21 b(the)g(cancellation)f(succeeds,)k(or)c(the)h(comm)o(unication)c(succeeds,)24 b(but)d(not)f(b)q(oth.)38 b(If)20 b(a)g(send)75 2341 y(is)e(mark)o(ed)f(for)h (cancellation,)g(then)h(it)e(m)o(ust)g(b)q(e)i(the)g(case)g(that)f(either)h (the)g(send)g(completes)f(normally)m(,)d(in)75 2391 y(whic)o(h)e(case)h(the)g (message)e(sen)o(t)i(w)o(as)f(receiv)o(ed)h(at)f(the)h(destination)f(pro)q (cess,)h(or)f(that)g(the)h(send)g(is)f(successfully)75 2441 y(cancelled,)g(in)e(whic)o(h)h(case)h(no)f(part)g(of)f(the)i(message)f(w)o (as)g(receiv)o(ed)h(at)f(the)h(destination.)k(Then,)12 b(an)o(y)g(matc)o (hing)75 2491 y(receiv)o(e)k(has)f(to)f(b)q(e)h(satis\014ed)h(b)o(y)e (another)h(send.)22 b(If)14 b(a)g(receiv)o(e)i(is)f(mark)o(ed)e(for)h (cancellation,)g(then)h(it)g(m)o(ust)e(b)q(e)75 2540 y(the)i(case)h(that)f (either)h(the)f(receiv)o(e)i(completes)d(normally)m(,)e(or)j(that)g(the)g (receiv)o(e)h(is)f(successfully)h(cancelled,)f(in)75 2590 y(whic)o(h)g(case)g (no)g(part)g(of)f(the)h(receiv)o(e)h(bu\013er)g(is)e(altered.)21 b(Then,)15 b(an)o(y)g(matc)o(hing)d(send)k(has)f(to)f(b)q(e)i(satis\014ed)f (b)o(y)75 2640 y(another)f(receiv)o(e.)p eop %%Page: 19 19 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(19)75 45 y Fu(MPI)p 144 45 14 2 v 15 w(IS)p 203 45 V 15 w(CANCELLED\(status,)19 b(flag\))117 95 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(ob)r(ject)117 140 y(in)g Fu(flag)697 b Ft(\(logical\))158 216 y Fx(Returns)16 b Fu(flag)21 b(=)g(true)14 b Fx(if)g(the)i(comm)o(unicatio)o(n)c(asso)q (ciated)k(with)e(the)i(return)g(status)f(ob)r(ject)h(w)o(as)f(can-)75 266 y(celled)h(successfully)m(.)24 b(In)16 b(suc)o(h)g(case,)h(all)e(other)h (\014elds)g(of)f Fu(status)f Fx(\(suc)o(h)j(as)f Fu(length)e Fx(or)i Fu(tag)p Fx(\))f(are)h(unde\014ned.)75 316 y(Returns)h Fu(flag)k(=)h(false)p Fx(,)15 b(otherwise.)27 b(If)16 b(a)g(receiv)o(e)i(op)q (eration)e(migh)o(t)f(b)q(e)i(cancelled)g(then)g(one)g(should)f(call)75 366 y Fu(MPI)p 144 366 V 15 w(IS)p 203 366 V 15 w(CANCELLED)11 b Fx(\014rst,)j(to)f(c)o(hec)o(k)h(whether)g(the)g(op)q(eration)f(w)o(as)f (cancelled,)i(b)q(efore)g(c)o(hec)o(king)f(on)g(the)g(other)75 415 y(v)n(alues)h(of)f(the)h(return)h(status.)158 544 y Fo(Implemen)o(tation) e(note:)158 589 y Ft(The)g(cancel)h(op)q(eration)h(ma)o(y)e(b)q(e)g(exp)q (ensiv)o(e)j(to)c(implemen)o(t)j(on)f(some)f(systems.)75 809 y Fw(1.10)59 b(P)n(ersistent)19 b(comm)n(un)o(ication)e(objects)75 978 y Fo(Discussion:)158 1028 y(there)e(are)h(some)e(c)o(hanges)h(here)g(not) g(y)o(et)g(appro)o(v)o(ed)f(b)o(y)h(MPIF)158 1161 y Fx(Often)o(times)i(a)g (comm)o(uni)o(cation)d(with)j(the)h(same)e(parameter)h(list)g(is)g(rep)q (eatedly)h(executed)h(within)d(the)75 1210 y(inner)f(lo)q(op)e(of)h(a)f (parallel)g(computation.)18 b(In)c(suc)o(h)h(situation,)e(it)h(ma)o(y)e(b)q (e)j(p)q(ossible)f(to)g(optimize)f(the)i(comm)o(u-)75 1260 y(nication)g(b)o(y)h(binding)g(once)h(the)f(list)g(of)g(comm)o(unicatio)o(n)e (parameters)i(to)g(the)h(comm)o(unicatio)o(n)d(ob)r(ject)j(and,)75 1310 y(then,)f(using)e(rep)q(eatedly)i(the)g(comm)o(unication)c(handle)j(to)f (initiate)h(and)f(complete)h(messages.)21 b(The)16 b(comm)o(u-)75 1360 y(nication)g(handle)g(th)o(us)h(created)h(can)f(b)q(e)g(though)o(t)f(of) g(as)h(a)f(comm)o(unication)d(p)q(ort)k(or)g(a)f(\\half-c)o(hannel")f(.)26 b(It)75 1410 y(do)q(es)18 b(not)f(pro)o(vide)g(the)g(full)f(functionalit)o(y) f(of)i(a)g(con)o(v)o(en)o(tional)f(c)o(hannel,)h(since)h(there)g(is)f(no)g (binding)f(of)g(the)75 1459 y(send)i(p)q(ort)g(to)f(the)h(receiv)o(e)h(p)q (ort:)25 b(this)17 b(construct)i(allo)o(w)d(to)h(reduce)i(the)f(o)o(v)o (erhead)g(of)e(the)i(comm)o(unication)75 1509 y(b)q(et)o(w)o(een)e(pro)q (cessor)h(and)e(comm)o(unicatio)o(n)d(con)o(troller,)j(but)g(not)g(the)h(o)o (v)o(erhead)f(of)f(comm)o(unication)e(b)q(et)o(w)o(een)75 1559 y(one)i(comm)o(unication)d(con)o(troller)i(and)h(another.)158 1609 y(A)g(comm)o(unication)c(ob)r(ject)15 b(is)f(created)h(using)f(one)g(of) f(the)i(four)e(follo)o(wing)e(calls.)18 b(These)d(calls)f(in)o(v)o(olv)o(e)e (no)75 1659 y(comm)o(unication.)158 1709 y(The)k(function)f Fu(MPI)p 478 1709 V 15 w(CREATE)p 625 1709 V 14 w(SEND)g Fx(creates)i(a)e (comm)o(uni)o(cation)d(ob)r(ject)k(for)f(a)g(standard)h(mo)q(de)e(send)i(op-) 75 1758 y(eration,)d(and)h(binds)g(to)g(it)f(all)g(the)h(parameters)g(of)g(a) f(send)i(op)q(eration.)75 1855 y Fu(MPI)p 144 1855 V 15 w(CREATE)p 291 1855 V 15 w(SEND\(handle,)k(start,)h(count,)h(datatype,)f(dest,)h(tag,)g (comm\))117 1905 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1950 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 1995 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(sen)o (t)g(\(in)o(teger\))117 2040 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 2085 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2130 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2175 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 2251 y Fx(The)h Fu(handle)e Fx(parameter)i(should)f(b)q(e)h(n)o(ull)f(when)h(the)g (call)f(is)g(executed.)25 b(The)16 b(call)f(asso)q(ciates)i(it)e(to)g(the)75 2301 y(newly)f(created)h(comm)o(unication)c(ob)r(ject.)158 2430 y Fo(Discussion:)158 2475 y Ft(Tw)o(o)h(c)o(hanges:)158 2521 y(1.)29 b Fm(buffer)p 339 2521 12 2 v 12 w(handle)15 b Ft(is)j(replaced)g(with)g Fm(start,)f(count,)h(datatype)p Ft(,)c(to)j(b)q(e)h (in)g(sync)g(with)f(the)h(new)f(datat)o(yp)q(e)75 2567 y(prop)q(osal.)158 2612 y(2.)25 b(T)m(o)q(ok)15 b(out)h Fm(persistenc)o(e)p Ft(.)22 b(Do)q(es)16 b(not)g(seem)f(v)o(ery)h(useful)h(here.)25 b(Instead,)17 b(I)e(in)o(tro)q(duce)i(in)g(the)f(last)g(section,)75 2658 y(a)e(\\univ)o(ersal")j(comm)o(unication)f(function,)f(whic)o(h)g(w)o(ould)g (not)f(usually)i(used)f(directly)g(b)o(y)f(programmers,)h(but)f(can)h(b)q(e) 75 2704 y(used)f(to)f(de\014ne)h(the)f(other)g(comm)o(unication)j(functions.) p eop %%Page: 20 20 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(20)158 45 y(The)14 b(function)f Fu(MPI)p 474 45 14 2 v 16 w(CREATE)p 622 45 V 14 w(RSEND)g Fx(creates)i(a)e(comm)o (unication)e(ob)r(ject)j(for)f(a)h(ready)g(mo)q(de)e(send)j(op)q(era-)75 95 y(tion.)75 192 y Fu(MPI)p 144 192 V 15 w(CREATE)p 291 192 V 15 w(RSEND\(handle,)k(start,)h(count,)h(datatype,)f(dest,)g(tag,)h(comm\)) 117 242 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j (ob)r(ject)117 287 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send) i(bu\013er)117 333 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(sen)o(t)g(\(in)o(teger\))117 378 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 424 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 469 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 515 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 591 y Fx(The)f(function)g Fu(MPI)p 475 591 V 15 w(CREATE)p 622 591 V 14 w(SSEND)f Fx(creates)i(a)f(comm)o(unicati)o(on)d(ob)r(ject)j (for)g(a)f(sync)o(hon)o(uous)h(mo)q(de)f(send)75 641 y(op)q(eration.)75 738 y Fu(MPI)p 144 738 V 15 w(CREATE)p 291 738 V 15 w(SSEND\(handle,)19 b(start,)h(count,)h(datatype,)f(dest,)g(tag,)h(comm\))117 788 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 833 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)117 879 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(sen)o(t)g(\(in)o(teger\))117 924 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 970 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1015 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1061 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 1137 y Fx(The)f(function)g Fu(MPI)p 475 1137 V 15 w(CREATE)p 622 1137 V 14 w(RECV)f Fx(creates)j(a)e(comm)o(unicatio)o(n)d(ob)r(ject)k (for)e(a)h(receiv)o(e)h(op)q(eration.)75 1234 y Fu(MPI)p 144 1234 V 15 w(CREATE)p 291 1234 V 15 w(RECV\(handle,)k(start,)h(count,)h (datatype,)f(source,)g(tag,)h(comm\))117 1284 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1330 y(out)e Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i (bu\013er)117 1375 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(receiv)o(ed)h(\(in)o(teger\))117 1421 y(in)f Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 1466 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(source)h(or)f(MPI)p 1321 1466 12 2 v 14 w(ANY)p 1422 1466 V 13 w(SOUR)o(CE)g(\(in)o(teger\))117 1512 y(in)h Fu(tag)719 b Ft(message)13 b(tag)h(or)f(MPI)p 1285 1512 V 14 w(ANY)p 1386 1512 V 13 w(T)m(A)o(G)f(\(in)o(teger\))117 1557 y(in)i Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 1634 y Fx(A)i(comm)o(uni)o(cation)d(\(send)j(or)f(receiv)o(e\))i(that)f(uses) g(a)g(prede\014ned)h(handle)e(is)g(initiated)g(b)o(y)g(the)h(function)75 1683 y Fu(MPI)p 144 1683 14 2 v 15 w(START)p Fx(.)75 1780 y Fu(MPI)p 144 1780 V 15 w(START\(handle\))117 1830 y Ft(inout)e Fu(handle)597 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)158 1907 y Fx(The)i(comm)o(unication)c(ob)r(ject)19 b(asso)q(ciated)f(with)g Fu(handle)e Fx(should)h(b)q(e)i(an)e(ob)r(ject)i(that)e(w)o(as)h(created)h(b) o(y)75 1956 y(one)d(of)g(the)h(previous)f(four)g(functions,)g(so)g(that)h (all)d(the)j(comm)o(unication)c(parameters)j(are)h(already)e(de\014ned.)75 2006 y(A)g(send)g(can)g(b)q(e)g(started)h(pro)o(vided)f(that)f(the)h (previous)g(send)h(using)e(the)i(same)d(ob)r(ject)j(has)e(completed,)g(or)h (as)75 2056 y(so)q(on)i(as)f(the)h(ob)r(ject)h(is)e(created,)i(if)e(it)g(has) h(not)f(y)o(et)h(b)q(een)g(used)h(in)e(a)g(comm)o(unication.)23 b(In)16 b(addition,)g(if)f(the)75 2106 y(comm)o(unication)e(mo)q(de)i(is)g Fu(ready)g Fx(then)i(a)f(matc)o(hing)e(receiv)o(e)j(should)f(b)q(e)g(p)q (osted.)26 b(The)16 b(send)h(bu\013er)g(should)75 2156 y(not)d(b)q(e)g(up)q (dated)h(after)f(the)h(send)f(is)g(started,)h(un)o(til)e(the)h(op)q(eration)g (completes.)158 2205 y(A)i(receiv)o(e)i(can)e(b)q(e)h(started)h(pro)o(vided)e (that)g(the)h(preceding)g(receiv)o(e)h(using)e(the)h(same)e(ob)r(ject)i(has)f (com-)75 2255 y(pleted,)i(or)f(as)g(so)q(on)h(as)f(the)h(ob)r(ject)g(is)f (created,)i(if)d(it)h(has)g(not)g(y)o(et)g(b)q(een)i(used)f(in)e(a)h(comm)o (unication.)25 b(The)75 2305 y(receiv)o(e)15 b(bu\013er)g(should)f(not)g(b)q (e)g(accessed)i(after)e(the)h(receiv)o(e)g(is)f(started,)g(un)o(til)f(the)i (op)q(eration)f(completes.)158 2355 y(The)i(call)f(is)h(non)o(blo)q(c)o (king,)e(with)i(similar)d(seman)o(tics)j(as)f(the)i(non)o(blo)q(c)o(king)d (comm)o(unication)f(op)q(erations)75 2405 y(describ)q(ed)j(in)d(Section)h (1.8.)158 2455 y(A)d(comm)o(uni)o(cation)d(started)j(with)f(a)h(call)e(to)i Fu(MPI)p 935 2455 V 15 w(START)e Fx(is)i(completed)e(b)o(y)i(a)f(call)g(to)g Fu(MPI)p 1574 2455 V 15 w(WAIT)p Fx(,)f Fu(MPI)p 1764 2455 V 15 w(TEST)p Fx(,)75 2504 y(or)h(one)g(of)f(the)i(deriv)o(ed)f(functions)g (describ)q(ed)i(in)d(Section)h(1.8.4.)15 b(These)c(comm)o(unication)c (completion)h(functions)75 2554 y(do)13 b(not)g(deallo)q(cate)g(the)g(comm)o (unication)d(ob)r(ject,)j(and)g(this)g(can)g(b)q(e)h(reused)g(anew)f(b)o(y)g (a)g Fu(MPI)p 1578 2554 V 15 w(START)f Fx(call.)17 b(The)75 2604 y(ob)r(ject)j(need)h(b)q(e)f(explicitly)f(deallo)q(cated)g(b)o(y)h(a)f (call)g(to)g(the)h(function)f Fu(MPI)p 1336 2604 V 15 w(COMM)p 1439 2604 V 15 w(FREE)p Fx(,)g(b)q(elo)o(w.)34 b(W)m(e)20 b(th)o(us)75 2654 y(ha)o(v)o(e)e(t)o(yp)q(e)h(t)o(yp)q(es)g(of)e(comm)o(unication)e(ob)r (jects:)27 b Fv(p)q(ersisten)o(t)15 b Fx(ob)r(jects,)20 b(whic)o(h)e(are)g (allo)q(cated)g(b)o(y)g(a)g(call)f(to)75 2704 y Fu(MPI)p 144 2704 V 15 w(CREATE)p 291 2704 V 15 w(xxx)p Fx(,)i(and)g(are)g(explicitly)f (deallo)q(cated)h(b)o(y)g Fu(MPI)p 1100 2704 V 15 w(COMM)p 1203 2704 V 15 w(FREE)p Fx(,)e(and)i Fv(ephemeral)e Fx(ob)r(jects)j(that)p eop %%Page: 21 21 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(21)75 45 y(p)q(ersist)19 b(for)d(one)i(comm)o(unication)c (only:)24 b(they)18 b(are)f(created)i(b)o(y)e(a)g(non)o(blo)q(c)o(king)f (comm)o(unication)e(initiation)75 95 y(function,)f(and)h(are)g(freed)h(b)o(y) f(the)g(comm)o(unication)c(completion)j(call.)75 192 y Fu(MPI)p 144 192 14 2 v 15 w(COMM)p 247 192 V 15 w(FREE\(handle\))117 242 y Ft(inout)i Fu(handle)597 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)158 318 y Fx(Marks)g(the)h(comm)o(uni)o(cation)c(ob)r(ject)k(for)e (deallo)q(cation.)23 b(The)16 b(ob)r(ject)h(will)d(b)q(e)i(deallo)q(cated)g (when)h(there)75 368 y(are)d(no)g(p)q(ending)g(comm)o(unicatio)o(ns)e(in)o(v) o(olving)f(this)j(ob)r(ject,)g(at)g(whic)o(h)g(p)q(oin)o(t)f(the)i(handle)f (b)q(ecomes)g(n)o(ull.)158 418 y(The)19 b(call)f(is)h(non)o(blo)q(c)o(king,)f (and)h(the)h(ob)r(ject)f(mna)o(y)e(not)i(b)q(e)h(deallo)q(cated)e(when)i(the) f(call)f(returns.)35 b(It)75 468 y(is)18 b(p)q(ermissible)g(to)g(call)f Fu(MPI)p 546 468 V 16 w(COMM)p 650 468 V 14 w(FREE\(handle\))f Fx(after)j(a)f(comm)o(unicati)o(on)e(that)i(uses)h Fu(handle)e Fx(has)i(b)q(een)75 517 y(initiated,)g(but)h(b)q(efore)g(it)f(has)g (completed.)34 b(The)20 b(ob)r(ject)g(will)e(b)q(e)i(deallo)q(cated)f(after)h (the)f(comm)o(unication)75 567 y(completes.)f(It)c(is)f(erroneous)j(to)d(try) i(to)e(reuse)j(the)e(handle)g(for)g(a)f(subsequen)o(t)j(comm)o(unicatio)o(n.) 158 617 y(A)g(correct)i(in)o(v)o(o)q(cation)c(of)i(the)g(functions)g(describ) q(ed)i(in)e(this)g(section)g(will)f(o)q(ccur)i(in)e(a)h(sequence)i(of)d(the) 75 667 y(form)179 750 y Fv(Create)g Fx(\()p Fv(Start)g(Complete)p Fx(\))691 733 y Fk(\003)724 750 y Fv(F)l(ree)158 833 y Fx(If)e(the)h(same)f (comm)o(unicatio)o(n)e(ob)r(ject)j(is)f(used)i(in)e(sev)o(eral)h(concurren)o (t)h(threads,)f(it)f(is)h(the)g(user)g(resp)q(onsi-)75 883 y(bilit)o(y)e(to)i(co)q(ordinate)g(calls)g(so)g(that)g(the)g(correct)i (sequence)g(is)e(ob)q(ey)o(ed.)158 932 y(A)k(send)h(op)q(eration)f(initiated) f(with)h Fu(MPI)p 834 932 V 15 w(START)f Fx(can)h(b)q(e)h(matc)o(hed)e(with)h (an)o(y)f(receiv)o(e)j(op)q(eration)e(and,)75 982 y(lik)o(ewise,)f(a)g (receiv)o(e)h(op)q(eration)f(initiated)g(with)g Fu(MPI)p 946 982 V 15 w(START)f Fx(can)h(receiv)o(e)i(messages)e(generate)h(b)o(y)f(an)o (y)g(send)75 1032 y(op)q(eration.)75 1169 y Fw(1.11)59 b(Send-receive)75 1339 y Fo(Discussion:)158 1389 y(This)15 b(section)g(has)g(not)g(y)o(et)g(b)q (een)g(appro)o(v)o(ed)f(b)o(y)h(MPIF.)158 1521 y Fx(The)f Fv(send-receiv)o(e) c Fx(op)q(erations)k(com)o(bines)e(in)h(one)h(call)e(the)i(sending)g(of)e(a)h (message)g(to)h(one)f(destination)75 1571 y(and)h(the)g(receiving)h(of)e (another)h(message,)f(from)g(another)h(destination,)f(p)q(ossibly)h(the)g (same.)k(The)c Fv(exc)o(hange)75 1621 y Fx(op)q(erations)20 b(are)g(the)h(same)e(as)h(send-receiv)o(e,)j(except)e(that)f(the)h(send)f (bu\013er)h(and)f(the)g(receiv)o(e)i(bu\013er)f(are)75 1671 y(iden)o(tical.)j(A)17 b(send-receiv)o(e)h(op)q(eration)e(is)g(v)o(ery)h (useful)f(for)g(executing)h(a)f(shift)g(op)q(eration)g(across)h(a)f(c)o(hain) g(of)75 1720 y(pro)q(cesses.)25 b(If)15 b(blo)q(c)o(king)f(sends)j(and)e (receiv)o(es)i(are)e(used)i(for)e(suc)o(h)h(shift,)f(then)g(one)h(needs)h(to) e(order)h(correctly)75 1770 y(the)d(sends)g(and)f(receiv)o(es)i(\(e.g.)j(ev)o (en)c(sends,)g(next)g(receiv)o(es,)h(o)q(dd)e(receiv)o(es)i(\014rst,)e(next)h (sends\))h(so)e(as)g(to)g(prev)o(en)o(t)75 1820 y(cyclic)j(dep)q(endencies)i (that)e(lead)f(to)h(deadlo)q(c)o(k.)20 b(When)15 b(a)f(send-receiv)o(e)j(or)e (exc)o(hange)g(op)q(eration)g(is)f(used,)h(the)75 1870 y(comm)o(unication)g (subsystem)20 b(tak)o(es)f(care)h(of)e(these)i(issues.)35 b(Also,)19 b(a)g(send-receiv)o(e)i(op)q(eration)d(is)h(useful)g(to)75 1920 y(implemen)o(t)11 b(remote)j(pro)q(cedure)h(calls.)158 1969 y(A)i(message)f(sen)o(t)h(b)o(y)g(a)f(send-receiv)o(e)j(or)d(exc)o (hange)i(op)q(eration)e(can)h(b)q(e)g(receiv)o(ed)h(b)o(y)f(a)f(regular)h (receiv)o(e)75 2019 y(op)q(eration,)c(and)h(vice)g(v)o(ersa.)75 2116 y Fu(MPI)p 144 2116 V 15 w(SENDRECV\(send)p 445 2116 V 13 w(start,)21 b(send)p 699 2116 V 14 w(count,)g(send)p 954 2116 V 15 w(type,)g(dest,)f(recv)p 1318 2116 V 15 w(start,)h(recv)p 1574 2116 V 15 w(count,)f(recv)p 1829 2116 V 15 w(type,)75 2166 y(source,)g(tag,)h(comm,)g(status\))117 2216 y Ft(in)14 b Fu(send)p 253 2216 V 15 w(start)572 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 2262 y(in)g Fu(send)p 253 2262 V 15 w(count)572 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g (bu\013er)g(\(in)o(teger\))117 2307 y(in)g Fu(send)p 253 2307 V 15 w(type)594 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(send)f(bu\013er) 117 2353 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\)) 117 2399 y(out)e Fu(recv)p 277 2399 V 15 w(start)548 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i(bu\013er)g(\(c)o(hoice\))117 2444 y(in)g Fu(recv)p 253 2444 V 15 w(count)572 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(receiv)o(e)g(bu\013er)g(\(in)o(teger\))117 2490 y(in)g Fu(recv)p 253 2490 V 15 w(type)594 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(receiv)o(e)g(bu\013er)117 2536 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 2581 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2627 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 2673 y(out)h Fu(status)629 b Ft(status)13 b(ob)r(ject)p eop %%Page: 22 22 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(22)158 45 y(Execute)17 b(a)d(blo)q(c)o(king)g(send)i(and)f (receiv)o(e)h(op)q(eration.)21 b(Both)16 b(send)g(and)e(receiv)o(e)j(use)e (the)h(same)e(tag)h(v)n(alue)75 95 y(and)d(the)h(same)e(comm)o(unicator.)k (Ho)o(w)o(ev)o(er,)d(the)h(send)g(bu\013er)g(and)f(receiv)o(e)h(bu\013er)g (are)g(disjoin)o(t,)e(and)h(ma)o(y)e(ha)o(v)o(e)75 145 y(di\013eren)o(t)15 b(length)f(and)f(di\013eren)o(t)i(datat)o(yp)q(es.)158 273 y Fo(Discussion:)158 319 y Ft(F)m(or)d(a)f(shift)i(it's)e(more)h(natural)h (to)f(ha)o(v)o(e)g(same)g(t)o(yp)q(e)g(and)g(same)g(length)h(b)q(oth)g(for)e (send)h(and)h(receiv)o(e;)g(for)e(a)h(remote)75 364 y(pro)q(cedure)i(call,)g (the)g(additional)i(freedom)d(mak)o(es)h(sense.)75 544 y Fu(MPI)p 144 544 14 2 v 15 w(EXCHANGE\(start,)19 b(count,)h(datatype,)g(dest,)h (source,)f(tag,)h(comm,)g(status\))117 594 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(and)g(receiv)o(e)g(bu\013er)g(\(c)o (hoice\))117 640 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(and)f(receiv)o(e)h(bu\013er)g(\(in)o (teger\))117 685 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(send)f(and)h(receiv)o(e)g(bu\013er)117 731 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 777 y(in)e Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 822 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 868 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 914 y(out)h Fu(status)629 b Ft(status)13 b(ob)r(ject)158 990 y Fx(Execute)20 b(a)e(blo)q(c)o(king)f(send)i(and)f(receiv)o(e;)j(the)e(same) e(bu\013er)i(is)f(used)i(b)q(oth)e(for)g(the)g(send)i(and)e(for)f(the)75 1040 y(receiv)o(e.)158 1168 y Fo(Discussion:)158 1214 y Ft(I)10 b(omitted)h(the)g(ready)g(and)f(sync)o(hronous)j(mo)q(des)e(for)f (send/receiv)o(e)i(and)f(exc)o(hange.)17 b(Also,)11 b(I)f(omitted)h(non)o (blo)q(c)o(kin)q(g)75 1259 y(send/receiv)o(e)k(and)f(exc)o(hange.)k(Do)13 b(w)o(e)g(w)o(an)o(t)g(an)o(y)g(of)g(those?)158 1392 y Fx(The)d(seman)o(tics) f(of)f(a)h(send-receiv)o(e)j(op)q(eration)d(is)g(what)g(w)o(ould)g(obtain)f (if)h(the)h(caller)f(fork)o(ed)g(t)o(w)o(o)g(concurren)o(t)75 1442 y(threads,)18 b(one)g(to)f(execute)h(the)g(send,)g(and)f(one)g(to)g (execute)i(the)f(receiv)o(e,)h(follo)o(w)o(ed)c(b)o(y)i(a)g(join)f(of)g (these)i(t)o(w)o(o)75 1491 y(threads.)26 b(A)16 b(send-receiv)o(e)j(cannot)d (b)q(e)h(implem)o(en)o(ted)e(a)h(blo)q(c)o(king)f(send)i(follo)o(w)o(ed)d(b)o (y)i(a)g(blo)q(c)o(king)f(receiv)o(e)j(or)75 1541 y(a)e(blo)q(c)o(king)e (receiv)o(e,)k(follo)o(w)o(ed)c(b)o(y)i(a)f(blo)q(c)o(king)g(send.)25 b(Consider)16 b(the)h(follo)o(wing)c(co)q(de,)k(where)g(t)o(w)o(o)e(pro)q (cesses)75 1591 y(exc)o(hange)g(messages:)75 1682 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1732 y(IF)g(\(rank.EQ.0\))42 b(THEN)140 1782 y(CALL)21 b(MPI_SENDRECV\(send_b)o(uff,)d(count,)j(MPI_REAL,) f(0,)h(recv_buff,)f(count,)h(MPI_REAL,)271 1832 y(1,)h(tag,)e(comm,)h (status,)g(ierr\))424 1882 y(ELSE)140 1931 y(CALL)g(MPI_SENDRECV\(send_b)o (uff,)d(count,)j(MPI_REAL,)f(1,)h(recv_buff,)f(count,)h(MPI_REAL,)271 1981 y(0,)h(tag,)e(comm,)h(status,)g(ierr\))158 2073 y Fx(If)10 b(the)g(send)h(receiv)o(es)h(are)f(replaced)g(either)g(b)o(y)e(blo)q(c)o (king)h(send,)h(follo)o(w)o(ed)d(b)o(y)i(blo)q(c)o(king)f(receiv)o(e,)j(or)e (blo)q(c)o(king)75 2122 y(receiv)o(e,)j(follo)o(w)o(ed)e(b)o(y)h(blo)q(c)o (king)f(send,)i(then)g(the)g(co)q(de)g(will)e(deadlo)q(c)o(k.)17 b(On)c(the)g(other)f(hand,)g(send-receiv)o(e)j(can)75 2172 y(b)q(e)g(implem)o(en)o(ted)d(using)i(non)o(blo)q(c)o(king)f(sends)i(and)f (receiv)o(es.)20 b(Note)14 b(that)g(some)f(system)h(bu\013ering)g(is)g (required)75 2222 y(for)9 b(a)h(correct)h(implemen)o(tatio)o(n)c(of)i Fu(MPI)p 703 2222 V 15 w(EXCHANGE)p Fx(.)f(\(Consider)i(the)g(last)f (example,)g(with)g(send-receiv)o(e)j(replaced)75 2272 y(b)o(y)i(exc)o (hange.\))75 2409 y Fw(1.12)59 b(Null)21 b(p)n(ro)r(cesses)75 2583 y Fo(Discussion:)35 b Ft(This)14 b(section)g(has)f(not)h(b)q(een)f (review)o(ed)h(b)o(y)g(MPIF.)p eop %%Page: 23 23 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(23)158 45 y(In)16 b(man)o(y)d(instances,)k(it)e(is)h(con)o(v)o (enien)o(t)g(to)f(sp)q(ecify)i(a)e(\\dumm)o(y")d(source)17 b(or)f(destination)f(for)g(comm)o(uni-)75 95 y(cation.)27 b(This)17 b(simpli\014es)e(the)j(co)q(de)f(that)g(is)g(needed)i(for)d(dealing)g(with)h (b)q(oundaries,)h(e.g.,)e(in)g(the)i(case)g(of)e(a)75 145 y(noncircular)e (shift)g(done)g(with)f(calls)h(to)g(send-receiv)o(e.)158 195 y(The)d(sp)q(ecial)g(v)n(alue)f Fu(MPI)p 545 195 14 2 v 15 w(PROCNULL)f Fx(can)i(b)q(e)g(used)h(instead)f(of)f(a)g(rank)h(wherev)o(er)h (a)e(source)i(or)f(a)f(destination)75 244 y(parameter)k(is)g(required)h(in)f (a)g(call.)k(A)c(comm)o(unication)d(with)j(pro)q(cess)i Fu(MPI)p 1300 244 V 15 w(PROCNULL)c Fx(has)j(no)f(e\013ect:)20 b(a)14 b(send)75 294 y(to)k Fu(MPI)p 199 294 V 15 w(PROCNULL)e Fx(succeeds)k(and)d (returns)j(asap.)29 b(A)18 b(receiv)o(e)h(from)d Fu(MPI)p 1287 294 V 15 w(PROCNULL)g Fx(succeeds)k(and)d(returns)75 344 y(asap)e(with)h(no)f (mo)q(di\014cations)f(to)h(the)h(receiv)o(e)h(bu\013er.)24 b(When)16 b(a)f(receiv)o(e)i(with)e Fu(source)21 b(=)g(MPI)p 1645 344 V 15 w(PROCNULL)14 b Fx(is)75 394 y(executed)i(then)e(the)h(status)g (ob)r(ject)f(returns)i Fu(source)k(=)i(MPI)p 1070 394 V 15 w(PROCNULL)p Fx(,)12 b Fu(tag=MPI)p 1439 394 V 14 w(ANY)p 1519 394 V 15 w(TAG)h Fx(and)h Fu(count=0)p Fx(.)158 522 y Fo(Discussion:)158 568 y Ft(It)f(w)o(ould)h(b)q(e)f(nice)h(to)f(ha)o(v)o(e)g Fm(MPI)p 640 568 12 2 v 13 w(PROCNULL)d Ft(\\equiv)n(alenced")16 b(to)d(-1)g(and)g(to) g(group)p 1393 568 V 15 w(size,)h(so)f(that,)f(when)i(a)f(shift)g(is)75 613 y(executed)h(in)g(a)f(group)h(the)f(\014rst)g(and)h(last)f(pro)q(cessors) i(automatically)h(comm)o(unicate)e(with)g(the)f(n)o(ull)i(pro)q(cess.)158 659 y(A)e(c)o(hoice)i(is)g(to)e(sa)o(y)i(that)e(a)h(source/destinati)q(on)j (with)d(rank)g(-1)g(or)g(group)p 1278 659 V 15 w(size)g(is)g(alw)o(a)o(ys)h (the)f(n)o(ull)i(pro)q(cess.)k(W)m(e)75 705 y(then)13 b(lo)q(ose)i(some)e (error)g(catc)o(hing)i(capabilit)o(y)m(.)158 750 y(Another)f(c)o(hoice)g(is)f (to)g(sa)o(y)h(that)f(only)h(for)f(send-receiv)o(e)h(and)g(exc)o(hange.)158 796 y(A)d(third)h(c)o(hoice)g(is)g(to)f(liv)o(e)i(without)f(this)g(nicet)o(y) m(.)17 b(It)11 b(means)g(that)h(one)f(still)i(ha)o(v)o(e)f(to)f(execute)h (conditional)i(co)q(de:)j(\(if)75 842 y(m)o(yrank=0)c(then)g(m)o(yneigh)o(b)q (or=mpi)p 640 842 V 17 w(pro)q(cn)o(ull\).)19 b(But)13 b(this)h(co)q(de)f (need)g(b)q(e)f(executed)i(only)g(when)e(the)h(comm)o(unication)75 887 y(pattern)h(is)f(set,)g(and)h(the)f(conditional)j(b)q(oundary)f(co)q(de)e (is)h(not)f(needed)h(in)g(the)f(inner)i(lo)q(op.)75 1106 y Fw(1.13)59 b(Derived)19 b(datat)n(yp)r(es)75 1275 y Fo(Discussion:)158 1321 y(This)c(is)g(a)g(new)g(section,)h(and)e(the)h(material)g(has)h(not)e(y) o(et)i(b)q(een)e(discussed)h(b)o(y)g(MPIF.)158 1453 y Fx(Up)k(to)g(no)o(w,)g (all)f(p)q(oin)o(t)h(to)f(p)q(oin)o(t)h(comm)o(unicatio)o(n)d(in)o(v)o(olv)o (ed)i(only)g(con)o(tiguous)h(bu\013ers)h(con)o(taining)e(a)75 1503 y(sequence)c(of)d(elemen)o(ts)g(of)g(the)i(same)d(t)o(yp)q(e.)18 b(This)12 b(is)f(to)q(o)h(constraining)f(on)g(t)o(w)o(o)h(accoun)o(ts:)17 b(One)c(often)e(w)o(an)o(ts)h(to)75 1553 y(pass)h(messages)g(that)f(con)o (tain)g(v)n(alues)h(with)f(di\013eren)o(t)i(datat)o(yp)q(es)f(\(e.g.,)e(an)i (in)o(teger)g(coun)o(t,)f(follo)o(w)o(ed)f(b)o(y)i(coun)o(t)75 1603 y(real)h(n)o(um)o(b)q(ers\);)g(and)f(one)i(often)f(w)o(an)o(ts)g(to)g (send)h(noncon)o(tiguous)f(data)f(\(e.g.)19 b(a)14 b(subblo)q(c)o(k)g(of)f(a) h(matrix\).)j(One)75 1653 y(solution)12 b(is)h(to)g(pro)o(vide)g(functions)g (that)g(pac)o(k)g(noncon)o(tiguous)f(data)h(in)o(to)f(a)h(con)o(tiguous)g (bu\013er)h(at)f(the)g(sender)75 1702 y(site)k(and)f(unpac)o(k)g(it)g(bac)o (k)g(at)h(the)g(receiv)o(er)g(site.)26 b(This)16 b(has)h(the)g(disadv)n(an)o (tage)e(of)h(requiring)g(an)g(additional)75 1752 y(memory)h(to)h(memory)f (cop)o(y)i(op)q(eration)g(at)g(b)q(oth)g(sites,)i(ev)o(en)f(when)f(the)h (comm)o(unicatio)o(n)c(subsystem)k(has)75 1802 y(scatter-gather)d (capabilities.)k(Instead,)16 b(MPI)f(pro)o(vides)g(mec)o(hanisms)e(to)i(sp)q (ecify)h(more)e(general,)h(mixed)f(and)75 1852 y(noncon)o(tiguous)d(comm)o (unication)d(bu\013ers.)19 b(It)12 b(is)f(up)h(to)f(the)h(implemen)o(tation)c (to)k(decide)g(whether)h(data)e(should)75 1902 y(b)q(e)h(\014rst)g(pac)o(k)o (ed)g(in)f(a)g(con)o(tiguous)g(bu\013er)h(b)q(efore)g(b)q(een)g(transmitted,) f(or)h(whether)g(it)f(can)h(b)q(e)g(collected)g(directly)75 1951 y(from)g(where)j(it)f(resides.)158 2001 y(The)j(general)g(mec)o(hanisms) d(pro)o(vided)j(here)g(allo)o(w)e(to)i(transfer)g(directly)m(,)g(without)f (cop)o(ying,)g(ob)r(jects)i(of)75 2051 y(v)n(arious)11 b(shap)q(e)h(and)g (size.)18 b(It)11 b(is)h(not)f(assumed)h(that)f(the)h(MPI)g(library)f(is)h (cognizan)o(t)f(of)g(the)h(ob)r(jects)h(declared)f(in)75 2101 y(the)h(host)g(language;)e(th)o(us,)i(if)f(one)h(w)o(an)o(ts)f(to)g(transfer) i(a)e(structure,)j(or)d(an)g(arra)o(y)h(section,)g(it)f(will)f(b)q(e)i (necessary)75 2151 y(to)e(pro)o(vide)g(in)g(MPI)h(a)f(de\014nition)f(of)h(a)g (comm)o(unication)d(bu\013er)k(that)f(mimic)o(ks)e(the)i(de\014nition)g(of)g (the)h(structure)75 2201 y(or)e(arra)o(y)g(section)g(in)g(question.)17 b(These)11 b(facilities)e(can)i(b)q(e)f(used)h(b)o(y)f(library)f(designers)i (to)f(de\014ne)h(comm)o(unication)75 2250 y(functions)16 b(that)g(can)g (transfer)h(ob)r(jects)g(de\014ned)g(in)e(the)h(host)h(language)e({)g(b)o(y)h (deco)q(ding)g(their)g(de\014nitions)g(as)75 2300 y(a)o(v)n(ailable)f(in)i(a) g(sym)o(b)q(ol)e(table)j(or)f(a)g(dop)q(e)h(v)o(ector.)29 b(Suc)o(h)17 b(higher-lev)o(el)g(comm)o(unication)d(functions)j(are)h(not)75 2350 y(part)c(of)f(MPI.)158 2400 y(More)k(general)g(comm)o(unicatio)o(n)d (bu\013ers)k(are)f(sp)q(eci\014ed)h(b)o(y)e(replacing)h(the)g(basic)f(datat)o (yp)q(es)i(that)e(ha)o(v)o(e)75 2450 y(used)c(so)f(far,)f(with)h(deriv)o(ed)g (datat)o(yp)q(es,)h(that)f(are)g(constructed)i(from)c(basic)i(datat)o(yp)q (es)h(using)e(the)i(constructors)75 2499 y(describ)q(ed)f(in)e(this)h (section.)17 b(These)10 b(metho)q(ds)f(of)g(constructing)i(deriv)o(ed)e (datat)o(yp)q(es)h(can)g(b)q(e)g(applied)f(recursiv)o(ely)m(.)158 2549 y(A)14 b Fv(general)g(datat)o(yp)q(e)f Fx(is)g(an)h(opaque)g(ob)r(ject)h (that)f(sp)q(eci\014es)h(t)o(w)o(o)f(things:)137 2624 y Fn(\017)21 b Fx(A)14 b(sequence)i(of)d(basic)h(t)o(yp)q(es)137 2704 y Fn(\017)21 b Fx(A)14 b(sequence)i(of)d(in)o(teger)h(\(b)o(yte\))h (displacemen)o(ts)p eop %%Page: 24 24 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(24)158 45 y(The)18 b(displacemen)o(ts)f(are)h(not)f(required)h (to)g(b)q(e)g(p)q(ositiv)o(e,)f(nor)h(are)f(they)h(required)h(to)e(b)q(e)h (in)f(increasing)75 95 y(order;)d(the)h(order)f(of)g(items)e(ma)o(y)g(not)i (coincide)g(to)g(their)g(order)h(in)e(storage.)158 145 y(Let)179 228 y Fj(D)q(ataty)q(pe)g Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 234 y Fi(0)538 228 y Fj(;)7 b(disp)633 234 y Fi(0)651 228 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 234 y Fh(n)p Fk(\000)p Fi(1)897 228 y Fj(;)g(disp)992 234 y Fh(n)p Fk(\000)p Fi(1)1057 228 y Fx(\))p Fn(g)p Fj(;)75 311 y Fx(b)q(e)17 b(suc)o(h)g(general)g(datat)o (yp)q(e,)g(where)h Fj(ty)q(pe)764 317 y Fh(i)795 311 y Fx(are)f(basic)f(t)o (yp)q(es,)i(and)e Fj(disp)1257 317 y Fh(i)1288 311 y Fx(are)h(displacemen)o (ts.)25 b(This)17 b(general)75 361 y(datat)o(yp)q(e,)11 b(together)f(with)g (an)g(base)g(address)i Fu(start)p Fx(,)d(sp)q(eci\014es)j(a)d(comm)o (unication)e(bu\013er:)17 b(the)10 b(comm)o(unication)75 410 y(bu\013er)k(that)e(consists)i(of)e Fj(n)g Fx(en)o(tries,)h(where)h(the)f Fj(i)p Fx(-th)g(en)o(try)g(is)g(at)f(address)i Fu(start)5 b Fx(+)i Fj(disp)1502 416 y Fh(i)1529 410 y Fx(and)12 b(has)h(t)o(yp)q(e)g Fj(ty)q(pe)1849 416 y Fh(i)1863 410 y Fx(.)75 460 y(A)18 b(message)f(assem)o (bled)g(from)f(this)i(comm)o(unicatio)o(n)d(bu\013er)k(will)d(con)o(tain)h(a) h(sequence)h(of)e Fj(n)h Fx(v)n(alues,)g(where)75 510 y(v)n(alue)d Fj(i)h Fx(has)g(t)o(yp)q(e)g Fj(ty)q(pe)461 516 y Fh(i)476 510 y Fx(.)23 b(The)16 b(general)g(datat)o(yp)q(e)g(determines)g(the)g Fv(t)o(yp)q(e)h(signature)c Fx(of)i(the)h(message)g(\(i.e.)75 560 y(the)e(n)o(um)o(b)q(er)g(of)f(v)n(alues)h(carried)g(b)o(y)g(the)g (message)g(and)g(the)g(basic)g(t)o(yp)q(e)g(of)g(eac)o(h)g(v)n(alue.)158 610 y(W)m(e)h(can)i(use)f(a)g(handle)f(to)h(a)g(general)g(datat)o(yp)q(e)g (as)g(an)f(argumen)o(t)g(in)g(a)h(send)h(or)f(receiv)o(e)h(op)q(eration,)e (in)75 659 y(replacemen)o(t)j(of)g(a)g(basic)g(datat)o(yp)q(e)h(argumen)o(t.) 30 b(The)19 b(op)q(eration)f Fu(MPI)p 1253 659 14 2 v 15 w(SEND\(start,)h(1,) j(datatype,...\))75 709 y Fx(will)11 b(use)j(the)f(send)h(bu\013er)f (de\014ned)h(b)o(y)f(the)g(base)g(address)h Fu(start)e Fx(and)h(the)g (general)g(datat)o(yp)q(e)g(asso)q(ciated)g(with)75 759 y Fu(datatype)p Fx(;)d(it)h(will)e(generate)k(a)e(message)f(with)h(the)h(t)o(yp)q(e)g (signature)f(determined)g(b)o(y)g(the)h Fu(datatype)e Fx(argumen)o(t.)75 809 y Fu(MPI)p 144 809 V 15 w(RECV\(start,)20 b(1,)h(datatype,...\))15 b Fx(will)d(use)i(the)g(receiv)o(e)g(bu\013er)g(de\014ned)g(b)o(y)f(the)h (base)g(address)g Fu(start)75 859 y Fx(and)g(the)g(general)g(datat)o(yp)q(e)g (asso)q(ciated)h(with)f Fu(datatype)p Fx(.)158 909 y(General)k(datat)o(yp)q (es)h(can)f(b)q(e)g(used)h(in)f(all)e(send)j(and)f(receiv)o(e)h(op)q (erations.)31 b(W)m(e)17 b(address)j(later)e(in)f(Sec-)75 958 y(tion)c(1.13.3)f(the)j(case)g(where)g(the)f(second)h(argumen)o(t)e Fu(count)g Fx(has)h(v)n(alue)f Fj(>)f Fx(1.)158 1008 y(The)21 b(prede\014ned)h(basic)e(datat)o(yp)q(es)h(presen)o(ted)h(in)e(Section)g (1.2.1)e(are)j(particular)f(cases)h(of)f(a)g(general)75 1058 y(datat)o(yp)q(e.)h(Th)o(us,)15 b Fu(MPI)p 455 1058 V 15 w(INT)f Fx(is)h(a)f(prede\014ned)j(handle)d(to)h(the)g(datat)o(yp)q(e)h Fn(f)p Fx(\()p Fu(int)o Fj(;)7 b Fx(0\))p Fn(g)p Fx(,)13 b(with)i(one)g(en)o (try)g(of)f(t)o(yp)q(e)75 1108 y Fu(int)f Fx(and)h(displacemen)o(t)f(zero.)19 b(And)14 b(similarly)d(for)i(all)g(other)i(basic)f(datat)o(yp)q(es.)158 1158 y(The)f Fv(exten)o(t)d Fx(of)i(a)g(datat)o(yp)q(e)h(is)f(de\014ned)i(to) e(b)q(e)h(the)g(span)f(from)f(the)i(\014rst)g(b)o(yte)g(to)f(the)h(last)f(b)o (yte)h(o)q(ccupied)75 1207 y(b)o(y)h(en)o(tries)h(in)e(this)h(datat)o(yp)q (e,)g(rounded)g(to)g(satisfy)g(alignmen)o(t)d(requiremen)o(ts.)18 b(I.e.,)13 b(if)179 1290 y Fj(D)q(ataty)q(pe)g Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 1296 y Fi(0)538 1290 y Fj(;)7 b(disp)633 1296 y Fi(0)651 1290 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 1296 y Fh(n)p Fk(\000)p Fi(1)897 1290 y Fj(;)g(disp)992 1296 y Fh(n)p Fk(\000)p Fi(1)1057 1290 y Fx(\))p Fn(g)p Fj(;)75 1373 y(disp)151 1379 y Fh(r)181 1373 y Fx(=)12 b(min)294 1379 y Fh(j)319 1373 y Fj(disp)395 1379 y Fh(j)426 1373 y Fx(and)i Fj(disp)583 1379 y Fh(s)612 1373 y Fx(=)e(max)733 1379 y Fh(j)758 1373 y Fj(disp)834 1379 y Fh(j)865 1373 y Fx(then)179 1465 y Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))h(=)f Fj(disp)631 1471 y Fh(s)658 1465 y Fx(+)e Fj(siz)r(eof)t Fx(\()p Fj(ty)q(pe)909 1471 y Fh(s)930 1465 y Fx(\))f Fn(\000)h Fj(disp)1073 1471 y Fh(r)1091 1465 y Fx(;)-1028 b(\(1.1\))75 1556 y(if)8 b(furthermore,)i Fj(ty)q(pe)422 1562 y Fh(r)451 1556 y Fx(requires)g(alignmen)o(t)d(to)i(a)g (b)o(yte)h(address)h(that)e(is)g(is)g(a)g(m)o(ultiple)e(of)i(2)1509 1541 y Fh(k)1529 1556 y Fx(,)h(then)g Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))75 1606 y(is)k(rounded)g(up)g(to)g(the)g(next)h(m)o(ultiple)c(of)j(2) 785 1591 y Fh(k)805 1606 y Fx(.)158 1656 y(Example:)j(Assume)c(that)179 1739 y Fj(T)6 b(y)q(pe)p Fx(1)12 b(=)g Fn(f)p Fx(\()p Fu(double)o Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(8\))p Fn(g)75 1822 y Fx(\(a)15 b Fu(double)g Fx(at)g(displacemen)o(t)g(zero,)h (follo)o(w)o(ed)e(b)o(y)i(a)f Fu(char)g Fx(at)g(displacemen)o(t)g(eigh)o (t\).)23 b(Assume,)15 b(furthermore,)75 1872 y(that)f(doubles)g(ha)o(v)o(e)g (to)f(b)q(e)i(strictly)f(aligned)f(at)g(addresses)j(that)e(are)g(m)o(ultiple) e(of)h(eigh)o(t.)18 b(Then,)13 b(the)i(exten)o(t)g(of)75 1921 y(this)f(datat)o(yp)q(e)g(is)g(16)f(\(9)h(rounded)g(to)g(the)h(next)f(m)o (ultiple)e(of)h(8\).)75 2038 y Fs(1.13.1)49 b(Datat)o(yp)q(e)17 b(constructo)o(rs)75 2114 y Fx(The)h(simplest)f(datat)o(yp)q(e)i(constructor) g(is)f Fu(MPI)p 847 2114 V 15 w(TYPE)p 950 2114 V 15 w(CONTIGUOUS)e Fx(whic)o(h)i(allo)o(ws)f(to)g(replicate)i(a)f(datat)o(yp)q(e)75 2164 y(in)o(to)13 b(con)o(tiguous)h(lo)q(cations.)75 2261 y Fu(MPI)p 144 2261 V 15 w(TYPE)p 247 2261 V 15 w(CONTIGUOUS\(count,)k (oldtype,)i(newtype\))117 2311 y Ft(in)14 b Fu(count)675 b Ft(replication)16 b(coun)o(t)d(\(in)o(teger\))117 2357 y(in)h Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob)r(ject)117 2402 y(out)g Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g (ob)r(ject)158 2479 y Fu(newtype)e Fx(is)i(the)g(datat)o(yp)q(e)g(obtained)g (b)o(y)g(concatenating)g Fu(count)e Fx(copies)j(of)e Fu(oldtype)p Fx(.)j(P)o(adding)d(ma)o(y)f(b)q(e)75 2528 y(added)i(to)g(satisfy)g (alignemen)o(t)e(requiremen)o(ts)i(of)f(the)i(underlying)e(arc)o(hitecture.) 158 2578 y(Example:)k(let)d Fu(oldtype)e Fx(b)q(e)i(a)g(handle)g(to)f(the)i (datat)o(yp)q(e)179 2661 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)p eop %%Page: 25 25 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(25)75 45 y(with)14 b(exten)o(t)g(16,)f(and)h(let)g Fu(count)c Fx(=)i(3.)18 b(The)c(resulting)g(datat)o(yp)q(e)g(returned)i(b)o (y)e Fu(newtype)e Fx(is)179 128 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(32\))p Fj(;)g Fx(\()p Fu(char)m Fj(;)g Fx(40\))p Fn(g)p Fx(;)75 211 y(i.e.,)12 b(alternating)i Fu(double)e Fx(and)i Fu(char)f Fx(elemen)o(ts,)g(with)h(displacemen)o(ts)f(0)p Fj(;)7 b Fx(8)p Fj(;)g Fx(16)p Fj(;)g Fx(24)p Fj(;)g Fx(3)o(2)p Fj(;)g Fx(4)o(0.)158 261 y(In)14 b(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle)f(to)h(the)h(datat)o(yp)q(e)179 344 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 350 y Fi(0)311 344 y Fj(;)7 b(disp)406 350 y Fi(0)425 344 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 350 y Fh(n)p Fk(\000)p Fi(1)671 344 y Fj(;)g(disp)766 350 y Fh(n)p Fk(\000)p Fi(1)830 344 y Fx(\))p Fn(g)p Fj(;)75 427 y Fx(with)14 b(exten)o(t)g Fj(extent)p Fx(.)19 b(Then)14 b Fu(newtype)e Fx(is)i(a)g(handle)g(to)f(the)i(datat)o(yp)q(e)f(with)f Fu(count)c Fn(\001)f Fu(n)14 b Fx(en)o(tries)h(de\014ned)g(b)o(y:)179 518 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 524 y Fi(0)311 518 y Fj(;)7 b(disp)406 524 y Fi(0)425 518 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 524 y Fh(n)p Fk(\000)p Fi(1)671 518 y Fj(;)g(disp)766 524 y Fh(n)p Fk(\000)p Fi(1)830 518 y Fx(\))p Fj(;)g Fx(\()p Fj(ty)q(pe)957 524 y Fi(0)977 518 y Fj(;)g(disp)1072 524 y Fi(0)1099 518 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)1440 524 y Fh(n)p Fk(\000)p Fi(1)1504 518 y Fj(;)g(disp)1599 524 y Fh(n)p Fk(\000)p Fi(1)1673 518 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)179 610 y Fx(\()p Fj(ty)q(pe)271 616 y Fi(0)291 610 y Fj(;)g(disp)386 616 y Fi(0)413 610 y Fx(+)j Fj(extent)f Fn(\001)g Fx(\()p Fu(count)f Fn(\000)i Fx(1\)\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)997 616 y Fh(n)p Fk(\000)p Fi(1)1061 610 y Fj(;)g(disp)1156 616 y Fh(n)p Fk(\000)p Fi(1)1230 610 y Fx(+)i Fj(extent)h Fn(\001)e Fx(\()p Fu(count)h Fn(\000)g Fx(1\)\))p Fn(g)p Fj(:)158 701 y Fu(MPI)p 227 701 14 2 v 15 w(TYPE)p 330 701 V 15 w(VECTOR)g Fx(is)i(a)g(more)e(general)i(constructor)i(that)e(allo)o (ws)e(to)i(replicate)g(a)f(datat)o(yp)q(e)h(in)o(to)f(lo)q(cations)75 751 y(that)j(consists)h(of)e(equally)g(spaced)i(con)o(tiguous)f(blo)q(c)o(ks) g(of)f(equal)g(size)i({)f(blo)q(c)o(k)f(sizes)i(and)f(blo)q(c)o(k)f (displacemen)o(ts)75 801 y(are)i(all)f(m)o(ultiples)f(of)h(the)i(old)e(t)o (yp)q(e)h(exten)o(t.)75 898 y Fu(MPI)p 144 898 V 15 w(TYPE)p 247 898 V 15 w(VECTOR\()20 b(count,)h(oldtype,)f(stride,)g(blocklen,)g (newtype\))117 947 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j(\(in)o(teger\))117 993 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob)r(ject)117 1039 y(in)g Fu(stride)653 b Ft(n)o(um)o(b)q(er)10 b(of)f(elemen)o(ts)i(b)q (et)o(w)o(een)f(start)f(of)g(eac)o(h)h(blo)q(c)o(k)h(\(in)o(teger\))117 1084 y(in)j Fu(blocklen)609 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (eac)o(h)g(blo)q(c)o(k)g(\(p)q(ositiv)o(e)h(in)o(teger\))117 1130 y(out)f Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g (ob)r(ject)158 1206 y Fx(Example:)j(Assume,)c(again,)f(that)i Fu(oldtype)e Fx(p)q(oin)o(ts)i(to)g(the)g(t)o(yp)q(e)179 1289 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 1372 y Fx(with)j(exten)o(t)h(16.)16 b(A)10 b(call)f(to)h Fu(MPI)p 588 1372 V 15 w(TYPE)p 691 1372 V 15 w(VECTOR\()20 b(2,)i(oldtype,)e(4,)h(3,)h(newtype\))8 b Fx(will)h(create)i(the)g(datat)o(yp)q(e)179 1455 y Fn(f)p Fx(\()p Fu(double)n Fj(;)c Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(32\))p Fj(;)g Fx(\()p Fu(char)m Fj(;)g Fx(40\))p Fj(;)179 1547 y Fx(\()p Fu(double)o Fj(;)g Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(72\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(80\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(88\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(96\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(104\))p Fn(g)j Fx(:)75 1638 y(t)o(w)o(o)j(blo)q(c)o(ks)h (with)g(three)h(copies)g(eac)o(h)f(of)f(the)i(old)e(t)o(yp)q(e,)h(starting)g (4*16)f(apart.)158 1688 y(A)h(call)f(to)h Fu(MPI)p 399 1688 V 15 w(TYPE)p 502 1688 V 15 w(VECTOR\(oldtype,)k(3,)k(-2,)f(1,)g(newtype\))13 b Fx(will)f(create)j(the)g(datat)o(yp)q(e)179 1771 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fn(\000)p Fx(32\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fn(\000)p Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fn(\000)p Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fn(\000)p Fx(56\))p Fn(g)p Fj(:)158 1854 y Fx(In)14 b(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle) f(to)h(the)h(datat)o(yp)q(e)179 1937 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 1943 y Fi(0)311 1937 y Fj(;)7 b(disp)406 1943 y Fi(0)425 1937 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 1943 y Fh(n)p Fk(\000)p Fi(1)671 1937 y Fj(;)g(disp)766 1943 y Fh(n)p Fk(\000)p Fi(1)830 1937 y Fx(\))p Fn(g)p Fj(;)75 2020 y Fx(with)15 b(exten)o(t)h Fj(extent)p Fx(.)22 b(The)16 b(newly)f(created)h(datat)o(yp)q(e)g(is)f(a)g (sequence)i(of)d(length)i Fu(count)9 b Fn(\001)g Fu(blocklen)g Fn(\001)g Fj(n)15 b Fx(with)75 2070 y(en)o(tries:)179 2153 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 2159 y Fi(0)311 2153 y Fj(;)7 b(disp)406 2159 y Fi(0)425 2153 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 2159 y Fh(n)p Fk(\000)p Fi(1)671 2153 y Fj(;)g(disp)766 2159 y Fh(n)p Fk(\000)p Fi(1)830 2153 y Fx(\))p Fj(;)179 2244 y Fx(\()p Fj(ty)q(pe)271 2250 y Fi(0)291 2244 y Fj(;)g(disp)386 2250 y Fi(0)413 2244 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)754 2250 y Fh(n)p Fk(\000)p Fi(1)818 2244 y Fj(;)g(disp)913 2250 y Fh(n)p Fk(\000)p Fi(1)987 2244 y Fx(+)i Fj(extent)p Fx(\))p Fj(;)e(:::;)179 2336 y Fx(\()p Fj(ty)q(pe)271 2342 y Fi(0)291 2336 y Fj(;)g(disp)386 2342 y Fi(0)413 2336 y Fx(+)j(\()p Fu(blocklen)d Fn(\000)j Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1062 2342 y Fh(n)p Fk(\000)p Fi(1)1126 2336 y Fj(;)g(disp)1221 2342 y Fh(n)p Fk(\000)p Fi(1)1295 2336 y Fx(+)i(\()p Fu(blocklen)f Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)179 2427 y Fx(\()p Fj(ty)q(pe)271 2433 y Fi(0)291 2427 y Fj(;)e(disp)386 2433 y Fi(0)413 2427 y Fx(+)j Fu(stride)e Fn(\001)g Fj(extent)p Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)914 2433 y Fh(n)p Fk(\000)p Fi(1)979 2427 y Fj(;)g(disp)1074 2433 y Fh(n)p Fk(\000)p Fi(1)1148 2427 y Fx(+)i Fu(stride)f Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)179 2518 y Fx(\()p Fj(ty)q(pe)271 2524 y Fi(0)291 2518 y Fj(;)g(disp)386 2524 y Fi(0)413 2518 y Fx(+)j(\()p Fu(stride)e Fx(+)h Fu(blocklen)f Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 2609 y Fx(\()p Fj(ty)q(pe)271 2615 y Fh(n)p Fk(\000)p Fi(1)337 2609 y Fj(;)g(disp)432 2615 y Fh(n)p Fk(\000)p Fi(1)506 2609 y Fx(+)j(\()p Fu(stride)e Fx(+)h Fu(blocklen)f Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(::::;)179 2701 y Fx(\()p Fj(ty)q(pe)271 2707 y Fi(0)291 2701 y Fj(;)g(disp)386 2707 y Fi(0)413 2701 y Fx(+)j Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)p eop %%Page: 26 26 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(26)179 45 y(\()p Fj(ty)q(pe)271 51 y Fh(n)p Fk(\000)p Fi(1)337 45 y Fj(;)7 b(disp)432 51 y Fh(n)p Fk(\000)p Fi(1)506 45 y Fx(+)j Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 136 y Fx(\()p Fj(ty)q(pe)271 142 y Fi(0)291 136 y Fj(;)g(disp)386 142 y Fi(0)413 136 y Fx(+)j(\()p Fu(stride)e Fn(\001)h Fx(\()p Fu(count)f Fn(\000)h Fx(1\))g(+)h Fu(blocklen)e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 228 y Fx(\()p Fj(ty)q(pe)271 234 y Fh(n)p Fk(\000)p Fi(1)337 228 y Fj(;)g(disp)432 234 y Fh(n)p Fk(\000)p Fi(1)506 228 y Fx(+)j(\()p Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g(+)h Fu(blocklen)e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fn(g)158 398 y Fo(Discussion:)158 443 y Ft(I)17 b(c)o(hanged)h Fm(count)d Ft(from)i(b)q(eing)h(the)f(n)o(um)o(b)q(er)h(of)f(en)o(tries)h(\(or)f (replicates\),)i(as)f(in)f(the)h(old)g(draft,)f(to)g(b)q(eing)i(the)75 489 y(n)o(um)o(b)q(er)13 b(of)f(blo)q(c)o(ks.)18 b(The)12 b(reason)h(is)g (that)f(in)h(the)g(next)f(calls)i(I)e(allo)o(w)h(blo)q(c)o(k)h(lengths)g(to)e (v)n(ary;)g(it)h(is)g(easy)g(to)f(sa)o(y)g(what)g(it)75 535 y(means)i(to)f(ha)o(v)o(e)h Fg(n)g Ft(blo)q(c)o(ks)h(of)e(sizes)h Fg(b)631 539 y Ff(0)649 535 y Fg(;)6 b(:::;)f(b)731 539 y Fe(n)p Fd(\000)p Ff(1)792 535 y Ft(;)13 b(it)h(is)g(harder)g(to)g(explain)i(what)d (it)h(means)g(to)f(ha)o(v)o(e)h Fg(n)g Ft(elemen)o(ts)g(\014lling)75 580 y(up)f(successiv)o(e)i(p)q(ositions)h(in)d(blo)q(c)o(ks)i(of)e(sizes)h Fg(b)779 584 y Ff(0)796 580 y Fg(;)6 b(b)829 584 y Ff(1)847 580 y Fg(;)g(::::)p Ft(.)158 626 y(W)m(e)16 b(lo)q(ose)h(the)g(abilit)o(y)h (to)e(use)h Fm(MPI)p 699 626 12 2 v 13 w(TYPE)p 792 626 V 12 w(VECTOR)d Ft(to)i(create)g(a)g(partially)j(\014lled)f(last)f(blo)q(c)o(k)g (\(the)f(more)g(general)75 672 y Fm(MPI)p 137 672 V 13 w(TYPE)p 230 672 V 13 w(INDEXED)d Ft(function)18 b(is)f(needed\);)h(w)o(e)e(gain)h (consistency)i(with)d(the)h(next)g(calls,)h(and)f(con)o(v)o(enience)h(in)f (these)75 717 y(calls.)158 763 y(Oppinions?)158 895 y Fx(A)d(call)f(to)h Fu(MPI)p 399 895 14 2 v 15 w(TYPE)p 502 895 V 15 w(CONTIGUOUS\(oldty)o(pe,)19 b(count,)h(newtype\))12 b Fx(is)i(equiv)n(alen)o(t)f(to)h(a)g(call)f(to)75 945 y Fu(MPI)p 144 945 V 15 w(TYPE)p 247 945 V 15 w(VECTOR\(oldtype,)19 b(count,)h(1,)i(1,)f(newtype\))p Fx(.)158 995 y(The)16 b(function)f Fu(MPI)p 478 995 V 15 w(TYPE)p 581 995 V 15 w(HVECTOR)f Fx(is)h(iden)o(tical) g(to)h Fu(MPI)p 1095 995 V 15 w(TYPE)p 1198 995 V 15 w(VECTOR)p Fx(,)d(except)k(that)f Fu(stride)e Fx(is)h(giv)o(en)75 1045 y(in)i(b)o(ytes,)i(rather)f(then)g(in)f(elemen)o(ts.)29 b(The)18 b(use)h(for)e(b)q(oth)g(t)o(yp)q(es)i(of)e(v)o(ector)h(constructors)i(is)d (illustrated)g(in)75 1095 y(Section)d(1.13.4.)i(\()p Fu(H)e Fx(is)g(used)h(for)e(\\heterogeneous"\).)75 1192 y Fu(MPI)p 144 1192 V 15 w(TYPE)p 247 1192 V 15 w(HVECTOR\()20 b(count,)h(oldtype,)f (stride,)g(blocklen,)g(newtype\))117 1242 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j(\(in)o(teger\))117 1287 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob) r(ject)117 1333 y(in)g Fu(stride)653 b Ft(n)o(um)o(b)q(er)14 b(of)e(b)o(ytes)i(b)q(et)o(w)o(een)f(start)g(of)g(eac)o(h)g(blo)q(c)o(k)i (\(in)o(teger\))117 1378 y(in)f Fu(blocklen)609 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g(\(p)q(ositiv)o(e)h(in)o (teger\))117 1424 y(out)f Fu(newtype)607 b Ft(handle)14 b(to)f(output)h (datat)o(yp)q(e)g(ob)r(ject)158 1500 y Fu(MPI)p 227 1500 V 15 w(TYPE)p 330 1500 V 15 w(INDEXED)9 b Fx(is)i(a)f(more)f(general)i (function)g(that)f(allo)o(ws)g(to)g(replicate)h(an)g(old)e(t)o(yp)q(e)j(in)o (to)d(a)i(sequence)75 1550 y(of)e(con)o(tiguous)h(blo)q(c)o(ks,)g(where)h (blo)q(c)o(k)f(lengths)g(and)g(blo)q(c)o(k)f(displacemen)o(ts)h(can)g(all)f (di\013er)h({)g(but)g(are)g(all)f(m)o(ultiples)75 1600 y(of)k(the)i(old)e(t)o (yp)q(e)h(exten)o(t.)75 1697 y Fu(MPI)p 144 1697 V 15 w(TYPE)p 247 1697 V 15 w(INDEXED\()20 b(count,)h(oldtype,)f(array)p 917 1697 V 14 w(of)p 975 1697 V 16 w(indices,)g(array)p 1297 1697 V 14 w(of)p 1355 1697 V 16 w(blocklen,)g(newtype\))117 1747 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j (\(in)o(teger\))117 1793 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob)r(ject)117 1838 y(in)g Fu(array)p 275 1838 V 15 w(of)p 334 1838 V 15 w(indices)447 b Ft(displacemen)o(t)14 b(for)d(eac)o(h)h(blo)q(c)o(k,)g(in)g(m)o(ultiples)i(of)d Fm(oldtype)e Ft(ex-)905 1888 y(ten)o(t)k(\(arra)o(y)h(of)e(in)o(teger\))117 1934 y(in)i Fu(array)p 275 1934 V 15 w(of)p 334 1934 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(p)q(er)g(blo)q(c)o(k)g(\(arra)o (y)f(of)g(in)o(teger\))117 1979 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g(ob)r(ject)158 2056 y Fx(Example:)j(let)d Fu(oldtype)e Fx(b)q(e)i(a)g(handle)g(to)f(the)i(t)o(yp)q(e)179 2139 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 2222 y Fx(with)15 b(exten)o(t)h(16.)22 b(Let)16 b Fu(I)21 b(=)h(\(64,)f(0\))15 b Fx(and)g(let)g Fu(B)22 b(=)f(\(3,)g(1\))p Fx(.)h(A)15 b(call)g(to)g Fu(MPI)p 1346 2222 V 15 w(TYPE)p 1449 2222 V 15 w(INDEXED\(2,)20 b(oldtype,)75 2272 y(I,)h(B,)h(newtype\))12 b Fx(returns)j(in)f Fu(newtype)e Fx(a)i(handle)g(to)f(the)i(t)o(yp)q(e)179 2355 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(72\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(80\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(88\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(96\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(104\))179 2446 y(\()p Fu(double)o Fj(;)g Fx(0\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(8\))p Fn(g)j Fx(:)75 2537 y(three)15 b(copies)g(of)e(the)h(old)g(t)o(yp)q(e)g(starting)g(at)g (displacemen)o(t)f(64,)g(and)g(one)h(cop)o(y)g(starting)g(at)g(displacemen)o (t)f(0.)158 2587 y(In)h(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle)f(to)h(the)h(datat)o(yp)q(e)179 2670 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 2676 y Fi(0)311 2670 y Fj(;)7 b(disp)406 2676 y Fi(0)425 2670 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 2676 y Fh(n)p Fk(\000)p Fi(1)671 2670 y Fj(;)g(disp)766 2676 y Fh(n)p Fk(\000)p Fi(1)830 2670 y Fx(\))p Fn(g)p Fj(;)p eop %%Page: 27 27 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(27)75 45 y(with)17 b(exten)o(t)h Fp(extent)p Fx(.)29 b(Let)17 b Fu(B)g Fx(b)q(e)h(the)g Fu(array)p 816 45 14 2 v 15 w(of)p 875 45 V 15 w(blocklen)e Fx(parameter)h(and)g Fu(I)g Fx(b)q(e)h(the)g Fu(array)p 1652 45 V 14 w(of)p 1710 45 V 15 w(indices)75 102 y Fx(parameter.)f(The)12 b(newly)f(created)i(t)o(yp) q(e)f(returned)h(b)o(y)e Fu(newtype)f Fx(is)h(the)h(sequence)h(of)e Fj(n)t Fn(\001)1474 71 y Fc(P)1517 81 y Fu(count)o Fk(\000)p Fi(1)1517 115 y Fh(i)p Fi(=0)1678 102 y Fu(B)p Fx([)p Fu(i)p Fx(])f(en)o(tries)179 185 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 191 y Fi(0)311 185 y Fj(;)d(disp)406 191 y Fi(0)434 185 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])f Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)871 191 y Fh(n)p Fk(\000)p Fi(1)935 185 y Fj(;)g(disp)1030 191 y Fh(n)p Fk(\000)p Fi(1)1104 185 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])e Fn(\001)g Fj(extent)p Fx(\))p Fj(;)f(:::;)179 277 y Fx(\()p Fj(ty)q(pe)271 283 y Fi(0)291 277 y Fj(;)g(disp)386 283 y Fi(0)413 277 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(0)p Fx(])e(+)h Fu(B)p Fx([)p Fu(0)p Fx(])f Fn(\000)i Fx(1\))f Fn(\001)f Fj(extent)p Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)1071 283 y Fh(n)p Fk(\000)p Fi(1)1136 277 y Fj(;)g(disp)1231 283 y Fh(n)p Fk(\000)p Fi(1)1305 277 y Fx(+)i(\()p Fu(I)p Fx([)p Fu(0)p Fx(])f(+)i Fu(B)p Fx([)p Fu(0)p Fx(])e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 368 y Fx(\()p Fj(ty)q(pe)271 374 y Fi(0)291 368 y Fj(;)g(disp)386 374 y Fi(0)413 368 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1010 374 y Fh(n)p Fk(\000)p Fi(1)1074 368 y Fj(;)g(disp)1169 374 y Fh(n)p Fk(\000)p Fi(1)1243 368 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)179 459 y Fx(\()p Fj(ty)q(pe)271 465 y Fi(0)291 459 y Fj(;)g(disp)386 465 y Fi(0)413 459 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 551 y Fx(\()p Fj(ty)q(pe)271 557 y Fh(n)p Fk(\000)p Fi(1)337 551 y Fj(;)g(disp)432 557 y Fh(n)p Fk(\000)p Fi(1)506 551 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fn(g)p Fj(:)158 642 y Fx(A)15 b(call)e(to)i Fu(MPI)p 401 642 V 15 w(TYPE)p 504 642 V 15 w(VECTOR\(count,)k(oldtype,)h(stride,)g(blocklen,)g(newtype\))13 b Fx(is)h(equiv)n(alen)o(t)g(to)75 692 y(a)g(call)f(to)75 742 y Fu(MPI)p 144 742 V 15 w(TYPE)p 247 742 V 15 w(INDEX\(count,)19 b(oldtype,)h(I,)i(B,)f(newtype\))12 b Fx(where)179 825 y Fu(I)p Fx([)p Fu(j)p Fx(])e(=)i Fj(j)f Fn(\001)e Fu(stride)k Fj(;)7 b(j)13 b Fx(=)f(0)p Fj(;)7 b(:::;)g Fu(count)f Fn(\000)j Fx(1)p Fj(;)75 908 y Fx(and)179 991 y Fu(B)p Fx([)p Fu(j)p Fx(])h(=)i Fu(blocklen)g Fj(;)7 b(j)13 b Fx(=)f(0)p Fj(;)7 b(:::;)g Fu(count)f Fn(\000)j Fx(1)p Fj(:)158 1074 y Fx(The)k(function)g Fu(MPI)p 473 1074 V 15 w(TYPE)p 576 1074 V 15 w(HINDEXED)e Fx(is)h(iden)o(tical)g(to)h Fu(MPI)p 1100 1074 V 15 w(TYPE)p 1203 1074 V 15 w(INDEXED)p Fx(,)e(except)j(that)f(blo)q(c)o(k)f(displace-)75 1123 y(men)o(ts)h(in)h Fu(array)p 358 1123 V 14 w(of)p 416 1123 V 16 w(indices)e Fx(are)i(sp)q (eci\014ed)i(in)d(b)o(ytes,)h(rather)h(than)f(in)f(m)o(ultiples)f(of)i(the)g (old)f(t)o(yp)q(e)i(exten)o(t.)75 1221 y Fu(MPI)p 144 1221 V 15 w(TYPE)p 247 1221 V 15 w(HINDEXED\()20 b(count,)g(oldtype,)h(array)p 939 1221 V 14 w(of)p 997 1221 V 15 w(indices,)g(array)p 1319 1221 V 14 w(of)p 1377 1221 V 16 w(blocklen,)e(newtype\))117 1270 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j (\(in)o(teger\))117 1316 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob)r(ject)117 1362 y(in)g Fu(array)p 275 1362 V 15 w(of)p 334 1362 V 15 w(indices)447 b Ft(b)o(yte)13 b(displacemen)o(t)j(of)c(eac)o(h)i(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o (teger\))117 1407 y(in)h Fu(array)p 275 1407 V 15 w(of)p 334 1407 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o(teger\))117 1453 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g (ob)r(ject)158 1529 y Fu(MPI)p 227 1529 V 15 w(TYPE)p 330 1529 V 15 w(STRUCT)j Fx(is)h(the)g(most)f(general)i(constructor:)28 b(It)18 b(further)h(generalizes)g(the)g(previous)f(one)g(in)75 1579 y(that)c(it)f(allo)o(ws)g(eac)o(h)h(blo)q(c)o(k)g(to)g(consist)g(of)g (replications)f(of)h(a)f(di\013eren)o(t)i(datat)o(yp)q(e.)75 1676 y Fu(MPI)p 144 1676 V 15 w(TYPE)p 247 1676 V 15 w(STRUCT\(count,)k (array)p 677 1676 V 15 w(of)p 736 1676 V 15 w(types,)i(array)p 1014 1676 V 14 w(of)p 1072 1676 V 15 w(indices,)g(array)p 1394 1676 V 14 w(of)p 1452 1676 V 16 w(blocklen,)e(newtype\))117 1726 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j (\(in)o(teger\))117 1772 y(in)f Fu(array)p 275 1772 V 15 w(of)p 334 1772 V 15 w(types)491 b Ft(t)o(yp)q(e)17 b(of)g(elemen)o(ts)h(in)f(eac)o (h)h(blo)q(c)o(k)g(\(arra)o(y)f(of)g(datat)o(yp)q(e)h(ob-)905 1821 y(jects\))117 1867 y(in)c Fu(array)p 275 1867 V 15 w(of)p 334 1867 V 15 w(indices)447 b Ft(b)o(yte)13 b(displacemen)o(t)j(of)c(eac)o(h) i(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o(teger\))117 1913 y(in)h Fu(array)p 275 1913 V 15 w(of)p 334 1913 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g (\(arra)o(y)f(of)g(in)o(teger\))117 1958 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g(ob)r(ject)158 2035 y Fx(Example:)j(Let)d Fu(type1)f Fx(b)q(e)h(a)g(handle)g(to)f(the)i(t)o (yp)q(e)179 2118 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 2201 y Fx(with)14 b(exten)o(t)i(16.)j(Let)c Fu(I)22 b(=)f(\(0,)g(16,)h(26\))p Fx(,)13 b Fu(B)22 b(=)f(\(2,)g(1,)h(3\))14 b Fx(and)g Fu(T)22 b(=)f(\(MPI)p 1356 2201 V 15 w(FLOAT,)g(type1,)f(MPI)p 1742 2201 V 16 w(BYTE\))p Fx(.)75 2251 y(Then)11 b(a)f(call)f(to)h Fu(MPI)p 399 2251 V 15 w(TYPE)p 502 2251 V 15 w(STRUC\(3,)20 b(T,)i(I,)f(B,)h(newtype\))8 b Fx(returns)k(in)e Fu(newtype)f Fx(a)h(handle)g(to)g(the)h(datat)o(yp)q(e)179 2334 y Fn(f)p Fx(\()p Fu(float)o Fj(;)c Fx(0\))p Fj(;)g Fx(\()p Fu(float)n Fj(;)g Fx(4\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(26\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(27\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(28\))p Fn(g)j Fx(:)75 2417 y(t)o(w)o(o)17 b(copies)h(of)e Fu(MPI)p 402 2417 V 15 w(FLOAT)g Fx(starting)i(at)f(0,)g(follo)o(w)o(ed)f(b)o(y)h(one)g(cop)o(y)g(of)g Fu(type1)f Fx(starting)h(at)g(16,)h(follo)o(w)o(ed)d(b)o(y)75 2466 y(three)g(copies)g(of)e Fu(MPI)p 420 2466 V 15 w(CHAR)p Fx(,)g(starting)g(at)h(26.)k(\(W)m(e)13 b(assume)h(that)g(a)f Fu(float)g Fx(o)q(ccupies)i(four)f(b)o(ytes.\))158 2516 y(In)g(general,)g (let)g Fu(T)f Fx(b)q(e)i(the)f Fu(array)p 701 2516 V 15 w(of)p 760 2516 V 15 w(types)f Fx(parameter,)g(where)i Fu(T[i])e Fx(is)h(a)f(handle) h(to)179 2599 y Fj(ty)q(pe)255 2605 y Fh(i)281 2599 y Fx(=)e Fn(f)p Fx(\()p Fj(ty)q(pe)438 2582 y Fh(i)438 2610 y Fi(0)458 2599 y Fj(;)7 b(disp)553 2582 y Fh(i)553 2610 y Fi(0)571 2599 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)753 2582 y Fh(i)753 2610 y(n)774 2614 y Fb(i)787 2599 y Fj(;)g(disp)882 2582 y Fh(i)882 2610 y(n)903 2614 y Fb(i)918 2599 y Fx(\))p Fn(g)p Fj(;)p eop %%Page: 28 28 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(28)75 45 y(with)15 b(exten)o(t)i Fj(extent)417 51 y Fh(i)431 45 y Fx(.)24 b(Let)16 b Fu(B)g Fx(b)q(e)g(the)g Fu(array)p 825 45 14 2 v 15 w(of)p 884 45 V 15 w(blocklen)e Fx(parameter)i(and)f Fu(I)h Fx(b)q(e)g(the)h Fu(array)p 1652 45 V 14 w(of)p 1710 45 V 15 w(indices)75 102 y Fx(parameter.)g(Then)c(the)h (newly)e(created)i(datat)o(yp)q(e)f(is)g(the)g(sequence)i(of)d(length)1355 71 y Fc(P)1398 81 y Fu(count)o Fk(\000)p Fi(1)1398 115 y Fh(i)p Fi(=0)1559 102 y Fu(B)p Fx([)p Fu(i)p Fx(])5 b Fn(\001)i Fj(extent)1768 108 y Fh(i)1794 102 y Fx(with)75 152 y(en)o(tries)179 235 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 218 y Fi(0)292 245 y(0)311 235 y Fj(;)g(disp)406 218 y Fi(0)406 245 y(0)434 235 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])o(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)724 218 y Fi(0)724 245 y Fh(n)745 249 y Fr(0)762 235 y Fj(;)g(disp)857 218 y Fi(0)857 245 y Fh(n)878 249 y Fr(0)904 235 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])o(\))p Fj(;)d(:::;)179 331 y Fx(\()p Fj(ty)q(pe)271 314 y Fi(0)271 341 y(0)291 331 y Fj(;)g(disp)386 314 y Fi(0)386 341 y(0)413 331 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])e(+)h(\()p Fu(B)p Fx([)p Fu(0)p Fx(])f Fn(\000)i Fx(1\))f Fn(\001)f Fj(extent)889 337 y Fi(0)908 331 y Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)1090 314 y Fi(0)1090 341 y Fh(n)1111 345 y Fr(0)1128 331 y Fj(;)g(disp)1223 314 y Fi(0)1223 341 y Fh(n)1244 345 y Fr(0)1271 331 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])f(+)i(\()p Fu(B)p Fx([)p Fu(0)p Fx(])e Fn(\000)h Fu(1)p Fx(\))g Fn(\001)g Fj(extent)1748 337 y Fi(0)1767 331 y Fx(\))p Fj(;)e(:::;)179 432 y Fx(\()p Fj(ty)q(pe)271 414 y Fu(count)p Fk(\000)p Fi(1)271 443 y(0)425 432 y Fj(;)g(disp)520 414 y Fu(count)o Fk(\000)p Fi(1)520 443 y(0)683 432 y Fx(+)i Fu(I)p Fx([)p Fu(count)f Fn(\000)h Fu(1)p Fx(]\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1133 415 y Fu(count)n Fk(\000)p Fi(1)1133 442 y Fh(n)1154 455 y Fu(count)m Fa(\000)p Fr(1)1302 432 y Fj(;)g(disp)1397 415 y Fu(count)o Fk(\000)p Fi(1)1397 442 y Fh(n)1418 455 y Fu(count)n Fa(\000)p Fr(1)1576 432 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])o(\))p Fj(;)d(:::;)179 545 y Fx(\()p Fj(ty)q(pe)271 527 y Fu(count)p Fk(\000)p Fi(1)271 556 y(0)425 545 y Fj(;)g(disp)520 527 y Fu(count)o Fk(\000)p Fi(1)520 556 y(0)683 545 y Fx(+)i Fu(I)p Fx([)p Fu(count)f Fn(\000)h Fu(1)p Fx(])g(+)h(\()p Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)1479 554 y Fu(count)o Fk(\000)p Fi(1)1632 545 y Fx(\))p Fj(;)e(:::;)179 643 y Fx(\()p Fj(ty)q(pe)271 626 y Fu(count)p Fk(\000)p Fi(1)271 653 y Fh(n)292 666 y Fu(count)o Fa(\000)p Fr(1)442 643 y Fj(;)g(disp)537 626 y Fu(count)o Fk(\000)p Fi(1)537 653 y Fh(n)558 666 y Fu(count)n Fa(\000)p Fr(1)716 643 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i(\()p Fu(B)p Fx([)p Fu(count)e Fn(\000)h Fu(1)p Fx(])g Fn(\000)g Fu(1)p Fx(\))g Fn(\001)g Fj(extent)1513 652 y Fu(count)o Fk(\000)p Fi(1)1667 643 y Fx(\))p Fn(g)158 744 y Fx(A)14 b(call)f(to)h Fu(MPI)p 399 744 V 15 w(TYPE)p 502 744 V 15 w(HINDEXED\()20 b(count,)g(oldtype,)g(I,)i(B,)f(newtype\))12 b Fx(is)i(equiv)n(alen)o(t)f(to)h(a)g(call)f(to)75 794 y Fu(MPI)p 144 794 V 15 w(TYPE)p 247 794 V 15 w(STRUC\()20 b(count,)h(T,)g(I,)h(B,)f (newtype\))p Fx(,)12 b(where)j(eac)o(h)f(en)o(try)h(of)e Fu(T)h Fx(is)f(equal)h(to)g Fu(oldtype)p Fx(.)158 922 y Fo(Discussion:)158 968 y Ft(I)f(am)g(op)q(en)g(to)g(suggestions)j(for)c(b)q(etter)i(names)f(to)g (these)g(functions.)75 1167 y Fs(1.13.2)49 b(Additional)19 b(functions)75 1243 y Fx(The)c(displacemen)o(ts)g(in)g(a)f(general)h(datat)o (yp)q(e)h(are)f(relativ)o(e)g(to)g(some)f(start)h(address.)23 b Fv(Absolute)15 b(addresses)75 1293 y Fx(can)10 b(b)q(e)g(substituted)h(for) e(these)i(displacemen)o(ts:)16 b(w)o(e)9 b(treat)i(them)d(as)i(displacemen)o (ts)f(relativ)o(e)h(to)f(\\address)i(zero",)75 1343 y(the)17 b(start)g(of)f(the)h(address)h(space.)26 b(This)16 b(initial)f(address)j (zero)f(is)f(indicated)h(b)o(y)f(the)h(constan)o(t)g Fu(MPI)p 1721 1343 V 15 w(BOTTOM)p Fx(.)75 1393 y(Th)o(us,)h(a)f(datat)o(yp)q(e)g(can) h(sp)q(ecify)f(the)h(absolute)f(address)i(of)e(the)g(en)o(tries)i(in)d(the)i (comm)o(unication)c(bu\013er,)k(in)75 1443 y(whic)o(h)c(case)h(the)f Fu(start)f Fx(parameter)g(is)h(passed)h(the)g(v)n(alue)e Fu(MPI)p 1096 1443 V 15 w(BOTTOM)p Fx(.)158 1492 y(The)h(address)i(of)d(a)g(lo)q (cation)g(in)h(memory)d(can)j(b)q(e)h(found)e(b)o(y)h(in)o(v)o(oking)e(the)i (function)g Fu(MPI)p 1616 1492 V 15 w(ADDRESS)p Fx(.)75 1590 y Fu(MPI)p 144 1590 V 15 w(ADDRESS\(location,)k(address\))117 1639 y Ft(in)c Fu(location)609 b Ft(lo)q(cation)15 b(in)f(caller)g(memory)g (\(c)o(hoice\))117 1685 y(out)g Fu(address)607 b Ft(address)14 b(of)f(lo)q(cation)i(\(in)o(teger\))158 1761 y Fx(Returns)g(the)f(\(b)o (yte\))h(address)g(of)e Fu(location)p Fx(.)158 1811 y(Another)f(useful)g (auxiliary)d(function)j(is)f Fu(MPI)p 874 1811 V 15 w(EXTENT)p Fx(,)f(that)h(returns)i(the)f(exetend)h(of)e(a)g(datat)o(yp)q(e)h({)f(where) 75 1861 y(exten)o(t)k(is)f(as)g(de\014ned)h(in)e(Eq.)g(1.1.)75 1958 y Fu(MPI)p 144 1958 V 15 w(TYPE)p 247 1958 V 15 w(EXTENT\(datatype,)18 b(extent\))117 2008 y Ft(in)c Fu(datatype)609 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r(ject)117 2053 y(out)g Fu(extent)629 b Ft(datat)o(yp)q(e)14 b(exten)o(t)f(\(in)o(teger\))158 2130 y Fx(As)i(de\014ned)h(no)o(w,)f(the)g(exten)o(t)h(of)e(a)h(datat)o(yp)q(e)g (go)q(es)h(from)d(its)i(\014rst)g(o)q(ccupied)h(b)o(yte)g(to)e(its)h(last)g (o)q(ccupied)75 2180 y(b)o(yte.)j(It)c(is)f(often)g(con)o(v)o(enien)o(t)h(to) f(de\014ne)h(a)f(datat)o(yp)q(e)g(that)h(has)f(a)g(\\hole")f(at)i(its)f(b)q (egining)f(and/or)h(its)g(end;)h(or)75 2229 y(a)f(datat)o(yp)q(e)g(with)g(en) o(tries)h(that)f(extend)h(b)q(ey)o(ond)f(the)h(datat)o(yp)q(e)f(\\exten)o (t".)18 b(This)13 b(allo)o(ws)f(to)g(use)i(the)g(functions)75 2279 y Fu(MPI)p 144 2279 V 15 w(TYPE)p 247 2279 V 15 w(CONTIGUOUS)p Fx(,)g Fu(MPI)p 574 2279 V 16 w(TYPE)p 678 2279 V 14 w(VECTOR)p Fx(,)i(or)h Fu(MPI)p 972 2279 V 15 w(TYPE)p 1075 2279 V 15 w(INDEX)f Fx(to)h(replicate)g(a)g(more)f(general)i(pattern;)75 2329 y(examples)h(of)f(suc)o(h)j(usage)f(are)f(pro)o(vided)h(in)f(Section)h (1.13.4.)33 b(T)m(o)18 b(ac)o(hiev)o(e)i(this,)h(w)o(e)e(add)h(t)o(w)o(o)f (additional)75 2379 y(\\pseudo-datat)o(yp)q(es")f Fu(MPI)p 518 2379 V 15 w(LB)p Fx(,)e(and)h Fu(MPI)p 755 2379 V 15 w(UB)g Fx(that)g(can)h(b)q(e)g(used,)g(resp)q(ectiv)o(ely)m(,)h(to)e(mark)e(the)j (lo)o(w)o(er)f(b)q(ound)75 2429 y(or)j(the)h(upp)q(er)h(b)q(ound)e(of)g(a)g (datat)o(yp)q(e.)38 b(these)22 b(pseudo-datat)o(yp)q(es)g(o)q(ccup)o(y)f(no)f (space)i(\()p Fj(extent)p Fx(\()p Fu(MP)p 1748 2429 V 16 w(LB)o Fx(\))h(=)75 2478 y Fj(extent)p Fx(\()p Fu(MP)p 255 2478 V 16 w(UB)o Fx(\))d(=)g(0\).)32 b(They)19 b(do)g(not)g(a\013ect)g(the)h (signature)f(of)f(a)g(message)h(created)h(with)e(this)h(datat)o(yp)q(e.)75 2528 y(Ho)o(w)o(ev)o(er,)12 b(they)h(do)f(a\013ect)g(the)h(de\014nition)f(of) f(the)i(exten)o(t)g(of)e(a)h(datat)o(yp)q(e)g(and,)g(therefore,)h(a\013ects)g (the)g(outcome)75 2578 y(of)g(a)h(replication)f(of)h(this)f(datat)o(yp)q(e)i (b)o(y)e(a)h(datat)o(yp)q(e)g(constructor.)158 2628 y(Example:)i(Let)e Fu(I)21 b(=)h(\(-3,)f(0,)g(6\))p Fx(;)12 b Fu(T)22 b(=)g(\(MPI)p 928 2628 V 14 w(LB,)g(MPI)p 1096 2628 V 15 w(INT,)f(MPI)p 1286 2628 V 15 w(UB\))p Fx(,)12 b(and)g Fu(B)22 b(=)g(\(1,)f(1,)g(1\))p Fx(.)c(Then)75 2678 y(a)22 b(call)g(to)g Fu(MPI)p 331 2678 V 15 w(TYPE)p 434 2678 V 15 w(STRUC\(3,)e(T,)i(I,)f(B,)g(type1\))h Fx(creates)i(a)e(new)h(datat)o(yp)q(e)g(that)f(has)h(an)f(exten)o(t)i(of)p eop %%Page: 29 29 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(29)75 45 y(9)20 b(\(from)g(-3)g(to)h(6\),)g(and)g(con)o(tains) g(an)f(in)o(teger)i(at)e(displacemen)o(t)g(0.)39 b(This)20 b(is)h(the)g(datat)o(yp)q(e)g(de\014ned)h(b)o(y)75 95 y(the)h(sequence)h Fn(f)p Fu(\(lb,)c(-3\),)h(\(int,)g(0\),)g(\(ub,)g(6\))p Fn(g)g Fx(.)43 b(If)21 b(this)h(t)o(yp)q(e)h(is)f(replicated)g(t)o(wice)h(b)o(y)e(a) h(call)f(to)75 145 y Fu(MPI)p 144 145 14 2 v 15 w(TYPE)p 247 145 V 15 w(CONTIGUOUS\(2,)e(type1,)i(type2\))d Fx(then)i(the)h(newly)e (created)j(t)o(yp)q(e)e(can)g(b)q(e)g(describ)q(ed)i(b)o(y)d(the)75 195 y(sequence)d Fn(f)p Fu(\(lb,)21 b(-3\),)g(\(int,)g(0\),)g(\(int,9\),)f (\(ub,)h(15\))p Fn(g)g Fx(.)d(\(En)o(tries)d(of)e(t)o(yp)q(e)i Fu(lb)f Fx(or)f Fu(ub)h Fx(can)g(b)q(e)h(deleted)75 244 y(if)e(they)h(are)h (not)f(at)f(the)i(endp)q(oin)o(ts)f(of)f(the)i(datat)o(yp)q(e.\))158 294 y(In)f(general,)g(if)179 377 y Fj(D)q(ataty)q(pe)f Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 383 y Fi(0)538 377 y Fj(;)7 b(disp)633 383 y Fi(0)651 377 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 383 y Fh(n)p Fk(\000)p Fi(1)897 377 y Fj(;)g(disp)992 383 y Fh(n)p Fk(\000)p Fi(1)1057 377 y Fx(\))p Fn(g)p Fj(;)75 460 y Fx(then)15 b(the)f Fv(lo)o(w)o(er)g(b)q(ound)f Fx(of)g Fj(D)q(ataty)q(pe)i Fx(is)f(de\014ned)h(to)f(b)q(e)179 566 y Fj(l)q(b)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f(=)469 507 y Fc(\032)521 541 y Fx(min)590 547 y Fh(j)614 541 y Fj(disp)690 547 y Fh(j)1036 541 y Fx(if)g(no)h(en)o(try) g(has)h(basic)f(t)o(yp)q(e)g Fj(ty)q(pe)1588 547 y Fh(j)1618 541 y Fx(=)e Fu(lb)521 590 y Fx(min)n Fn(f)p Fj(disp)687 596 y Fh(j)730 590 y Fx(:)25 b Fj(ty)q(pe)843 596 y Fh(j)873 590 y Fx(=)12 b Fu(lb)o Fn(g)41 b Fx(otherwise)158 671 y(Similarly)l(,)11 b(the)j Fv(upp)q(er)g(b)q(ound)f Fx(of)g Fj(D)q(ataty)q(pe)i Fx(is)f(de\014ned)h(to)f(b)q(e)179 777 y Fj(ub)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f(=)479 719 y Fc(\032)531 752 y Fx(max)609 758 y Fh(j)633 752 y Fj(disp)709 758 y Fh(j)736 752 y Fx(+)c Fj(siz)r(eof)t Fx(\()p Fj(ty)r(pe)987 758 y Fh(j)1007 752 y Fx(\))56 b(if)13 b(no)g(en)o(try)i(has)f(basic)g(t)o(yp)q(e)g Fj(ty)q(pe)1630 758 y Fh(j)1660 752 y Fx(=)e Fu(ub)531 802 y Fx(max)o Fn(f)p Fj(disp)706 808 y Fh(j)748 802 y Fx(:)25 b Fj(ty)q(pe)861 808 y Fh(j)891 802 y Fx(=)12 b Fu(ub)p Fn(g)65 b Fx(otherwise)158 883 y(Then)179 966 y Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))13 b(=)f Fj(ub)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f Fn(\000)e Fj(l)q(b)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))p Fj(:)158 1049 y Fx(The)j(formal)c(de\014nitions)j(giv)o(en)g(for)g(the)h(v)n(arious)e (datat)o(yp)q(e)i(constructors)h(apply)d(no)o(w,)h(with)g(the)h(amended)75 1099 y(de\014nition)h(of)h Fj(extent)p Fx(.)158 1148 y(A)i(datat)o(yp)q(e)g (ob)r(ject)h(has)f(to)g(b)q(e)h Fv(committed)c Fx(b)q(efore)k(it)f(can)g(b)q (e)h(used)g(in)e(a)h(comm)o(unication.)21 b(A)16 b(com-)75 1198 y(mitted)h(datat)o(yp)q(e)i(ob)r(ject)h(cannot)e(b)q(e)i(used)f(to)f (build)g(new)h(deriv)o(ed)g(datat)o(yp)q(es.)33 b(The)19 b(system)f(ma)o(y)f (use)i(a)75 1248 y(di\013eren)o(t)e(in)o(ternal)g(represen)o(tation)h(for)e (committed)e(datat)o(yp)q(e)j(ob)r(ject)g(so)g(as)f(to)g(facilitate)g(comm)o (unicatio)o(n,)75 1298 y(e.g.)h(c)o(hange)c(from)e(a)i(compacted)f(represen)o (tation)j(to)d(a)h(\015at)f(represen)o(tation)i(of)f(the)g(datat)o(yp)q(e,)g (and)f(select)i(the)75 1348 y(most)f(con)o(v)o(enien)o(t)h(transfer)h(mec)o (hanism.)75 1445 y Fu(MPI)p 144 1445 V 15 w(TYPE)p 247 1445 V 15 w(COMMIT\(intype,)k(outtype\))117 1494 y Ft(in)14 b Fu(intype)653 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r(ject)g(to)f(b)q(e)g(commited)117 1540 y(out)h Fu(outtype)607 b Ft(handle)14 b(to)f(committed)h(datat)o(yp)q(e) g(ob)r(ject)158 1616 y Fx(There)h(is)f(no)g(need)h(to)f(commit)e(basic)i (datat)o(yp)q(es;)g(they)h(are)g(\\precommited".)h(They)f(are)f(the)h (exception)75 1666 y(to)f(the)g(rule)g(that)g(committed)e(datat)o(yp)q(es)i (cannot)g(b)q(e)h(used)g(to)e(form)g(deriv)o(ed)h(datat)o(yp)q(es.)158 1795 y Fo(Discussion:)158 1840 y Ft(1.)j(Do)c(w)o(e)g(really)i(need)e(a)g (commit)h(here?)158 1886 y(2.)j(Do)c(w)o(e)g(w)o(an)o(t)g(one)g(inout)h (argumen)o(t,)g(or)f(t)o(w)o(o)f(distinct)j(argumen)o(ts?)158 1932 y(3.)h(Are)11 b(committed)h(and)g(uncommited)h(ob)r(jects)e(of)g(the)h (same)f(t)o(yp)q(e?)17 b(\(The)11 b(answ)o(er)g(m)o(ust)h(b)q(e)f(y)o(es,)h (if)f(it)g(is)h(an)g(inout)75 1977 y(argumen)o(t.\))158 2027 y(4.)18 b(If)13 b(w)o(e)f(ha)o(v)o(e)i(only)h(one)e(inout)i(argumen)o(t,)f (do)q(es)g(a)f(commit)h(c)o(hange)g(the)g(datat)o(yp)q(e)g(ob)r(ject,)f(or)h (do)q(es)g(it)f(create)h(a)75 2077 y(new)d(ob)r(ject)g(and)h(return)g(a)f(p)q (oin)o(ter)h(to)f(it)h(in)g Fm(datatype)p Ft(?)h(I.e.,)d(if)i(w)o(e)e(ha)o(v) o(e)i(another)g(p)q(oin)o(ter)h(to)e(the)g(same)g(uncommitted)75 2127 y(ob)r(ject,)i(can)g(w)o(e)g(con)o(tin)o(ue)h(to)f(use)h(it?)75 2306 y Fu(MPI)p 144 2306 V 15 w(TYPE)p 247 2306 V 15 w(FREE\(datatype\))117 2356 y Ft(inout)h Fu(datatype)553 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r (ject)158 2433 y Fx(Marks)g(the)g(datat)o(yp)q(e)f(ob)r(ject)h(asso)q(ciated) g(with)f Fu(datatype)f Fx(for)h(deallo)q(cation.)k(The)c(ob)r(ject)i(will)c (b)q(e)j(deal-)75 2482 y(lo)q(cated)f(after)h(an)o(y)f(p)q(ending)g(comm)o (unication)d(that)j(uses)i(this)e(ob)r(ject)h(completes,)f(at)g(whic)o(h)g(p) q(oin)o(t)g Fu(datatype)75 2532 y Fx(b)q(ecomes)h(n)o(ull.)158 2661 y Fo(Discussion:)p eop %%Page: 30 30 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(30)158 45 y Ft(If)13 b(w)o(e)g(ha)o(v)o(e)h(di\013eren)o(t)h (datat)o(yp)q(es)g(for)e(commited)i(and)f(uncommited,)i(then)e(w)o(e)f(ha)o (v)o(e)h(2)f(free)h(functions)h(\(or)e(nev)o(er)75 91 y(free)g(uncommited)h (datat)o(yp)q(e\))75 290 y Fs(1.13.3)49 b(Use)16 b(of)h(general)f(datat)o(yp) q(es)g(in)h(comm)n(unication)75 366 y Fx(Handles)c(to)g(deriv)o(ed)g(datat)o (yp)q(es)g(can)g(b)q(e)g(passed)h(to)f(a)f(comm)o(unication)d(call)j(wherev)o (er)j(a)d(datat)o(yp)q(e)h(argumen)o(t)75 416 y(is)i(required.)23 b(A)16 b(call)e(of)h(the)h(form)d Fu(MPI)p 715 416 14 2 v 15 w(SEND\(start,)20 b(count,)g(datatype,)g(...\))p Fx(,)14 b(where)j Fu(count)c Fx(=)h Fu(1)h Fx(uses)75 466 y(the)e(send)g(bu\013er)h(de\014ned)g (b)o(y)e Fu(start)f Fx(and)i Fu(datatype)p Fx(;)d(i.e.)17 b(if)12 b Fu(datatype)f Fx(is)h(a)g(handle)h(to)f(the)h(datat)o(yp)q(e)g(de\014ned)75 516 y(b)o(y)h(the)i(sequence)h Fn(f)p Fx(\()p Fj(ty)q(pe)492 522 y Fi(0)511 516 y Fj(;)7 b(disp)606 522 y Fi(0)625 516 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)807 522 y Fh(n)p Fk(\000)p Fi(1)870 516 y Fj(;)g(disp)965 522 y Fh(n)p Fk(\000)p Fi(1)1030 516 y Fx(\))p Fn(g)p Fx(,)15 b(\(with)f(empt)o(y)g(en)o(tries)h(deleted\))h (then)g(the)f(send)75 566 y(bu\013er)f(consists)f(of)f(the)i(en)o(tries)f(of) f(t)o(yp)q(es)i Fj(ty)q(pe)823 572 y Fh(i)838 566 y Fx(,)e(at)g(addresses)j Fu(start)6 b Fx(+)h Fj(disp)1324 572 y Fh(i)1338 566 y Fx(,)12 b Fj(i)g Fx(=)g(0)p Fj(;)7 b(:::;)g(n)g Fn(\000)g Fx(1.)14 b(The)f(message)75 615 y(generated)h(consists)f(of)f(a)g(sequence)i(of)e Fj(n)g Fx(en)o(tries,)h(of)f(t)o(yp)q(es)h Fj(ty)q(pe)1119 621 y Fi(0)1139 615 y Fj(;)7 b(:::;)g(ty)q(pe)1289 621 y Fh(n)p Fk(\000)p Fi(1)1353 615 y Fx(.)17 b(The)c(receiv)o(e)g(bu\013er)h(for)e(a)g (call)75 665 y Fu(MPI)p 144 665 V 15 w(RECV\()21 b(start,)f(count,)h (datatype)f(,)i(...\))p Fx(,)10 b(is)h(similarly)e(de\014ned,)j(for)f Fu(count)21 b(=1)p Fx(.)c(The)12 b(same)e(applies)75 715 y(to)k(all)e(other)j (send)g(and)e(receiv)o(e)j(op)q(erations)e(that)g(ha)o(v)o(e)f(a)h Fu(datatype)e Fx(parameter.)158 765 y(A)h(datat)o(yp)q(e)g(used)h(in)e(a)h (receiv)o(e)h(op)q(eration)f(ma)o(y)e(not)h(ha)o(v)o(e)h(o)o(v)o(erlaping)f (en)o(tries;)h(otherwise)h(the)g(outcome)75 815 y(of)f(the)i(receiv)o(e)g(op) q(eration)f(is)f(unde\014ned.)158 864 y(If)i(a)g(send)i(op)q(eration)e(using) g(a)g(general)h(datat)o(yp)q(e)f Fu(type1)g Fx(is)g(matc)o(hed)g(b)o(y)g(a)g (receiv)o(e)i(op)q(eration)e(using)g(a)75 914 y(general)d(datat)o(yp)q(e)f Fu(type2)f Fx(then)i(the)g(sequence)i(of)c(\(nonempt)o(y\))h(basic)g(t)o(yp)q (e)h(en)o(tries)g(de\014ned)h(b)o(y)e(eac)o(h)h(datat)o(yp)q(e)75 964 y(should)i(b)q(e)h(iden)o(tical.)j(F)m(urthermore,)13 b(eac)o(h)i(en)o (try)f(in)g(the)h(send)g(bu\013er)g(should)f(b)q(e)g(of)g(t)o(yp)q(e)g(that)h (matc)o(hes)e(the)75 1014 y(t)o(yp)q(e)h(of)g(the)g(corresp)q(onding)h(en)o (try)g(in)e Fu(type1)g Fx(and)h(eac)o(h)g(en)o(try)h(in)e(the)i(receiv)o(e)g (bu\013er)g(should)f(of)f(a)g(t)o(yp)q(e)i(that)75 1064 y(matc)o(hes)c(the)g (t)o(yp)q(e)h(of)f(the)g(corresp)q(onding)h(en)o(try)g(in)f Fu(type2)f Fx(\(with)h(t)o(yp)q(e)g(matc)o(hing)f(de\014ned)i(as)f(in)g (Section)g(1.5\).)75 1113 y(Th)o(us,)17 b(t)o(yp)q(e)h(matc)o(hing)d(is)h (de\014ned)i(b)o(y)f(the)g(\\\015at")f(structure)j(of)d(a)h(datat)o(yp)q(e,)g (and)g(do)q(es)g(not)g(dep)q(end)h(of)e(its)75 1163 y(de\014ning)e (expresssion.)158 1213 y(Example:)75 1296 y Fu(...)75 1346 y(MPI_TYPE_CONTIGUO)o(US\()19 b(MPI_REAL,)h(2,)h(type2\))75 1396 y(MPI_TYPE_CONTIGUO)o(US\()e(MPI_REAL,)h(4,)h(type4\))75 1446 y(MPI_TYPE_CONTIGUO)o(US\()e(type2,)h(2,)i(type22\))158 1529 y Fx(A)14 b(message)g(sen)o(t)g(with)g(datat)o(yp)q(e)g Fu(type4)f Fx(can)h(b)q(e)g(receiv)o(ed)i(with)d(datat)o(yp)q(e)h Fu(type22)p Fx(.)158 1578 y(T)o(yp)q(e)19 b(matc)o(hing)d(requiremen)o(ts)i (imply)e(that)j(\(nonempt)o(y\))e(en)o(tries)i(of)f(di\013eren)o(t)h(t)o(yp)q (es)g(in)f(a)g(datat)o(yp)q(e)75 1628 y(cannot)d(o)o(v)o(erlap,)e(unless)j (they)f(coincide)g(and)f(ha)o(v)o(e)h(the)g(same)f(basic)g(t)o(yp)q(e,)h(or)g (one)g(of)e(them)h(ha)o(v)o(e)h(basic)f(t)o(yp)q(e)75 1678 y Fu(byte)p Fx(.)158 1728 y(The)19 b(length)f(of)g(a)g(message)h(receiv)o(ed) g(\(i.e.,)g(the)g(v)n(alue)f(returned)i(when)f(the)g(receiv)o(e)h(status)f (ob)r(ject)g(is)75 1778 y(deco)q(ded)f(b)o(y)f(the)g(function)g Fu(MPI)p 608 1778 V 15 w(GET)p 689 1778 V 15 w(LEN)p Fx(\))f(is)h(equal)f(to) h(the)g(n)o(um)o(b)q(er)f(of)h(basic)g(en)o(tries)g(receiv)o(ed.)29 b(Consider)75 1827 y(again)13 b(the)i(datat)o(yp)q(es)h(de\014ned)f(in)f(the) h(last)g(example.)j(If)d(a)f(message)g(is)g(sen)o(t)i(using)e(either)h(datat) o(yp)q(e)g Fu(type22)75 1877 y Fx(or)d Fu(type4)f Fx(and)h(is)f(receiv)o(ed)j (using)d(either)i(of)f(these)h(t)o(w)o(o)e(datat)o(yp)q(es,)i(then)g(the)f (length)g(of)f(the)i(message)f(receiv)o(ed)75 1927 y(will)g(b)q(e)j(four.)158 2055 y Fo(Discussion:)158 2101 y Ft(An)f(alternativ)o(e)i(de\014nition)h (that)d(\014nds)h(fa)o(v)o(or)g(with)f(man)o(y)h(mem)o(b)q(ers)f(of)g(the)g (sub)q(committee)i(is)f(that)f(the)g(length)75 2147 y(of)g(the)g(message)h (receiv)o(ed)h(is)f(measured)g(in)g(terms)f(of)g(subunits)i(of)e(the)g (receiving)j(datat)o(yp)q(e.)k(Th)o(us,)15 b(if)f(four)h(reals)g(are)75 2192 y(receiv)o(ed)d(using)h Fm(type4)p Ft(,)c(the)i(length)i(of)e(the)g (receiv)o(ed)h(message)g(is)g(four;)f(if)h(they)f(are)g(receiv)o(ed)i(using)g Fm(type22)p Ft(,)8 b(the)j(length)75 2238 y(of)h(the)h(receiv)o(ed)h(message) g(is)f(t)o(w)o(o.)j(With)e(suc)o(h)g(de\014nition,)h(it)e(is)g(required)h (that)f(the)g(arriving)i(message)e(\014lls)i(an)e(in)o(teger)75 2284 y(n)o(um)o(b)q(er)h(of)f(subunits.)158 2329 y(The)f(rationale)i(is)e (that)g(the)g(these)g(are)g(the)g(units)h(that)f(are)f(meaningful)k(to)c(the) h(programmer.)18 b(Consider)13 b(the)f(case)75 2375 y(of)h(an)h(arra)o(y)f (of)g(structures)h(that)g(is)f(passed)i(in)f(a)f(message)h(\(example)g(4.1)g (b)q(elo)o(w\).)k(Then)13 b(one)h(migh)o(t)g(w)o(an)o(t)f(to)g(enforce)75 2421 y(the)i(restriction)h(that)f(only)h(complete)g(structures)f(are)g (passed;)h(and)f(is)g(natural)h(to)f(return)g(in)g(the)g(receiv)o(e)h(status) f(the)75 2466 y(n)o(um)o(b)q(er)c(of)f(structures)h(receiv)o(ed,)h(rather)f (than)g(this)g(n)o(um)o(b)q(er)g(m)o(ultipli)q(ed)i(b)o(y)e(14,)g(the)f(n)o (um)o(b)q(er)h(of)f(en)o(tries)i(p)q(er)e(structure.)158 2512 y(Consider,)19 b(on)f(the)f(other)g(hand,)i(the)e(case)h(of)f(a)g(sequence)h (of)f(reals)h(that)f(is)h(receiv)o(ed)h(in)e(a)h(3D)f(arra)o(y)h(section)75 2558 y(\(de\014ned)d(as)f(in)h(the)f(\014rst)g(example)i(b)q(elo)o(w\).)k(W)m (e)14 b(ma)o(y)g(w)o(an)o(t)g(to)g(receiv)o(e)h(an)f(arbitrary)i(n)o(um)o(b)q (er)e(of)g(reals,)h(and)f(the)g(1D)75 2603 y(and)g(2D)f(subsections)i(ma)o(y) e(not)h(carry)f(an)o(y)h(signi\014cance)h(for)e(the)g(purp)q(ose)i(of)d(this) i(comm)o(unication.)158 2649 y(Oppinions?)p eop %%Page: 31 31 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(31)158 45 y(A)19 b(call)e(of)h(the)i(form)c Fu(MPI)p 588 45 14 2 v 16 w(SEND\(start,)j(count,)i(datatype)f(,)h(...\))p Fx(,)e(where)g Fu(count)g Fj(>)g Fu(1)p Fx(,)g(is)g(in)o(ter-)75 95 y(preted)i(as)g(if)e(the)i(call)e(w)o(as)h(passed)h(a)f(new)g(datat)o(yp)q (e)g(whic)o(h)g(is)g(the)h(concatenation)f(of)g Fu(count)f Fx(copies)i(of)75 145 y Fu(datatype)p Fx(,)12 b(suc)o(h)j(as)f(w)o(ould)f(b)q (e)h(pro)q(duced)h(b)o(y)f(a)g(call)f(to)h Fu(MPI)p 1067 145 V 15 w(CONTIGUOUS\(count,)k(datatype,)i(newtype\))p Fx(,)75 195 y(and)12 b(a)f Fu(count)g Fx(argumen)o(t)f(of)h(one.)18 b(The)12 b(same)f(holds)h(true)g(for)g(the)g(other)g(comm)o(unication)d (functions)i(that)h(ha)o(v)o(e)75 244 y(a)i(datat)o(yp)q(e)g(argumen)o(t.)75 361 y Fs(1.13.4)49 b(Examples)75 437 y Fx(The)14 b(follo)o(wing)e(examples)h (illustrate)g(the)i(use)f(of)g(deriv)o(ed)g(datat)o(yp)q(es.)158 537 y(First)g(example:)j(extract)e(a)f(section)g(of)f(a)h(3D)f(matrix.)206 620 y Fu(REAL)21 b(a\(100,100,100\),)d(e\(9,9,9\))206 670 y(INTEGER)i (oneslice,)g(twoslice,)g(threeslice,)f(finaltype,)h(sizeofreal)75 769 y(C)131 b(extract)20 b(the)h(section)f(a\(1:17:2,)g(3:11,)h(2:10\))75 819 y(C)131 b(and)21 b(store)f(it)i(in)f(e\(*,*,*\).)206 919 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 1018 y(C)109 b(create)20 b(datatype)g(for)i(a)f(1D)h(section)206 1068 y(CALL)f(MPI_TYPE_VECTOR\()d(9,)k(MPI_REAL,)41 b(2,)22 b(1,)f(oneslice,)f(ierr\))75 1168 y(C)109 b(create)20 b(datatype)g(for)i(a)f (2D)h(section)206 1218 y(CALL)f(MPI_TYPE_HVECTOR)o(\()e(9,)i(oneslice,)42 b(100*sizeofreal,)19 b(1,)i(twoslice,)f(ierr\))75 1317 y(C)109 b(create)20 b(datatype)g(for)i(the)f(entire)f(section)206 1367 y(CALL)h(MPI_TYPE_HVECTOR)o(\()e(9,)i(twoslice,)42 b(100*100*sizeofreal)o(,) 19 b(1,)467 1417 y(threeslice,)h(ierr\))206 1516 y(CALL)h(MPI_TYPE_COMMIT\()d (threeslice,)i(finaltype,)f(ierr\))206 1566 y(CALL)i(MPI_SENDRECV\(a\(1)o (,3,2\))o(,)e(1,)i(finaltype,)f(MPI_ME,)g(e,)i(9*9*9,)467 1616 y(MPI_REAL,)e(MPI_ME,)h(0,)g(MPI_ALL,)f(status,)g(ierr\))158 1699 y Fx(Second)15 b(example:)h(Cop)o(y)e(the)g(\(strictly\))h(lo)o(w)o(er)e (triangular)g(part)h(of)g(a)f(matrix.)206 1782 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 1832 y(INTEGER)42 b(index\(100\),)19 b(blocklen\(100\),)g(ltype,)i(finaltype)75 1932 y(C)109 b(copy)21 b(lower)f(triangular)g(part)h(of)h(array)e(a)75 1981 y(C)109 b(onto)21 b(lower)f(triangular)g(part)h(of)h(array)e(b)75 2081 y(C)109 b(compute)20 b(start)h(and)g(size)g(of)g(each)g(column)206 2131 y(DO)g(i=1,)g(100)249 2181 y(index\(i\))f(=)i(100*\(i-1\))e(+)i(i)249 2230 y(block\(i\))e(=)i(100-i)206 2280 y(END)f(DO)75 2380 y(C)109 b(create)20 b(datatype)g(for)i(lower)e(triangular)g(part)206 2430 y(CALL)h(MPI_TYPE_INDEX\()d(MPI_REAL,)i(100,)h(index,)g(block,)f(ltype,) h(ierr\))206 2529 y(CALL)g(MPI_TYPE_COMMIT\()o(ltype)o(,)e(finaltype,)h (ierr\))206 2579 y(CALL)h(MPI_SENDRECV\()e(a,)i(1,)h(finaltype,)d(MPI_ME,)i (b,)g(1,)424 2629 y(finaltype,)e(MPI_ME,)i(0,)g(MPI_ALL,)f(status,)h(ierr\))p eop %%Page: 32 32 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(32)158 145 y(Third)14 b(example:)i(T)m(ransp)q(ose)f(a)f (matrix)206 212 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 261 y(INTEGER)h(row,)h(xpose,)g(finaltype,)e(sizeofreal)75 361 y(C)109 b(transpose)20 b(matrix)g(a)i(onto)f(b)206 510 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 610 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row)206 660 y(CALL)g(MPI_TYPE_VECTOR\()d(100,)j(MPI_REAL,)f(100,)h(1,)g(row,)g(ierr\))75 760 y(C)109 b(create)20 b(datatype)g(for)i(matrix)e(in)i(row-major)d(order) 206 809 y(CALL)i(MPI_TYPE_HVECTOR)o(\()e(100,)i(row,)g(sizeofreal,)e(1,)j (xpose,)e(ierr\))206 909 y(CALL)h(MPI_TYPE_COMMIT\()d(xpose,)j(finaltype,)e (ierr\))75 1009 y(C)109 b(send)21 b(matrix)f(in)i(row-major)e(order)g(and)i (receive)e(in)h(column)g(major)g(order)206 1058 y(CALL)g(MPI_SENDRECV\()e(a,) i(1,)h(finaltype,)d(MPI_ME,)i(b,)g(100*100,)424 1108 y(MPI_REAL,)f(MPI_ME,)g (0,)h(MPI_ALL,)f(status,)h(ierr\))158 1275 y Fx(Another)15 b(approac)o(h)f(to)f(the)i(transp)q(ose)g(problem:)206 1342 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 1391 y(INTEGER)42 b(index\(2\),)20 b(blen\(2\),)g(type\(2\),)g(row,)h(row1,)g(finaltype,)f (sizeofreal)75 1541 y(C)109 b(transpose)20 b(matrix)g(a)i(onto)f(b)206 1690 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 1790 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row)206 1840 y(CALL)g(MPI_TYPE_VECTOR\()d(MPI_REAL,)i(100,)h(100,)g(1,)g(row,)g(ierr\))75 1939 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row,)g(with)g(the)g(extent) f(of)i(one)f(real)g(number)206 1989 y(index\(1\))f(=)h(0)206 2039 y(index\(2\))f(=)h(sizeofreal)206 2089 y(type\(1\))42 b(=)21 b(row)206 2139 y(type\(2\))42 b(=)21 b(MPI_UB)206 2188 y(blen\(1\))42 b(=)21 b(1)206 2238 y(blen\(2\))42 b(=)21 b(1)206 2288 y(CALL)g(MPI_TYPE_STRUC\()d(2,)k(type,)f(index,)f(blen,)h(row1\))206 2388 y(CALL)g(MPI_TYPE_COMMIT\()d(row1,)j(finaltype,)f(ierr\))75 2487 y(C)109 b(send)21 b(100)g(rows)g(and)g(receive)f(in)i(column)e(major)h (order)206 2537 y(CALL)g(MPI_SENDRECV\()e(a,)i(100,)g(finaltype,)f(MPI_ME,)g (b,)i(100*100,)424 2587 y(MPI_REAL,)e(MPI_ME,)g(0,)h(MPI_ALL,)f(status,)h (ierr\))p eop %%Page: 33 33 bop 75 -100 a Fq(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(33)158 45 y(F)m(ourth)14 b(example:)j(manipulate)11 b(an)j(arra)o(y)g(of)f(structures.)75 113 y Fu(struct)21 b(Partstruct)140 163 y({)140 213 y(int)h(index;)85 b(/*)22 b(particle)e(type)h(*/)140 263 y(double)g(d[6])43 b(/*)21 b(particle)f(coordinates)g(*/)140 313 y(char)h(b[7])87 b(/*)21 b(some)g(additional)f(information)f(*/)140 362 y(};)75 462 y(struct)i(Partstruct)e(particle[1000];)315 562 y(/*)i(build)g(datatype)f(describing)g(first)g(array)h(entry)g(*/)75 661 y(MPI_datatype)e(Particletype;)75 711 y(MPI_datatype)g(stype[3])h(=)i ({MPI_int,)e(MPI_double,)f(MPI_char};)75 761 y(int)217 b(sblock[3])20 b(=)i({1,)f(6,)g(7};)75 811 y(int)217 b(sindex[3];)75 910 y(MPI_address\()19 b(\(void)i(*\)particle,)f(sindex\);)75 960 y(MPI_address\()f(\(void)i (*\)particle[0].d,)e(sindex+1\);)75 1010 y(MPI_address\()g(\(void)i (*\)particle[0].b,)e(sindex+2\);)75 1060 y(MPI_type_struc\()g(3,)i(stype,)g (sindex,)f(sblock,)g(Particletype\);)75 1110 y(/*)h(Particletype)f(describes) g(first)g(array)h(entry)g(--)g(using)g(absolute)140 1159 y(addresses)f(*/)249 1309 y(/*)i(4.1:)f(send)g(the)g(entire)f(array)h(*/)75 1408 y(MPI_datatype)e(Type1;)75 1458 y(MPI_commit\()h(Particletype,)f(Type1\);)75 1508 y(MPI_send\()h(MPI_BOTTOM,)f(1000,)i(Type1,)g(dest,)g(tag,)f(comm\);)249 1658 y(/*)i(4.2:)f(send)g(the)g(entries)f(with)h(index)g(value)g(zero)g(*/)75 1757 y(MPI_datatype)e(Zparticles;)63 b(/*)22 b(datatype)e(describing)g(all)h (particles)860 1807 y(with)g(index)f(zero)h(\(needs)g(to)g(be)h(recomputed) 860 1857 y(if)f(indices)f(change\))h(*/)75 1907 y(MPI_datatype)e(Type2;)75 2006 y(int)i(zindex[1000],)e(zblock[1000],)g(i,)j(j,)f(k;)75 2106 y(/*)g(compute)g(indices)f(of)h(particles)f(with)h(index)g(zero)g(*/)75 2156 y(j)h(=)f(0;)75 2205 y(for\(i=0;)f(i)i(<)f(1000;)g(i++\))119 2255 y(if)g(\(particle[i].inde)o(x==0\))d(then)184 2305 y({)184 2355 y(zindex[j])i(=)h(i;)184 2405 y(zblock[i])f(=)h(1;)184 2455 y(j++;)184 2504 y(})75 2604 y(/*)g(create)g(datatype)f(for)h(particles)f (with)h(index)g(zero)g(*/)75 2654 y(MPI_type_index\()e(j,)i(Particletype,)41 b(zindex,)20 b(zblock,)h(Zparticles\);)75 2704 y(MPI_type_commit\()d (Zparticles,)i(Type2\);)p eop %%Page: 34 34 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(34)75 95 y Fu(MPI_send\()20 b(MPI_BOTTOM,)f(1,)j(Type2,)e (dest,)h(tag,)g(comm\);)249 244 y(/*)h(A)f(possibly)f(more)h(efficient)f(way) i(of)f(defining)f(Zparticles)g(*/)75 344 y(/*)h(consecutive)f(particles)g (with)h(index)g(zero)g(are)g(handled)f(as)h(one)h(block)e(*/)75 394 y(j=0;)75 444 y(for)h(\(i=0;)g(i)h(<)f(1000;)g(i++\))119 493 y(if)g(\(particle[i].inde)o(x==0\))d(then)162 543 y({)162 593 y(for)j(\(k=i+1;)g(\(k)g(<)h(1000\)&&\(particle)o([k].i)o(ndex)c(==)k (0\))f(;)h(k++\);)162 643 y(zindex[j])e(=)i(i;)162 693 y(zblock[j])e(=)i (k-i;)162 742 y(j++;)162 792 y(i)g(=)f(k;)162 842 y(})75 892 y(MPI_type_index\()e(j,)i(Particletype,)e(zindex,)i(zblock,)f(Zparticles\);) 249 1041 y(/*)i(4.3:)f(send)g(the)g(first)g(two)g(coordinates)e(of)j(all)f (entries)f(*/)75 1141 y(MPI_datatype)f(Allpairs;)129 b(/*)22 b(datatype)e(for)h(all)g(pairs)g(of)g(coordinates)f(*/)75 1191 y(MPI_datatype)f(Type3;)75 1290 y(int)217 b(sizeofentry;)75 1390 y(MPI_extent\()20 b(Particletype,)f(sizeofentry\);)184 1490 y(/*)i(sizeofentry)f(can)h(also)g(be)g(computed)g(by)g(subtracting)e (the)j(address)249 1539 y(of)g(particle[0])d(from)i(the)g(address)g(of)g (particle[1])f(*/)75 1639 y(MPI_type_hvector\()e(1000,)j(MPI_real,)f (sizeofentry,)f(2,)j(Allpairs\);)75 1689 y(MPI_type_commit\()c(Allpairs,)i (Type3\);)75 1739 y(MPI_send\()g(particle.d,)f(1,)j(Type3,)e(dest,)h(tag,)g (comm\);)206 1838 y(/*)g(an)h(alternative)d(solution)h(to)i(4.3)f(*/)75 1938 y(MPI_datatype)e(Onepair;)64 b(/*)21 b(datatype)g(for)g(one)g(pair)g(of) g(coordinates,)f(with)642 1988 y(the)h(extent)f(of)i(one)f(particle)f(entry)h (*/)75 2038 y(MPI_datatype)e(Type4;)75 2137 y(int)i(indexp[3];)75 2187 y(int)g(typep[3])f(=)i({MPI_lb,)e(MPI_double,)g(MPI_ub};)75 2237 y(int)h(blockp[3])f(=)i({1,)f(2,)g(1};)75 2337 y(MPI_address\()e(\(void) i(*\)particle,)f(indexp\);)75 2386 y(MPI_address\()f(\(void)i (*\)particle[0].d,)e(indexp+1\);)75 2436 y(MPI_address\()g(\(void)i (*\)\(particle+1\),)e(indexp+2\);)75 2536 y(MPI_type_struc\()g(3,)i(typep,)g (indexp,)f(blockp,)g(Onepair\);)75 2586 y(MPI_commit\()g(Onepair,)g(Type4\);) 75 2635 y(MPI_send\()g(MPI_bottom,)f(1000,)i(Type4,)g(dest,)g(tag,)f(comm\);) p eop %%Page: 35 35 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(35)75 45 y Fs(1.13.5)49 b(Co)o(rrect)15 b(use)h(of)g (addresses)75 122 y Fx(Successiv)o(ely)i(declared)g(v)n(ariables)f(in)f(C)h (or)g(F)m(ortran)g(are)g(not)g(secessarily)h(stored)g(at)f(con)o(tiguous)g (lo)q(cations.)75 172 y(Th)o(us,)f(care)g(m)o(ust)f(b)q(e)h(exercised)i(that) e(displacemen)o(ts)f(do)g(not)h(cross)h(from)d(one)i(v)n(ariable)e(to)i (another.)24 b(Also,)75 221 y(in)18 b(mac)o(hines)f(with)g(segmen)o(ted)h (address)i(space,)f(addresses)h(are)f(not)f(unique)g(and)g(address)h (arithmetic)e(has)75 271 y(some)e(p)q(eculiar)g(prop)q(erties.)25 b(Th)o(us,)16 b(use)g(of)f Fv(addresses)p Fx(,)f(i.e.)22 b(displacemen)o(ts) 15 b(relativ)o(e)h(to)f(the)h(start)h(address)75 321 y Fu(MPI)p 144 321 14 2 v 15 w(BOTTOM)p Fx(,)12 b(has)i(to)g(b)q(e)g(restricted.)158 371 y(V)m(alid)e(addresses)k(are)f(de\014ned)g(recursiv)o(ely)g(as)f(follo)o (ws:)126 459 y(1.)20 b(The)15 b(function)f Fu(MPI)p 497 459 V 15 w(ADDRESS)f Fx(returns)j(a)e(v)n(alid)f(address,)j(when)f(passed)g(as)g (argumen)o(t)f(a)g(host)h(program)179 509 y(v)n(ariable.)126 590 y(2.)20 b(The)14 b Fu(start)g Fx(parameter)g(of)f(a)h(comm)o(unication)d (function)j(ev)n(aluates)g(to)g(a)g(v)n(alid)f(address,)i(when)g(passed)179 640 y(as)f(argumen)o(t)f(a)g(host)h(program)f(v)n(ariable.)126 722 y(3.)20 b(If)13 b Fu(v)g Fx(is)g(a)g(v)n(alid)f(address,)i(and)f Fu(i)g Fx(is)g(an)g(in)o(teger,)g(then)h Fu(v+i)f Fx(is)g(a)g(v)n(alid)f (address,)i(pro)o(vided)f(that)h Fu(v+i)e Fx(do)q(es)179 772 y(not)i(exten)o(t)g(b)q(ey)o(ond)h(the)f(outermost)f(arra)o(y)h(or)g (structure)i(that)e(con)o(tains)g Fu(v)p Fx(.)126 853 y(4.)20 b(If)13 b Fu(v)h Fx(is)g(a)f(v)n(alid)g(address)i(then)f Fu(MPI)p 744 853 V 15 w(BOTTOM)21 b(+)h(v)13 b Fx(is)h(a)f(v)n(alid)g(address.)158 942 y(A)e(correct)h(program)d(uses)i(only)f(v)n(alid)f(addresses)k(to)d(iden) o(tify)g(the)h(lo)q(cations)f(of)g(en)o(tries)h(in)f(comm)o(unication)75 991 y(bu\013ers.)29 b(F)m(urthermore,)18 b(if)e Fu(u)h Fx(and)g Fu(v)g Fx(are)g(t)o(w)o(o)g(v)n(alid)f(addresses,)j(then)f(the)g(\(in)o (teger\))g(di\013erence)h Fu(u)i(-)h(v)17 b Fx(can)75 1041 y(b)q(e)h(computed)g(only)f(if)g(b)q(oth)h Fu(u)f Fx(and)h Fu(v)g Fx(are)g(comp)q(onen)o(ts)f(within)g(the)i(same)e(arra)o(y)g(or)h (structure;)k(no)17 b(other)75 1091 y(arithmetic)c(op)q(erations)h(can)g(b)q (e)g(meaningfully)d(executed)16 b(on)e(addresses.)158 1141 y(W)m(e)c(sa)o(y)h(that)f(a)h(datat)o(yp)q(e)f(is)h Fv(absolute)d Fx(if)i(all)f(displacemen)o(ts)h(within)g(this)h(datat)o(yp)q(e)g(are)g(v)n (alid)e(\(absolute\))75 1191 y(addresses;)16 b(it)d(is)h Fv(relativ)o(e)d Fx(otherwise.)158 1240 y(A)j(correct)i(program)c(ob)q(eys)i(the)h(follo)o (wing)c(constrain)o(ts)k(in)e(the)i(use)f(of)g(datat)o(yp)q(es:)137 1328 y Fn(\017)21 b Fx(If)9 b(the)h Fu(oldtype)d Fx(argumen)o(t)h(used)j(in)d Fu(MPI)p 828 1328 V 16 w(TYPE)p 932 1328 V 14 w(CONTIGUOUS,)20 b(MPI)p 1274 1328 V 15 w(TYPE)p 1377 1328 V 15 w(VECTOR,)g(MPI)p 1632 1328 V 15 w(TYPE)p 1735 1328 V 15 w(HVECTOR,)179 1378 y(MPI)p 248 1378 V 15 w(TYPE)p 351 1378 V 15 w(INDEX)12 b Fx(or)i Fu(MPI)p 605 1378 V 15 w(TYPE)p 708 1378 V 15 w(HINDEX)e Fx(is)h(absolute,)g (then)h(all)f(addresses)i(within)e(it)g(m)o(ust)g(b)q(e)h(to)f(v)n(ari-)179 1428 y(ables)i(con)o(tained)h(within)e(the)i(same)f(arra)o(y)g(or)g (structure;)j(the)e(result)g(datat)o(yp)q(e)g(is)f(also)g(absolute,)g(and)179 1478 y(all)d(computed)i(addresses)i(m)o(ust)d(also)g(fall)f(within)h(the)i (same)e(arra)o(y)h(or)f(structure.)137 1560 y Fn(\017)21 b Fx(If)12 b(all)e(en)o(tries)k(of)d(the)i Fu(array)p 634 1560 V 14 w(of)p 692 1560 V 16 w(types)e Fx(argumen)o(t)g(of)g Fu(MPI)p 1124 1560 V 15 w(TYPE)p 1227 1560 V 15 w(STRUC)g Fx(are)i(absolute)f (addresses)i(\(com-)179 1609 y(puted)j(b)o(y)f Fu(MPI)p 428 1609 V 15 w(ADDR)p Fx(\),)f(then)i(the)g(result)g(datat)o(yp)q(e)f(is)h(also) e(absolute.)26 b(Eac)o(h)16 b(new)h(address)g(computed)179 1659 y(from)12 b(an)i(old)g(address)h(m)o(ust)f(fall)e(within)i(the)h(same)e (arra)o(y)h(or)h(record)g(as)f(the)h(old)f(address.)20 b(\(Addresses)179 1709 y(in)13 b(the)i(old)e(t)o(yp)q(e)h(ma)o(y)e(not)i(b)q(e)h(all)d(within)h (the)i(same)e(arra)o(y)g(or)h(record.\))137 1791 y Fn(\017)21 b Fx(If)12 b(a)g(comm)o(unication)e(call)h(is)i(in)o(v)o(ok)o(ed)f(with)g (parameters)h Fu(start)e Fx(and)i Fu(datatype)p Fx(,)e(then)i(either)g Fu(start)21 b(=)179 1841 y(MPI)p 248 1841 V 15 w(BOTTOM)12 b Fx(and)i Fu(datatype)e Fx(is)h(a)h(handle)f(to)h(an)f(absolute)h(datat)o (yp)q(e,)f(or)h Fu(start)f Fx(is)g(set)i(to)e(the)i(address)179 1890 y(of)e(a)h(program)e(v)n(ariable,)g(and)i(all)f(displacemen)o(ts)g(in)h Fu(datatype)e Fx(relativ)o(e)h(to)h(this)g(base)h(yield)e(addresses)179 1940 y(that)h(are)g(within)f(the)i(same)e(arra)o(y)g(or)h(record)h(as)f(this) g(v)n(ariable.)158 2107 y Fo(Discussion:)158 2153 y Ft(It)d(is)i(not)f(exp)q (ected)h(that)f(MPI)g(implemen)o(tations)j(will)e(b)q(e)g(able)g(to)e(detect) h(erroneous,)h(\\out)g(of)e(b)q(ound")j(displace-)75 2198 y(men)o(ts)h({)g (unless)i(those)f(o)o(v)o(er\015o)o(w)f(the)h(user)f(address)h(space)g({)f (since)h(the)g(MPI)f(call)h(ma)o(y)g(not)f(no)o(w)g(the)g(exten)o(t)h(of)f (the)75 2244 y(arra)o(ys)f(and)f(records)h(in)g(the)f(host)g(program.)75 2463 y Fw(1.14)59 b(Universal)20 b(comm)m(unica)o(tion)d(functions)75 2554 y Fx(Although)11 b(the)i(n)o(um)o(b)q(er)e(of)g(comm)o(unicati)o(on)e (functions)j(de\014ned)h(in)e(this)h(c)o(hapter)g(is)g(fairly)e(large,)i(it)f (is)h(p)q(ossible)75 2604 y(to)k(construct)j(all)c(of)h(them)g(from)f(a)h(m)o (uc)o(h)g(smaller)f(n)o(um)o(b)q(er)h(of)g(primitiv)o(e)e(functions.)26 b(The)17 b(construction)h(is)75 2654 y(outlined)c(in)g(this)h(section.)20 b(The)15 b(construction)h(can)f(b)q(e)g(used)g(to)g(build)e(a)i(p)q(ortable)f (implemen)o(tation)d(of)j(MPI,)75 2704 y(although)j(it)g(is)g(exp)q(ected)i (that)f(more)e(e\016cien)o(t)i(implemen)o(tation)c(will)i(implemen)o(t)f (directly)j(most)e(or)h(all)g(of)p eop %%Page: 36 36 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(36)75 45 y(the)15 b(functions)f(in)g(this)h(section.)20 b(Also,)14 b(this)g(construction)i(can)e(b)q(e)h(used)g(in)f(order)h(to)g (deriv)o(e)f(the)h(b)q(eha)o(vior)g(of)75 95 y(MPI)i(p)q(oin)o(t)f(to)h(p)q (oin)o(t)f(comm)o(unicatio)o(n)e(from)h(the)i(b)q(eha)o(vior)g(of)f(a)g (small)e(subset)19 b(of)d(functions.)26 b(W)m(e)16 b(assume)75 145 y(the)e(a)o(v)n(ailabili)o(t)o(y)c(of)j(the)h(primitiv)o(e)d(functions)i (listed)g(at)g(the)h(end)g(of)f(the)g(section,)h(and)f(the)h(abilit)o(y)d(to) i(allo)q(cate)75 195 y(memory)e(for)j(new)g(v)n(ariables.)158 244 y(W)m(e)d(assume)f(that)h(the)h(function)f Fu(MPI)p 752 244 14 2 v 15 w(TYPE)p 855 244 V 15 w(COMMIT)f Fx(is)g(not)h(c)o(hanging)g (the)g(represen)o(tation)i(of)d(a)h(datat)o(yp)q(e,)75 294 y(but)j(merely)f(creates)j(a)d(new)i(cop)o(y)f(of)f(it.)158 344 y(Comm)o(uni)o(cation)i(with)j(a)f Fu(count)g Fx(argumen)o(t)g(that)h(is) g Fj(>)h Fx(1)e(can)i(b)q(e)f(reduced)i(to)e(comm)o(unicatio)o(n)d(with)75 394 y Fu(count)21 b(=1)p Fx(,)d(b)o(y)g(replicating)g(the)h Fu(datatype)e Fx(argumen)o(t)g Fu(count)g Fx(times.)31 b(Comm)o(uni)o(cation) 15 b(with)j(a)g Fu(a)k(start)75 444 y Fx(argumen)o(t)14 b(that)i(is)f(not)g Fu(MPI)p 541 444 V 15 w(BOTTOM)f Fx(can)i(b)q(e)g(reduced)h(to)e(comm)o (unicatio)o(n)e(with)i Fu(start)20 b(=)i(MPI)p 1673 444 V 15 w(BOTTOM)14 b Fx(b)o(y)75 493 y(creating)g(a)g(suitable)g(displaced)g(new)g (datat)o(yp)q(e.)k(Th)o(us:)158 543 y Fu(MPI)p 227 543 V 15 w(SEND\()j(start,)f(count,)h(datatype,)f(dest,)h(tag,)g(comm\))12 b Fx(is)75 626 y Fu(Type[1])20 b(=)i(datatype)75 676 y(MPI_ADDRESS\()d (start,)i(Index[1]\);)75 726 y(Block[1])f(=)i(count;)75 776 y(MPI_TYPE_STRUCT\()c(1,)k(Type,)f(Index,)f(Block,)h(Newtype\))75 826 y(MPI_TYPE_COMMIT\()d(Newtype,)j(datatype1\))75 875 y(MPI_SEND\()f (MPI_BOTTOM,)f(1,)j(datatype1,)e(dest,)g(tag,)h(comm\))158 958 y Fx(The)13 b(same)e(construction)i(applies)f(to)g(all)f(other)i(comm)o (unicatio)o(n)d(functions)i(with)g(argumen)o(ts)f Fu(count)g Fx(and)75 1008 y Fu(datatype)p Fx(.)k(W)m(e)10 b(shall)g(henceforw)o(ard)h (restrict)h(ourselv)o(es)g(to)e(comm)o(unication)d(that)k(in)o(v)o(olv)o(es)e (only)h(one)g(elemen)o(t)75 1058 y(with)k(an)f(absolute)h(address)h(\()p Fu(count)21 b(=1)p Fx(,)13 b(and)h Fu(start)21 b(=)g(MPI)p 1079 1058 V 15 w(BOTTOM)p Fx(\).)158 1108 y(\(If)15 b(the)h(op)q(eration)f Fu(MPI)p 546 1108 V 15 w(TYPE)p 649 1108 V 15 w(COMMIT\))f Fx(is)h(not)h(assumed)f(to)g(b)q(e)h(trivial,)e(then)i(the)g(ab)q(o)o(v)o(e)f (construction)75 1158 y(cannot)h(b)q(e)h(used;)h(this)e(adds)h(one)f(more)f (primitiv)o(e)f(function,)i(namely)f Fu(MPI)p 1322 1158 V 15 w(TYPE)p 1425 1158 V 15 w(COMMIT)p Fx(,)f(to)i(our)g(list,)g(and)75 1207 y(in)o(tro)q(duces)f(small)d(c)o(hanges)i(in)g(the)g(construction.\))158 1257 y(It)j(is)h(con)o(v)o(enien)o(t)f(to)g(to)h(mak)o(e)d(explicit)i(the)h Fv(p)q(ersistence)d Fx(attribute)j(of)e(comm)o(unication)e(ob)r(jects.)30 b(A)75 1307 y Fu(persistent)16 b Fx(comm)o(unicatio)o(n)f(ob)r(ject)k(need)g (b)q(e)g(explicitly)e(deallo)q(cated)h(b)o(y)g(a)f Fu(MPI)p 1471 1307 V 16 w(COMM)p 1575 1307 V 14 w(FREE)h Fx(op)q(eration.)75 1357 y(On)d(the)h(other)f(hand,)g(an)f Fu(ephemeral)f Fx(comm)o(unication)f (ob)r(ject)k(is)e(go)q(o)q(d)h(for)f(a)h(single)g(comm)o(uni)o(cation.)j(It)d (is)75 1407 y(deallo)q(cated)f(b)o(y)g(the)g(system)g(when)g(the)h(\014rst)f (comm)o(unication)d(it)i(is)h(used)h(for)e(completes.)158 1456 y(The)h(function)g Fu(MPI)p 475 1456 V 15 w(COMM)p 578 1456 V 15 w(INIT)f Fx(is)g(a)h(univ)o(ersal)f(function)h(for)f(the)i(creation)f (of)f(comm)o(unicatio)o(n)e(ob)r(jects:)75 1554 y Fu(MPI)p 144 1554 V 15 w(COMM)p 247 1554 V 15 w(INIT\(handle,)19 b(datatype,)h (source-dest,)f(tag,)i(comm,)g(op-mode,)f(persistence\))117 1603 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(newly)h(created)g (comm)o(unication)h(ob)r(ject)117 1649 y(in)f Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(elemen)o(t)g(sen)o(t)h(or)f(receiv)o(ed)117 1695 y(in)h Fu(source-dest)543 b Ft(rank)13 b(of)g(destination)j(or)d(source) g(\(in)o(teger\))117 1740 y(in)h Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1786 y(in)g Fu(comm)697 b Ft(comm)o(unication)15 b(ob)r(ject)117 1832 y(in)f Fu(op-mode)631 b Ft(one)19 b(of)g Fm(MPI)p 1135 1832 12 2 v 13 w(STANDARD)p Ft(,)d Fm(MPI)p 1395 1832 V 13 w(READY)p Ft(,)g Fm(MPI)p 1595 1832 V 13 w(SYNCHRONOU)o(S)g Ft(or)905 1881 y Fm(MPI)p 967 1881 V 13 w(RECV)117 1927 y Ft(in)e Fu(persistence)543 b Ft(one)13 b(of)g Fm(MPI)p 1123 1927 V 13 w(PERSISTEN)o(T)d Ft(or)j Fm(MPI)p 1452 1927 V 13 w(EPHEMERAL)158 2003 y Fx(The)h(comm)o(unication)d(in)o(v)o(olv)o(es)i(only)g(one)h(elemen)o (t,)f(with)g(an)h(absolute)g(address.)75 2120 y Fs(1.14.1)49 b(P)o(ersistent)14 b(comm)o(unication)g(objects)75 2196 y Fu(MPI)p 144 2196 14 2 v 15 w(CREATE)p 291 2196 V 15 w(SEND\(handle,)19 b(MPI)p 655 2196 V 15 w(BOTTOM,)h(1,)i(datatype,)e(dest,)g(tag,)h(comm\))13 b Fx(is)75 2288 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,) g(comm,)g(MPI_STANDARD,)e(MPI_PERSISTENT\))158 2379 y Fx(The)14 b(functions)g Fu(MPI)p 491 2379 V 15 w(CREATE)p 638 2379 V 15 w(RSEND)p Fx(,)e Fu(MPI)p 853 2379 V 15 w(CREATE)p 1000 2379 V 14 w(RSEND)h Fx(and)h Fu(MPI)p 1284 2379 V 15 w(CREATE)p 1431 2379 V 14 w(RECV)f Fx(are)h(dealt)g(in)f(a)h(sim-)75 2429 y(ilar)f(manner.)158 2479 y(The)h(functions)g Fu(MPI)p 491 2479 V 15 w(START)f Fx(and)h Fu(MPI)p 776 2479 V 15 w(COMM)p 879 2479 V 15 w(FREE)f Fx(are)h(primitiv)o(e.)p eop %%Page: 37 37 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(37)75 45 y Fs(1.14.2)49 b(Nonblo)q(cking)19 b(comm)n(unication)14 b(initiation)75 122 y Fu(MPI)p 144 122 14 2 v 15 w(ISEND\(handle,)19 b(MPI)p 530 122 V 15 w(BOTTOM,)i(1,)g (datatype,)f(dest,)h(tag,)g(comm\))12 b Fx(is)75 213 y Fu(MPI_COMM_INIT\(han) o(dle,)18 b(datatype,)i(dest,)h(tag,)g(comm,)g(MPI_STANDARD,)e (MPI_EPHEMERAL\))75 263 y(MPI_START\(handle\))158 354 y Fx(The)14 b(functions)g Fu(MPI)p 491 354 V 15 w(IRSEND)p Fx(,)f Fu(MPI)p 729 354 V 15 w(ISSEND)f Fx(and)i Fu(MPI)p 1035 354 V 15 w(IRECV)f Fx(are)h(handled)g(in)f(a)h(similar)d(manner.)75 470 y Fs(1.14.3)49 b(Comm)o(unication)14 b(completion)75 547 y Fx(The)g(t)o(w)o(o)g(primitiv)o (e)d(completion)i(op)q(erations)h(are)g Fu(MPI)p 969 547 V 15 w(WAITANY)e Fx(and)i Fu(MPI)p 1297 547 V 15 w(TESTALL)p Fx(.)158 597 y(The)c(function)f Fu(MPI)p 466 597 V 15 w(WAIT)f Fx(can)i(b)q(e)g(implemen)o(ted)d(b)o(y)i(a)g(call)g(to)g Fu(MPI)p 1209 597 V 15 w(WAITANY)f Fx(with)h(an)g Fu(array-of-handles)75 647 y Fx(argumen)o(t)k(of)g(length)h(one.)158 696 y(The)j(function)e Fu(MPI)p 479 696 V 15 w(WAITALL)g Fx(can)h(b)q(e)h(implemen)o(ted)d(as)i(a)g (lo)q(op)f(where)j(an)d Fu(MPI)p 1479 696 V 15 w(WAIT)h Fx(call)f(is)h (executed)75 746 y(for)e(eac)o(h)g(successiv)o(e)i(handle)e(in)f(the)i Fu(array-of-handles)o Fx(.)158 796 y(The)d(function)e Fu(MPI)p 469 796 V 15 w(TEST)h Fx(can)g(b)q(e)g(implemen)o(ted)e(b)o(y)i(a)g(call)f (to)h Fu(MPI)p 1226 796 V 15 w(TESTALL)f Fx(with)g(an)h Fu(array-ofhandles)75 846 y Fx(of)i(length)h(one.)158 896 y(The)j(function)e Fu(MPI)p 479 896 V 15 w(TESTALL)g Fx(can)h(b)q(e)h(implemen)o(ted)d(as)i(a)g(lo)q(op)f (where)j(an)d Fu(MPI)p 1479 896 V 15 w(TEST)h Fx(call)f(is)h(executed)75 946 y(for)e(eac)o(h)g(successiv)o(e)i(handle)e(in)f(the)i Fu(array-of-handle) p Fx(.)75 1062 y Fs(1.14.4)49 b(Blo)q(cking)18 b(comm)o(unication)75 1138 y Fu(MPI)p 144 1138 V 15 w(SEND\(MPI)p 335 1138 V 14 w(BOTTOM,)i(1,)i (datatype,)e(dest,)h(tag,)f(comm\))13 b Fx(is)75 1230 y Fu (MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,)g(comm,)g (MPI_STANDARD,)e(MPI_EPHEMERAL\))75 1279 y(MPI_START\(handle\))75 1329 y(MPI_WAIT\(handle,)f(dontcarestatus\))158 1421 y Fx(The)c(functions)g Fu(MPI)p 491 1421 V 15 w(RSEND)f Fx(and)h Fu(MPI)p 776 1421 V 15 w(SSEND)f Fx(are)h(handled)g(in)f(a)h(similar)d(manner.)158 1470 y Fu(MPI)p 227 1470 V 15 w(RECV\(MPI)p 418 1470 V 14 w(BOTTOM,)20 b(1,)i(datatype,)e(source,)g(tag,)h(comm\))13 b Fx(is)75 1562 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,)g(comm,)g (MPI_RECV,)f(MPI_EPHEMERAL\))75 1612 y(MPI_START\(handle\))75 1661 y(MPI_WAIT\(handle,)e(status\))75 1778 y Fs(1.14.5)49 b(Prob)q(e)17 b(and)f(cancel)75 1854 y Fx(The)e(functions)g Fu(MPI)p 408 1854 V 15 w(PROBE)f Fx(and)h Fu(MPI)p 693 1854 V 15 w(CANCEL)f Fx(are)h(primitiv)o(e.)75 1970 y Fs(1.14.6)49 b(Return)16 b(status)75 2047 y Fx(The)e(functions)g Fu(MPI)p 408 2047 V 15 w(GET)p 489 2047 V 15 w(SOURCE)p Fx(,)e Fu(MPI)p 726 2047 V 15 w(GET)p 807 2047 V 15 w(TAG)p Fx(,)g Fu(MPI)p 978 2047 V 15 w(GET)p 1059 2047 V 15 w(LEN)p Fx(,)h Fu(MPI)p 1231 2047 V 15 w(PROBE)p 1356 2047 V 15 w(LEN)p Fx(,)f(and)i Fu(MPI)p 1608 2047 V 15 w(IS)p 1667 2047 V 15 w(CANCELLED)75 2097 y Fx(are)g(primitiv)o(e.)158 2147 y(These)h(functions)f(are)g(simple)f (macros)g(that)h(access)h(records)h(in)d(a)h(structure.)75 2263 y Fs(1.14.7)49 b(send-receive)15 b(and)i(exchange)75 2339 y Fu(MPI)p 144 2339 V 15 w(SENDRECV\()j(send)p 465 2339 V 15 w(start,)g(send)p 720 2339 V 15 w(count,)h(send)p 976 2339 V 15 w(type,)f(dest,)h(recv)p 1340 2339 V 15 w(start,)g(recv)p 1596 2339 V 14 w(count,)g(recv)p 1851 2339 V 15 w(type,)75 2389 y(source,)f(tag,)h(comm,)g(status\))12 b Fx(is)75 2481 y Fu(MPI_ISEND\(handle[)o(0],)19 b(send_start,)g(send_count,)h(send_type,)f (dest,)i(tag,)g(comm\))75 2530 y(MPI_IRECV\(handle[)o(1],)e(recv_start,)g (recv_count,)h(recv_type,)f(source,)i(tag,)g(comm\))75 2580 y(MPI_WAITALL\(2,)e(handle,)h(status_array\))75 2630 y(status)h(=)g (status_array[1])p eop %%Page: 38 38 bop 75 -100 a Fq(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(38)158 45 y(The)14 b(non)o(blo)q(c)o(king)f(sends)i(and)f (receiv)o(es)h(can)f(b)q(e)h(replaced)g(b)o(y)f(their)g(primitiv)o(e)d (implemen)o(tation.)158 95 y(The)17 b(function)f Fu(MPI)p 480 95 14 2 v 15 w(EXCHANGE)e Fx(can)i(b)q(e)h(handled)f(in)g(a)g(similar)e (manner;)i(a)g(temp)q(orary)f(bu\013er)i(need)g(b)q(e)75 145 y(allo)q(cated)d(to)g(replicate)h(the)g(send)g(and)f(receiv)o(e)i(bu\013er.) 21 b(A)14 b(size)h(for)f(this)g(bu\013er)i(can)e(b)q(e)h(computed)f(using)g (the)75 195 y(function)g Fu(MPI)p 307 195 V 15 w(TYPE)p 410 195 V 15 w(EXTENT)p Fx(.)75 311 y Fs(1.14.8)49 b(Derived)16 b(datat)o(yp)q(es)75 387 y Fx(W)m(e)22 b(ha)o(v)o(e)g(outlined)f(in)h (Section)g(1.13.1)f(ho)o(w)g(eac)o(h)i(datat)o(yp)q(e)f(constructor)i(can)e (b)q(e)h(expressed)i(in)c(terms)75 437 y(of)f(the)h(next)h(one.)38 b(Th)o(us)21 b(all)f(datat)o(yp)q(e)g(constructors)j(can)e(b)q(e)g(expressed) i(in)e(terms)f(of)g(the)h(constructor)75 487 y Fu(MPI)p 144 487 V 15 w(TYPE)p 247 487 V 15 w(STRUC)p Fx(;)12 b(the)j(functions)f Fu(MPI)p 713 487 V 15 w(TYPE)p 816 487 V 15 w(FREE)p Fx(,)e Fu(MPI)p 1009 487 V 15 w(TYPE)p 1112 487 V 15 w(EXTENT)h Fx(and)g Fu(MPI)p 1418 487 V 16 w(ADDRESS)f Fx(are)i(primitiv)o(e.)75 603 y Fs(1.14.9)49 b(Summ)o(a)o(r)o(y)75 680 y Fx(In)13 b(order)h(to)f (implemen)o(t)e(the)j(MPI)f(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication)e (one)k(needs)g(16)f(functions;)g(ab)q(out)g(half)g(of)75 730 y(those)i(are)f(trivial)e(macros:)126 821 y(1.)20 b Fu(MPI)p 248 821 V 15 w(COMM)p 351 821 V 15 w(INIT)126 904 y Fx(2.)g Fu(MPI)p 248 904 V 15 w(START)126 987 y Fx(3.)g Fu(MPI)p 248 987 V 15 w(WAITALL)126 1070 y Fx(4.)g Fu(MPI)p 248 1070 V 15 w(TESTANY)126 1153 y Fx(5.)g Fu(MPI)p 248 1153 V 15 w(PROBE)126 1236 y Fx(6.)g Fu(MPI)p 248 1236 V 15 w(CANCEL)126 1319 y Fx(7.)g Fu(MPI)p 248 1319 V 15 w(GET)p 329 1319 V 15 w(SOURCE)126 1402 y Fx(8.)g Fu(MPI)p 248 1402 V 15 w(GET)p 329 1402 V 15 w(TAG)126 1485 y Fx(9.)g Fu(MPI)p 248 1485 V 15 w(GET)p 329 1485 V 15 w(LEN)105 1568 y Fx(10.)g Fu(MPI)p 248 1568 V 15 w(PROBE)p 373 1568 V 15 w(LEN)105 1651 y Fx(11.)g Fu(MPI)p 248 1651 V 15 w(IS)p 307 1651 V 15 w(CANCELLED)105 1734 y Fx(12.)g Fu(MPI)p 248 1734 V 15 w(TYPE)p 351 1734 V 15 w(STRUC)105 1817 y Fx(13.)g Fu(MPI)p 248 1817 V 15 w(TYPE)p 351 1817 V 15 w(EXTENT)105 1900 y Fx(14.)g Fu(MPI)p 248 1900 V 15 w(ADDRESS)105 1983 y Fx(15.)g Fu(MPI)p 248 1983 V 15 w(TYPE)p 351 1983 V 15 w(FREE)105 2066 y Fx(16.)g Fu(MPI)p 248 2066 V 15 w(COMM)p 351 2066 V 15 w(FREE)158 2158 y Fx(On)13 b(the)h(other)g(hand,)e(w)o(e)i(exp)q(ect)g (most)e(implemen)o(tations)e(to)j(use)h(more)e(primitiv)o(es,)f(in)i(order)g (to)g(reduce)75 2207 y(comm)o(unication)d(o)o(v)o(erheads,)15 b(reduce)g(bu\013ering,)f(and)g(impro)o(v)o(e)e(error)j(rep)q(orting.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Fri Jul 23 18:44:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23604; Fri, 23 Jul 93 18:44:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16053; Fri, 23 Jul 93 18:44:17 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Jul 1993 18:44:14 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15988; Fri, 23 Jul 93 18:44:06 -0400 Message-Id: <9307232244.AA15988@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 2751; Fri, 23 Jul 93 18:44:02 EDT Date: Fri, 23 Jul 93 18:44:02 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: MPI-TERM.DVI.* %%Pages: 5 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 16 85 df<1230127812F81278127005057D840C> 46 D<13181370EA03F0120FEA1C701200A213E0A6EA01C0A6EA0380A61207EAFFF8A20D1C7C9B 15>49 D<14301470A214F0801301A2EB0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFE A23801801EA23803001F801206120F397FC0FFF012FF1C1D7F9C1F>65 D<903807F030EB1FFC90 387C0E603901F003E0EA03C038078001EA0F00000E1300001E14C05AA25A1500A25AA5EC0180EC 03001270127814066C5B001C131C000F5B3807C0E06CB45AD8007EC7FC1C1E7C9C1E>67 D<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A1538A3153C1538485A1578A31570 15F0484813E0140115C0EC03801407EC0E00380F007CB512F014C01E1C7E9B20>I<000FB512E0 A23800F003EC01C01400A33801E060A31500EBE1E013FF485B13C113C0A215C0ECC18038078001 A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000FB512C0A23800F007EC03801401A3 EA01E01461A2EC600014E013E148B45AA213C113C0A3380781800180C7FCA5120FEAFFFC5B1A1C 7E9B1B>I<390FFF9FFE143F3900F003C0A53901E00780A590B5FC481400EBC00FA53807801EA6 000F133E39FFF3FFE015C01F1C7E9B1F>72 D<380FFF80A23800F000A5485AA6485AA6485AA612 0FEAFFF8A2111C7F9B0F>I77 D<390FF80FFEA23900FC01E0EC00C0A213DEA239018F 0180A214811387A2EB83C1390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38 FFE00C13C01F1C7E9B1F>II<380FFFFC14FF3900F00F80140315C01401A2 3801E003A315801407EC0F003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B 1C>I<380FFFF814FE3900F00F80140315C01401A23801E003A3EC0780EC0F00143E3803FFF85C EBC07880A2143E3807803CA51518000F1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 DI<001FB512F05A383C07810038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6 137C381FFFE0A21C1C7C9B1E>I E /Fb 20 122 df97 D<123F5A120F120EA45AA4EA39C0EA3FE0EA3E70EA3830EA78381270A3EAE070A3136013E013C0 12E1EA6380EA7F00123C0D1D7B9C13>IIIII104 D<13C0EA01E013C01380C7FCA6120E123FEA33801263EAC700A21207120EA3 5A1340EA38C0A3EA3980EA3F00121E0B1C7D9B0D>I<130E131EA2130C1300A613F0EA01F8EA03 38EA06181338120C1200A21370A413E0A4EA01C0A4EA03801263EAF700A212FE12780F24819B0D >I108 D<391E1F07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380E0A3EC81C2 391C0701C6A2EC038CA239380E01F839180600F01F127D9122>IIIIIII<13C01201A3EA0380A4EAFFE0A2EA0700A212 0EA45AA31320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I<380F0180EA1F8312331263EB87 0012C3EA0707A2EA0E0EA31420380C1C60121C120C380E3CC0380FEF803803C70013127D9116> I<380F0180EA1F8312331263EB870012C3EA0707A2EA0E0EA4EA0C1C121C120CEA0E3CEA0FF812 03EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7FC111A7D9114>121 D E /Fc 31 122 df<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA212 01B3A7387FFFE0A213207C9F1C>49 DI I<14E013011303A21307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238 127012E0B512FEA2380007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014 005B13F8EA1BC00018C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A2123812 7C12FCA214F012F8386007E0003013C0381C1F80380FFF00EA03F815207D9F1C>I<1470A214F8 A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F140701E07F90 B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A>65 D67 D78 DI<3801FC043807FF8C381F03FC383C00 7C007C133C0078131CA200F8130CA27E1400B4FC13E06CB4FC14C06C13F06C13F86C13FC000313 FEEA003FEB03FFEB007F143FA200C0131FA36C131EA26C133C12FCB413F838C7FFE00080138018 227DA11F>83 D97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3 127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>II104 D<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I<131C133E 137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78FCEA7FF0EA1FC0102E83 A311>I108 D<3AFF87F00FE090399FFC3FF83A1FB8 7E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I<38FF87 E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FFC0380F 83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E03807FFC0 3800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2EC 3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>IIII<13C0A41201A2120312 07120F121FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA238 1F807EAF14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE0 0E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA213 0CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>121 D E /Fd 19 119 df63 D 97 DII101 D<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80AEEA7FF8A2121D80 9C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E1FEA1FFCEA33F000 30C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F0383E03E0381FFFC038 03FE00151B7F9118>I<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E> 105 D107 DI<39FF1FC0FE90 387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0 EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>I I<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7 FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEAFFFCA2EA1F00 A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03 F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6 A213FE6C5AA21378A2133016127F9119>I E /Fe 17 118 df<1306130E131E131C133C133813 781370A213F013E0120113C012031380120713005A120E121E121CA2123C12381278127012F05A A20F1D7E9914>47 D65 D71 D73 D76 D<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A21303A538FE0FE0A31317 809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136EA4EAFE3EA311177F96 14>I80 D83 DI<38FF8FF8A3381C01C0AEEA1E0300 0E1380EA0F073807FF006C5AEA00F81517819614>I<38FE3F80A3383C1E00EA1C1CEA1E3CEA0E 38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>89 D<1203EA0780A2EA0300C7FCA4EA FF80A31203AAEAFFFEA30F187D9714>105 D110 DI<1207A5EAFFFCA3EA0700A6130EA3139E13FCEA03F8 EA00E00F157F9414>116 DI E /Ff 55 122 df11 D<137CEA01FEEA0387485A120E130690C7FCA4B5FCA2EA0E07AC 387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07AC387F9FE0A213 1A809915>I 34 D<126012F012F812781218A31230127012E01240050B7D990B>39 D<13C0EA0180EA030012 06120E120C5A1238A212301270A21260A212E0AA1260A21270A212301238A212187E120E12067E EA0180EA00C00A267E9B0F>I<12C012607E7E121C120C7E1207A27E1380A21201A213C0AA1380 A21203A213005AA212065A121C12185A5A5A0A267E9B0F>I<126012F0A212701230A31260A212 C01240040B7D830B>44 D<126012F0A2126004047D830B>46 D48 D<1260EA7FFEA213FCEA600CEAC0181330A2EA006013C0A21201A2EA0380A41207A8EA03000F1A 7E9813>55 D57 D<126012F0A212601200A8126012F0A212701230A31260A212C0124004177D8F0B>59 D63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47E A2130000061378A3487F121E39FF81FFC0A21A1B7F9A1D>65 D67 D70 DI<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39FFF3FFC0A21A1A7F991D>II76 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3EB7818A3EB3C 30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>II<137F3801FFC03807C1F0380F0078001E7F001C 131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C131C001E133C6C5B3807C1 F03801FFC06C6CC7FC191C7E9A1E>II83 D<007FB5FCA238781E0F00601303A200E0148000C01301A300 0090C7FCAF3803FFF0A2191A7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A38 01E0E038007FC0011FC7FC1A1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC03 00A214FC5DD807801306EB819EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F6 13B001FE13F0EBFC03017C5BA2EB7801A201385BEB3000241B7F9927>87 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A2381F0F00EA 1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A61270131EEA 3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA03BCEA073CEA0E181300A5EAFF C0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870EA3FE0EA 7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F9013>I< 12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA21218C7 FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A21200B012 6012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B121DEA 1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80A2091A 80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D107F8F20> IIIIIII< 120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>II<38FF3F80A2381C0E00130CA26C5AA21338EA0730 A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E5B13 631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA03B8 13B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80A238 1C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE12781117 7F8F14>I E /Fg 15 118 df<127812FCA412781200A5127812FCA4127806117D900C>58 D68 D73 D97 D99 D101 D<121E123FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>105 D108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7 FE7FE0A223117F9026>IIII115 D<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03E00D187F9711>I<38FF1FE0A2EA 1F03AB1307130F380FFFFCEA03F316117F9019>I E /Fh 26 118 df<13E0487EA213B0A2EA03 B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816 >65 D<3801F180EA07FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C 07381F0F00EA0FFE6C5AEA01F011197E9816>67 DI<387FFFC0B5 FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816> I71 D73 D<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303 A538FE0FE0A313197F9816>77 D<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA2 13E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>III82 DI<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I<387F07F038 FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519809816>I<38 FC07E0EAFE0FEAFC07387001C0A300301380EA3803A313E3EA39F3A213B300191300A61313EA1B 1BEA0F1EA2EA0E0E13197F9816>87 D<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0EA26C5AA3EA 03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>89 D97 D101 D<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3 EA3C1E6C5AEA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3 EAF001387C07C0383FFF80380FFE00EA03F8131C7F9116>103 D105 D 108 D 110 DII<12035A A4EA7FFFB5FCA20007C7FCA75BEB0380A2130713873803FF005BEA00F811177F9616>116 D<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C13F83803F3F01512809116>I E /Fi 57 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B5 12C0A2EA0E01AE387F87F8A2151D809C17>II<90383F03F09038FFCFF83903C0FC1C390701F03CA2390E00 E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>I34 D<13C0EA0180EA03001206 120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C120C 120E12067EEA0180EA00C00A2A7D9E10>40 D<12C012607E7E121C120C120E120612077EA21380 A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<1270 12F012F8A212781218A31230A2127012601240050D7D840C>44 DI<12 7012F8A3127005057D840C>I<12035A123FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>49 DII<131CA2133C137CA213DC1201139C 1203EA071C1206120E120C121812381230126012E0B512C0A238001C00A63801FFC0A2121C7F9B 15>I<127012F8A312701200A8127012F8A3127005127D910C>58 D<127012F8A312701200A812 7012F012F8A212781218A31230A2127012601240051A7D910C>I<1306130FA3497EA4EB33C0A3 EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C1D7F 9C1F>65 D<90381F8080EBFFE13803F0333807801B380F000F001E1307001C1303123C5A140112 7012F091C7FCA70070EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F 80191E7E9C1E>67 DI70 D<39FFF3FFC0A2390F00 3C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>72 DI77 D79 DI<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFE EA07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E 9C17>83 D<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803 FFFCA21B1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0EB 7FC0011FC7FC1A1D7E9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA201801470000790 38678060A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13 F6017E137EA3013C133CA3011C133801181318281D7F9B2B>87 D<397FF0FFC0A23907C03E0000 031338EBE03000011370EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371 EB61F0EBC0F800011378EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I92 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E013 0014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E 00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0 A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>II<12FC A2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9F E0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83 B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFB EA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA 1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F 9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA31312 7F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713CCA2136038 03E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00 F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2 381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC12 7E123C131A7F9116>I123 D E /Fj 25 122 df<121C127FEAFF80A5EA 7F00121C09097B8813>46 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622 >49 DII<14075C5C5C5C5CA25B5B497E130F130E131C 1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D27 7EA622>I68 DI72 D78 D80 D<007FB712C0A39039807FC03FD87C0014 0700781503A20070150100F016E0A2481500A5C71500B3A490B612E0A32B287EA730>84 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA 1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 D99 DII<9038FF81F0 0003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B 380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F39 7E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E 287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFE A30F2B7DAA14>105 D108 D<38FFC07F9038C1FFC0 9038C787E0390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>110 DI<38FF E1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF001 4A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3 F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C 13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FFC000C11300161B 7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438A73803F8700001 13F03800FFE0EB1F8015267FA51B>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC01 00015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2 495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>121 D E /Fk 13 117 df<92380FFF804AB512F0020F14FC023F809139FFFC01FF01039039E0003F80 4990C7B47E4948497F49485B5C495AA24948497F705BA3705B705B71C7FC94C8FCA693383FFFF0 B9FCA526007FF0C7FC177FB3B1003FB5D8E03FB512E0A53B487EC742>12 D68 D<003FBBFCA5DA00039038F8003FD87FF80507138001E0170149834918 7F90C7173FA2007E191FA21A0F127CA400FC1AC0481907A5C894C7FCB3B3A40103B812F8A54A46 7CC553>84 D<903807FFFE017FEBFFE048B612F84815FE489039001FFF8003077F48D980017F6F 7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F013FEBF03F3901FFFE004813 F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB01EF003F140301FF90380F CFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF0010107903980007FF8362E7DAD 3A>97 D100 DI<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0 B5FCA512037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC 033F01FE011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026 C7C0019026E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5D B3A8B6D8C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13 FE92B6FC9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25C A25CB3A8B6D8C07FEBFFE0A53B2E7CAD42>II<9039FF803FC0B5EBFFF0028313FC02877F91388FE7FFEC9F070003D99E0F13806C13BCA214 F8A214F06F13004A6C5A6F5A92C8FCA25CB3A6B612E0A5292E7CAD31>114 D<90390FFF81E090B512F7000314FF5A380FFC01391FE0003FD83F80130F007F140790C7FC1503 5AA27F6D90C7FC13F013FF14F86CEBFFC015F86C14FE6C806C15806C15C06C15E0C615F0013F14 F81307EB001F020013FC153F0078140F00F81407A26C1403A27E16F86C14076D14F06D130F01F0 EB1FE001FEEBFFC090B6128048ECFE00D8F83F13F8D8F00313C0262E7CAD2F>II E /Fl 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879 B732>49 D67 D97 D<903801FFC0011F13F801 7F13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED 03FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB 01F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0 5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13 FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07 FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F 13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80 91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D A529>114 D116 D E /Fm 25 122 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D49 DII57 D<3803FFF0A238000F00B3A6127012F8A3EAF01EEA601CEA3878EA1FF0EA07 C014237EA119>74 D77 D79 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3FF0EA 1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF80138EF 038038C7FF00EA81FC14247DA21B>83 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C141C141EA7141C143C 000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II<14E013 0FA213011300AAEA03F0EA07FEEA1F07EA3C01EA38001278127012F0A712701278EA3801EA3C03 381E0EF0380FFCFEEA03F017237EA21B>II<121C 121E123E121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>105 D<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>108 D<390E1FC07F3AFE7FE1FF809039C0 F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038FE 7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 DI<1206A5120EA3121E123E EAFFF8A2EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA 1E00000E1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F800 0E1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<38FF C3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA31338A213 30A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>121 D E /Fn 22 118 df<14FF01077F90381F01C090383C00604913F0EBF001EA01E0A248486C5A92C7 FCAAB612F0A23803C0011400B3A7486C487E3A7FFE1FFF80A221327FB124>12 D45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F48130700381303007813011270 140012F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013 C0EC1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039 E780070038C1F01E38C07FFC38800FF01D337CB125>83 D<007FB712E0A29039000F800F007C15 0300781500127000601660A200E01670A2481630A6C71500B3AC4A7E010FB57EA22C317EB030> I97 D99 DIII<15F090387F03F83901FFCF1C3803C1 FC390780F818390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380F C1E0380DFFC0D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E000 38EB01F048EB00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC 38003FE01E2F7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6 EA0FC0EAFFF8A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F 9803E3007C2707B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2 341F7E9E38>109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA3 5BB3486C487E3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0F B0EA07F0EBE038EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03 F0EA38004813705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB00 7800C0133CA2141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13 C0A51201A31203A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB 1F00142C7FAB19>II E /Fo 5 85 df<1630167016F0A21501A2 1503A21507150FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430 A202607FA2ECC000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF013 01D8FFFE90383FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C17 7E4A143E83147E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F 18004A5CA2011F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE14 7E16F849495AED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F 8000173E92C7121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F0 5B15030107130F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291 C9FCA25BA2137EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F80 1FC0EE07E092380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E0 1303EE0F804AEB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA201 1F80A25CA2013F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F0 00FC70EE7FE0C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E004913 70121E001C130F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25C A2147EA214FEA25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007F B512C0A22D3174B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 947 a Fo(D)26 b(R)g(A)f(F)h(T)225 1038 y Fn(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)657 1130 y(T)-6 b(erms)21 b(and)g(De\014nitions)869 1312 y Fm(Marc)16 b(Snir)505 1371 y(to)h(b)q(e)f(assumed)g(and)g(subsumed)g(b)o(y)g(Stev)o(e)f (Otto)833 1489 y(July)h(23,)g(1993)p eop %%Page: 1 2 bop 75 356 a Fl(Chapter)32 b(1)75 564 y Fk(T)-10 b(erms)39 b(and)h(De\014nitions)75 805 y Fj(1.1)70 b(Notation)75 896 y Fi(Discussions)14 b(that)g(are)h(not)e(part)i(of)e(the)h(standard)h(are)f (indicated)g(b)o(y)f(small)f(t)o(yp)q(e.)158 945 y(The)k(MPI)g(pro)q(cedures) i(are)e(sp)q(eci\014ed)h(using)e(a)g(language)g(indep)q(enden)o(t)i (notation.)22 b(The)16 b(parameters)g(of)75 995 y(pro)q(cedure)j(calls)d(are) h(mark)o(ed)f(as)h Fh(in)p Fi(,)g Fh(out)f Fi(or)h Fh(inout)p Fi(.)25 b(The)18 b(call)e(uses)i(but)f(do)q(es)g(not)g(up)q(date)h(a)e (parameter)75 1045 y(mark)o(ed)e Fh(in)p Fi(,)h(ma)o(y)f(up)q(date)i(a)f (parameter)g(mark)o(ed)g Fh(out)f Fi(and)i(b)q(oh)o(t)f(uses)i(and)e(up)q (dates)i(a)e(parameter)g(mark)o(ed)75 1095 y Fh(inout)p Fi(.)i(If)c(a)h (parameter)f(is)h(a)f(handle)h(to)g(an)f(opaque)h(ob)r(ject,)g(and)g(the)g (ob)r(ject)h(is)f(up)q(dated)g(b)o(y)g(the)g(pro)q(cedure)75 1145 y(call,)f(then)h(the)h(parameter)e(is)h(mark)o(ed)f Fh(out)p Fi(.)158 1195 y(The)f(de\014nition)f(of)g(MPI)h(tries)g(to)g(a)o(v)o(oid,)e (to)h(the)h(largest)g(p)q(ossible)g(exten)o(t,)g(the)h(use)f(of)f Fh(inout)f Fi(parameters,)75 1244 y(b)q(ecause)16 b(suc)o(h)e(use)h(is)f (error-prone,)g(esp)q(ecially)g(for)g(scalar)g(parameters.)158 1294 y(A)j(frequen)o(t)g(situation)e(is)i(where)g(a)f(parameter)g(is)h(used)g (as)f Fh(in)g Fi(b)o(y)h(some)e(pro)q(cesses)k(and)d Fh(out)g Fi(b)o(y)g(other)75 1344 y(pro)q(cesses.)28 b(Suc)o(h)17 b(parameter)f(is,)h (syn)o(tactically)m(,)e(an)h Fh(input)g Fi(parameter,)g(although,)g(seman)o (tically)m(,)e(it)i(is)g(not)75 1394 y(used)f(in)e(one)h(call)f(b)q(oth)h (for)g(input)g(and)f(for)h(output.)158 1522 y Fg(Discussion:)158 1568 y Ff(W)m(e)f(migh)o(t)h(w)o(an)o(t)f(to)g(distinguish)j(this)e(use;)f (e.g.)k Fe(in/out)10 b Ff(vs)k Fe(inout)p Ff(.)158 1700 y Fi(Another)c (frequen)o(t)h(situation)d(is)i(where)g(a)g(parameter)f(v)n(alue)g(is)g (needed)i(only)d(b)o(y)i(a)f(subset)i(of)e(the)h(pro)q(cesses.)75 1750 y(When)i(a)g(parameter)g(is)f(not)h(signi\014can)o(t)g(at)g(a)f(pro)q (cess)j(then)f(an)e(arbitrary)h(v)n(alue)g(can)g(b)q(e)g(passed)h(as)g (argumen)o(t.)158 1800 y(An)h(comm)o(unication)d(op)q(eration)j(is)g(termed)g (to)g(b)q(e)h Fd(non)o(blo)q(c)o(kin)o(g)c Fi(if)i(it)h(ma)o(y)e(return)k(b)q (efore)f(the)f(related)75 1850 y(comm)o(unication)g(op)q(eration)j (completes;)i Fd(blo)q(c)o(king)p Fi(,)c(otherwise.)30 b(An)17 b(op)q(eration)g(is)h Fd(lo)q(cal)e Fi(if)h(its)g(execution)75 1900 y(do)q(es)f(not)e(en)o(tail)g(comm)o(unication,)e(and)i(do)q(es)i(not)f (dep)q(end)h(on)e(the)i(state)f(of)f(other)i(pro)q(cesses;)h Fd(global)p Fi(,)c(if)h(its)75 1949 y(completion)c(ma)o(y)f(dep)q(end)k(on)e (comm)o(unication)d(o)q(ccuring;)13 b Fd(collectiv)o(e)8 b Fi(if)j(sev)o(eral)h(pro)q(cesses)i(need)f(participate)75 1999 y(in)g(its)h(execution.)75 2136 y Fj(1.2)70 b(Data)23 b(T)n(yp)r(es)75 2236 y Fc(1.2.1)55 b(Opaque)19 b(ob)s(jects)75 2312 y Fi(MPI)12 b(pro)q(cedures)i(use)e(at)g(v)n(arious)f(places)h Fb(op)n(aque)i(obje)n(cts) p Fi(.)j(Opaque)12 b(ob)r(jects)h(hide)e(from)f(the)i(user)h(the)f(in)o (ternal)75 2362 y(represen)o(tation)17 b(used)g(for)e(v)n(arious)g(MPI)h(ob)r (jects,)h(th)o(us)g(allo)o(wing)c(to)j(ha)o(v)o(e)f(similar)f(calls)h(in)g(C) h(and)g(F)m(ortran,)75 2412 y(allo)o(wing)g(to)i(o)o(v)o(ercome)f(problems)g (with)h(the)h(t)o(yping)e(rules)i(in)f(these)i(languages,)e(and)g(allo)o (wing)e(for)h(future)75 2462 y(extension)h(of)f(their)h(functionalit)o(y)m(.) 26 b(The)18 b(mec)o(hanism)d(for)i(opaque)g(ob)r(jects)i(used)f(here)h(lo)q (osely)e(follo)o(ws)e(the)75 2512 y(POSIX)g(F)m(ortran)e(binding)g(standard.) 158 2561 y(An)i(opaque)g(ob)r(ject)h(can)g(b)q(e)g(allo)q(cated)e(on)h(user)i (storage)e(or)g(on)g(system)g(\(i.e.)22 b(MPI)15 b(managed\))f(storage.)75 2611 y(W)m(e)19 b(call)g(the)g(former)g(a)g Fb(user)h(obje)n(ct)f Fi(and)g(the)h(later)f(a)g Fb(system)h(obje)n(ct)p Fi(.)34 b(User)21 b(ob)r(jects)f(are)g(allo)q(cated)f(and)75 2661 y(deallo)q(cated)d (explicitly)f(b)o(y)g(the)i(user)f(as)g(a)g(regular)g(\(structured\))i(v)n (ariable.)k(The)17 b(t)o(yp)q(e)f(de\014nitions)g(for)f(suc)o(h)965 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fa(CHAPTER)14 b(1.)32 b(TERMS)13 b(AND)h(DEFINITIONS)941 b Fi(2)75 45 y(ob)r(jects)13 b(are)g(pro)o(vided)e(b)o(y)h(MPI.)g(The)g(ob)r (jects)i(can)e(b)q(e)g(accessed)j(and)d(mo)q(di\014ed)e(only)h(via)h(MPI)g (functions)g(that)75 95 y(tak)o(e)i(suc)o(h)h(ob)r(jects)g(as)f(parameters.) 158 145 y(System)f(ob)r(jects)j(are)e(accessed)i(only)e(via)f Fb(hand)r(les)p Fi(.)19 b(A)14 b(di\013eren)o(t)h(t)o(yp)q(e)f(of)g(handle)f (is)h(de\014ned)h(b)o(y)f(MPI)g(for)75 195 y(eac)o(h)g(t)o(yp)q(e)f(of)g (opaque)g(system)g(ob)r(ject)h(\(in)f(C;)g(in)f(F)m(ortran)h(all)f(handles)i (ha)o(v)o(e)f(t)o(yp)q(e)h Fh(INTEGER)p Fi(\).)d(The)j(allo)q(cation)75 244 y(and)i(deallo)q(cation)f(of)h(suc)o(h)h(ob)r(ject,)g(and)f(an)o(y)f (access)j(or)e(up)q(date)h(to)f(it,)g(are)h(done)f(only)f(via)h(MPI)g(calls)g (that)75 294 y(accept)f(as)f(parameters)g(handles)g(to)g(suc)o(h)h(ob)r (jects.)158 344 y(An)h(op)q(que)g(ob)r(ject,)h(and)e(the)i(handle)e(to)h(suc) o(h)h(ob)r(ject)f(are)g(signi\014can)o(t)g(only)f(at)g(the)i(pro)q(cess)g (where)g(the)75 394 y(ob)r(ject)e(w)o(as)e(created,)i(and)f(cannot)g(b)q(e)h (transferred)g(to)f(another)g(pro)q(cess)i(\(ob)r(jects)f(are)g(lo)q(cal\).) 158 444 y(Opaque)h(system)g(ob)r(ject)g(are)h(allo)q(cated)e(and)h(deallo)q (cated)f(created)j(b)o(y)d(calls)h(that)f(are)i(sp)q(eci\014c)g(to)e(eac)o(h) 75 493 y(ob)r(ject)h(t)o(yp)q(e.)24 b(These)17 b(are)f(listed)f(in)h(the)g (sections)g(where)h(the)f(ob)r(jects)h(are)f(describ)q(ed.)25 b(A)16 b(call)f(to)g(deallo)q(cate)75 543 y(an)e(opaque)g(system)g(ob)r(ject) i(do)q(es)f(not)f(necessarily)h(deallo)q(cate)g(the)f(ob)r(ject)i(imm)o (ediatly)l(,)10 b(but)k(only)e(mark)g(it)h(as)75 593 y(candidate)j(for)f (deallo)q(cation.)22 b(The)16 b(ob)r(ject)h(will)d(b)q(e)i(deallo)q(cated)g (as)f(as)h(so)q(on)g(as)f(there)i(there)g(is)e(no)h(p)q(ending)75 643 y(op)q(eration)d(that)h(is)g(using)f(this)h(ob)r(ject;)g(the)g(handle)f (then)i(b)q(ecomes)e(n)o(ull.)k(F)m(or)c(example,)f(if)h(a)g(comm)o (unication)75 693 y(ob)r(ject)f(is)g(freed,)h(after)e(the)i(comm)o(unicatio)o (n)c(started)k(but)f(b)q(efore)g(it)g(completed,)f(then)h(the)g(p)q(ending)g (op)q(eration)75 742 y(is)18 b(not)h(a\013ected;)i(one)e(can)g(use)g(the)g (handle)g(in)f(calls)g(to)g Fh(MPI)p 1098 742 14 2 v 15 w(WAIT)g Fi(or)g Fh(MPI)p 1340 742 V 15 w(STATUS)f Fi({)h(one,)i(in)e(fact,)h(m)o(ust) 75 792 y(complete)c(the)g(comm)o(unication)d(with)j(a)g(successful)i(call)e (to)g Fh(MPI)p 1136 792 V 15 w(WAIT)f Fi(or)h Fh(MPI)p 1371 792 V 15 w(STATUS)p Fi(.)f(Ho)o(w)o(ev)o(er,)h(once)h(the)75 842 y(op)q(eration)e(completes,)f(it)g(is)h(erroneous)h(to)f(reuse)i(the)e (freed)h(handle)f(for)f(a)h(new)g(comm)o(unication.)158 975 y Fg(Implemen)o(tation)k(note:)54 b Ff(Some)18 b(ob)r(jects,)h(suc)o(h)f(as)g (bu\013er)h(descriptor)g(ob)r(jects,)g(can)f(b)q(e)g(used)h(b)o(y)f(sev)o (eral)75 1024 y(distinct)h(concurren)o(t)f(comm)o(unication)h(op)q(erations.) 31 b(Th)o(us,)18 b(to)f(\014nd)g(out)g(whether)h(suc)o(h)f(ob)r(ject)g(can)h (b)q(e)f(freed,)g(one)75 1074 y(needs)d(a)f(reference)g(coun)o(t)h(of)e(p)q (ending)k(op)q(erations.)158 1285 y Fg(Discussion:)158 1331 y Ff(Need)g(a)g(test)g(to)g(see)g(of)g(a)g(handle)i(is)f(n)o(ull.)28 b(In)16 b(C,)f(if)i(handles)h(are)e(p)q(oin)o(ter,)i(a)e(comparision)i(th)f (NULL)e(can)i(b)q(e)75 1377 y(used.)g(In)12 b(F)m(ortran,)g(ma)o(y)g(either)g (ha)o(v)o(e)g(a)g(function)h Fe(MPI)p 891 1377 12 2 v 13 w(IS)p 944 1377 V 13 w(NULL)d Ff(\(and)i(mak)o(e)g(it)g(a)o(v)n(ailable)j(in)d(C,)f (to)q(o,)h(for)f(compatibili)q(t)o(y)75 1422 y(reasons\),)j(or)f(ha)o(v)o(e)g (a)g(NULL)f(v)n(alue,)i(that)g(can)f(b)q(e)g(used)h(in)g(comparisons,)g(in)g (amnner)g(similar)h(to)e(a)g(C)g(n)o(umll)h(p)q(oin)o(ter.)158 1468 y(Oppinions?)158 1600 y Fi(MPI)k(ma)o(y)e(pro)o(vide)h(prede\014ned)j (opaque)d(ob)r(jects)i(and)f(prede\014ned,)i(static)e(handles)g(to)f(these)i (ob)r(jects.)75 1650 y(Suc)o(h)14 b(ob)r(jects)h(ma)o(y)d(not)i(b)q(e)h (destro)o(y)o(ed.)75 1766 y Fc(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 1843 y Fi(An)d(MPI)g(call)e(ma)o(y)g(need)j(a)e(parameter)g(that)h(is)f(an)g Fb(arr)n(ay)h(of)h(hand)r(les)p Fi(.)23 b(The)17 b(arra)o(y-of-handles)d(is)i (a)f(regular)75 1893 y(arra)o(y)d(with)h(en)o(tries)g(that)g(are)g(handles)g (to)f(ob)r(jects)i(of)e(the)h(same)e(t)o(yp)q(e)j(in)e(consecutiv)o(e)i (initial)c(lo)q(cations)i(in)g(the)75 1943 y(arra)o(y)m(.)17 b(Whenev)o(er)c(suc)o(h)h(arra)o(y)e(is)g(used,)i(an)e(additional)f Fh(len)h Fi(parameter)g(is)g(required)i(to)e(indicate)h(the)g(n)o(um)o(b)q (er)75 1992 y(of)g(v)n(alid)g(en)o(tries)i(\(unless)f(this)g(n)o(um)o(b)q(er) f(can)i(b)q(e)f(deriv)o(ed)h(otherwise\).)75 2109 y Fc(1.2.3)55 b(State)75 2185 y Fi(MPI)18 b(pro)q(cedures)i(use)e(at)g(v)n(arious)f(places) h(argumen)o(ts)f(with)g Fb(state)g Fi(t)o(yp)q(es.)30 b(The)18 b(v)n(alues)g(of)f(suc)o(h)h(data)f(t)o(yp)q(e)75 2235 y(are)e(all)f(iden)o (ti\014ed)h(b)o(y)f(names,)g(and)h(no)f(op)q(eration)h(is)f(de\014ned)i(on)f (them.)20 b(F)m(or)14 b(example,)f(the)j Fh(MPI)p 1673 2235 14 2 v 15 w(COMM)p 1776 2235 V 15 w(INIT)75 2285 y Fi(routine)e(has)g(a)g (state)g(t)o(yp)q(e)h(parameter)e(with)h(v)n(alues)g Fh(MPI)p 1008 2285 V 15 w(STANDARD)p Fi(,)d Fh(MPI)p 1288 2285 V 15 w(READY)p Fi(,)i(etc.)75 2401 y Fc(1.2.4)55 b(Named)18 b(constan)n(ts)75 2478 y Fi(MPI)e(pro)q(cedures)j(sometimes)14 b(assign)i(a)g(sp)q(ecial)g (meaning)e(to)i(a)g(sp)q(ecial)g(v)n(alue)g(of)f(a)h(basic)g(t)o(yp)q(e)h (parameter;)75 2528 y(e.g.)i Fh(tag)14 b Fi(is)g(an)g(in)o(teger)g(v)n(alued) g(parameter)g(of)g(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n)f(op)q (erations,)i(with)g(a)g(sp)q(ecial)75 2577 y(wildcard)e(v)n(alue,)g Fh(MPI)p 429 2577 V 16 w(ANY)p 511 2577 V 15 w(TAG)p Fi(.)f(Suc)o(h)j (parameters)f(will)e(ha)o(v)o(e)i(a)f(range)h(of)g(regular)g(v)n(alues,)f (whic)o(h)h(is)f(a)h(prop)q(er)75 2627 y(subrange)f(of)f(the)h(range)g(of)e (v)n(alues)i(of)e(the)i(corresp)q(onding)h(basic)e(t)o(yp)q(e;)i(sp)q(ecial)e (v)n(alues)h(\(suc)o(h)g(as)f Fh(MPI)p 1700 2627 V 15 w(ANY)p 1781 2627 V 15 w(TAG)p Fi(\))p eop %%Page: 3 4 bop 75 -100 a Fa(CHAPTER)14 b(1.)32 b(TERMS)13 b(AND)h(DEFINITIONS)941 b Fi(3)75 45 y(will)14 b(b)q(e)j(outside)f(the)g(regular)g(range.)24 b(The)16 b(range)g(of)g(regular)f(v)n(alues)h(can)g(b)q(e)g(queried,)h(and)e (sometimes)g(set,)75 95 y(using)f(en)o(vironmen)o(t)e(inquiry)i(or)f(en)o (vironmen)o(t)g(setting)h(functions)g(\(Chapter)h Fd(??)p Fi(\).)158 223 y Fg(Discussion:)158 269 y Ff(Implemen)o(ters)i(should)g(detect,)f (whenev)o(er)g(p)q(ossible,)i(illegal)g(uses)e(of)f(\\sp)q(ecial)j(v)n (alues".)25 b(Th)o(us,)16 b(the)g(use)g(of)f(the)75 315 y Fe(MPI)p 137 315 12 2 v 13 w(ANY)p 210 315 V 13 w(TAG)d Ff(v)n(alue)i(to)f(tag)g(a)g (message)g(sen)o(t)h(will)g(b)q(e)g(\015agged)g(as)f(an)g(error.)75 513 y Fc(1.2.5)55 b(Choice)75 590 y Fi(MPI)15 b(functions)h(sometimes)d(use)j (parameters)f(with)g(a)f Fb(choic)n(e)i Fi(\(or)f(union\))g(data)f(t)o(yp)q (e.)23 b(I.e.,)14 b(distinct)h(calls)g(to)75 640 y(the)i(same)f(routine)h(ma) o(y)e(pass)i(b)o(y)f(reference)j(actual)d(parameters)h(of)f(di\013eren)o(t)i (t)o(yp)q(es.)27 b(The)17 b(mec)o(hanism)d(for)75 690 y(pro)o(viding)f(suc)o (h)h(parameters)g(will)f(di\013er)h(from)e(language)h(to)h(language.)158 818 y Fg(Discussion:)158 864 y Ff(The)i(F)m(ortran)g(77)g(standard)h(sp)q (eci\014es)h(that)e(the)g(t)o(yp)q(e)g(of)g(actual)h(argumen)o(ts)g(need)f (to)g(agree)g(with)h(the)f(t)o(yp)q(e)g(of)75 909 y(dumm)o(y)g(argumen)o(ts;) i(no)f(construct)f(equiv)n(alen)o(t)i(to)e(C)g(p)q(oin)o(ters)h(is)f(a)o(v)n (ailable.)29 b(Th)o(us,)16 b(it)g(w)o(ould)h(seem)f(that)g(there)g(is)75 955 y(no)f(standard)h(conforming)g(mec)o(hanism)g(to)e(supp)q(ort)i(c)o (hoice)g(parameters.)22 b(Ho)o(w)o(ev)o(er,)14 b(most)h(F)m(ortran)g (compiler)h(either)75 1001 y(don't)g(c)o(hec)o(k)g(t)o(yp)q(e)g(consistency)i (of)e(calls)h(to)f(external)h(routines,)g(or)f(supp)q(ort)h(a)f(sp)q(ecial)h (mec)o(hanism)h(to)d(link)j(foreign)75 1046 y(\(e.g.,)c(C\))g(routines.)23 b(W)m(e)15 b(accept)g(this)h(nonconformit)o(y)g(with)f(F)m(ortran)g(77)g (standard.)23 b(I.e.,)14 b(w)o(e)g(accept)h(that)g(the)g(same)75 1092 y(routine)f(ma)o(y)g(b)q(e)f(passed)h(an)f(actual)h(parameter)g(of)f(a)g (di\013eren)o(t)h(t)o(yp)q(e)g(at)f(distinct)i(calls.)158 1138 y(Generic)h(routines)g(can)f(b)q(e)g(used)g(in)g(F)m(ortran)g(90)g(to)g(pro)o (vide)h(a)e(standard)i(conforming)g(solution.)24 b(This)16 b(solution)75 1183 y(will)f(b)q(e)e(consisten)o(t)h(with)g(our)f(nonstandard) i(conforming)g(F)m(ortran)e(77)g(solution.)75 1403 y Fj(1.3)70 b(Pro)r(cesses)75 1494 y Fi(An)11 b(MPI)h(program)d(is)i(executed)i(b)o(y)e (sev)o(eral)h(autonomous)d(pro)q(cesses)14 b(that)d(eac)o(h)g(execute)i (their)f(o)o(wn)f(co)q(de,)h(in)e(a)75 1544 y(MIMD)h(st)o(yle.)18 b(The)12 b(co)q(des)h(executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)h(need)g(not)e (b)q(e)h(iden)o(tical.)17 b(The)12 b(pro)q(cesses)i(comm)o(unicate)75 1594 y(via)h(calls)h(to)g(MPI)h(comm)o(uni)o(cation)c(primitiv)o(es.)23 b(T)o(ypically)m(,)15 b(eac)o(h)h(pro)q(cessor)j(executes)f(in)e(its)g(o)o (wn)g(address)75 1644 y(space,)g(although)d(shared-memory)g(implemen)o (tations)f(of)i(MPI)h(are)g(p)q(ossible.)21 b(This)15 b(do)q(cumen)o(t)f(sp)q (eci\014es)i(the)75 1693 y(b)q(eha)o(vior)j(of)f(a)h(parallel)e(program)h (assuming)f(that)i(only)f(MPI)h(calls)g(are)g(used)h(for)f(comm)o(unicatio)o (n.)31 b(The)75 1743 y(in)o(teraction)14 b(of)g(an)g(MPI)g(program)f(with)g (other)i(p)q(ossible)g(means)e(of)g(comm)o(unication)e(\(e.g.,)i(shared)i (memory\))75 1793 y(is)f(not)g(sp)q(eci\014ed.)158 1843 y(MPI)f(do)q(es)g (not)f(sp)q(ecify)h(the)g(execution)g(mo)q(del)e(for)h(eac)o(h)h(pro)q(cess.) 20 b(A)12 b(pro)q(cess)i(can)f(b)q(e)g(sequen)o(tial,)f(or)h(can)75 1893 y(b)q(e)h(m)o(ultithreaded,)f(with)g(threads)i(p)q(ossibly)e(executing)i (concurren)o(tly)m(.)j(Care)c(has)g(b)q(een)h(tak)o(en)f(to)f(mak)o(e)g(MPI) 75 1943 y(\\thread-safe",)h(b)o(y)f(a)o(v)o(oiding)f(the)j(use)f(of)g (implicit)d(global)h(states.)158 1992 y(MPI)h(do)q(es)g(not)f(sp)q(ecify)i (either)f(the)g(in)o(teraction)f(of)g(pro)q(cesses)k(with)c(signals,)f(in)h (a)h(Unix)f(en)o(vironmen)o(t,)f(or)75 2042 y(other)h(ev)o(en)o(ts)h(that)f (do)g(not)g(relate)g(to)g(MPI)g(comm)o(unicatio)o(n.)j(I.e.,)c(signals)g(are) h(not)g(signi\014can)o(t)g(from)e(the)i(view)75 2092 y(p)q(oin)o(t)h(of)g (MPI,)g(and)h(implemen)o(ters)e(should)h(attempt)g(to)g(implemen)o(t)e(MPI)j (so)g(that)f(signals)g(are)h(transparen)o(t:)75 2142 y(an)g(MPI)g(call)f (susp)q(ended)j(b)o(y)d(a)h(signal)f(should)h(resume)g(and)f(complete)h (after)g(the)g(signal)f(is)h(handled.)158 2192 y(MPI)21 b(do)q(es)g(not)f (pro)o(vide)g(mec)o(hanisms)f(to)h(sp)q(ecify)h(the)g(initial)d(allo)q (cation)h(of)g(pro)q(cesses)k(to)e(an)f(MPI)75 2241 y(computation)e(and)i (their)g(binding)f(to)h(ph)o(ysical)f(pro)q(cessors.)38 b(It)20 b(is)f(exp)q(ected)j(that)e(v)o(endors)h(will)d(pro)o(vide)75 2291 y(mec)o(hanisms)9 b(to)j(do)f(so)g(either)h(at)g(load)e(time)g(or)i(at)f (run)h(time.)j(Suc)o(h)d(mec)o(hanisms)e(will)g(allo)o(w)f(the)j(sp)q (eci\014cation)75 2341 y(of)h(the)h(initial)e(n)o(um)o(b)q(er)h(of)g (required)i(pro)q(cesses,)h(the)e(co)q(de)h(to)e(b)q(e)i(executed)g(b)o(y)f (eac)o(h)g(initial)e(pro)q(cess,)j(and)e(the)75 2391 y(allo)q(cation)19 b(of)g(pro)q(cesses)k(to)d(pro)q(cessors.)40 b(Also,)21 b(the)g(curren)o(t)h (prop)q(osal)d(do)q(es)i(not)g(pro)o(vide)f(for)g(dynamic)75 2441 y(creation)14 b(or)g(deletion)g(of)g(pro)q(cesses)i(during)e(program)f (execution)h(\(the)h(total)f(n)o(um)o(b)q(er)f(of)g(pro)q(cesses)k(is)d (\014xed\),)75 2490 y(although)f(it)g(is)h(in)o(tended)h(to)e(b)q(e)i (consisten)o(t)g(with)e(suc)o(h)i(extension.)k(Finally)m(,)11 b(the)k(curren)o(t)g(prop)q(osal)e(do)q(es)i(not)75 2540 y(sp)q(ecify)g(a)g (naming)e(sc)o(heme)i(for)f(pro)q(cesses.)24 b(W)m(e)15 b(prop)q(ose)h(to)e (alw)o(a)o(ys)g(iden)o(tify)g(pro)q(cesses)k(according)d(to)g(their)75 2590 y(relativ)o(e)f(rank)g(in)f(a)h(group,)f(so)h(that,)f(e\013ectiv)o(ely)m (,)h(pro)q(cesses)j(are)d(iden)o(ti\014ed)g(b)o(y)g(consecutiv)o(e)h(in)o (tegers.)p eop %%Page: 4 5 bop 75 -100 a Fa(CHAPTER)14 b(1.)32 b(TERMS)13 b(AND)h(DEFINITIONS)941 b Fi(4)75 45 y Fj(1.4)70 b(Error)24 b(Handling)75 136 y Fi(MPI)16 b(pro)o(vides)g(the)h(user)g(with)e(reliable)h(message)g(transmission:)k(A)c (message)g(sen)o(t)h(is)f(alw)o(a)o(ys)e(receiv)o(ed)k(cor-)75 186 y(rectly)m(,)c(and)g(the)h(user)g(do)q(es)g(not)g(need)g(to)f(c)o(hec)o (k)h(for)f(transmission)f(errors,)i(time-outs,)e(or)h(other)h(error)g(condi-) 75 236 y(tions.)i(In)12 b(other)g(w)o(ords,)g(MPI)g(do)q(es)g(not)f(pro)o (vide)h(mec)o(hanisms)d(for)j(dealing)e(with)i(failures)f(in)g(the)h(comm)o (unica-)75 285 y(tion)f(system.)16 b(Where)c(the)g(MPI)f(implemen)o(tation)d (is)j(built)f(on)h(an)g(unreliable)g(underlying)g(mec)o(hanism,)e(then)i(it) 75 335 y(is)h(the)h(job)f(of)g(the)g(implemen)o(ter)f(of)g(the)i(MPI)g (subsystem)f(to)h(insulate)f(the)h(user)g(from)e(this)h(unreliabilit)o(y)m(,) e(or)i(to)75 385 y(re\015ect)i(unreco)o(v)o(erable)g(errors)g(as)f(global)d (program)h(failures.)18 b(Similarl)o(y)10 b(MPI)j(itself)f(pro)o(vides)h(no)f (mec)o(hanisms)75 435 y(for)i(handling)e(no)q(de)j(failures.)158 485 y(Of)10 b(course,)i(MPI)e(programs)f(ma)o(y)f(still)i(b)q(e)h(erroneous.) 18 b(A)10 b Fd(program)g(error)f Fi(can)h(o)q(ccur)i(when)e(an)g(MPI)h(call) 75 535 y(is)i(called)g(with)g(an)g(incorrect)h(parameter)f(\(non-existing)g (destination)g(in)f(a)h(send)h(op)q(eration,)f(bu\013er)h(to)q(o)f(small)75 584 y(in)k(a)f(receiv)o(e)j(op)q(eration,)e(etc.\))29 b(This)17 b(t)o(yp)q(e)g(of)g(error)h(w)o(ould)e(o)q(ccur)i(in)f(an)o(y)f(implemen)o (tation.)24 b(In)18 b(addition,)75 634 y(a)f Fd(resource)i(error)e Fi(ma)o(y)f(o)q(ccur)i(when)h(a)e(program)f(exceeds)k(the)e(amoun)o(t)e(of)i (a)o(v)n(ailable)d(system)j(resources)75 684 y(\(n)o(um)o(b)q(er)12 b(of)f(p)q(ending)i(messages,)f(system)g(bu\013ers,)i(etc.\).)k(The)12 b(o)q(ccurrence)k(of)11 b(this)i(t)o(yp)q(e)f(of)g(error)h(dep)q(ends)h(on)75 734 y(the)j(amoun)o(t)d(of)i(a)o(v)n(ailable)d(resources)19 b(in)d(the)g(system)g(and)g(the)h(resource)h(allo)q(cation)c(mec)o(hanism)g (used;)k(this)75 784 y(ma)o(y)13 b(di\013er)j(from)d(system)i(to)h(system.)21 b(The)16 b(recommended)e(implemen)o(tation)e(will)i(pro)o(vide)h(generous)h (limits)75 833 y(on)h(some)f(of)h(the)g(resources)j(so)d(as)g(to)g(alleviate) f(the)i(p)q(ortabilit)o(y)e(problem)g(this)h(represen)o(ts.)30 b(One)18 b(can)g(also)75 883 y(write)c Fd(safe)g Fi(programs,)e(that)i(are)g (not)g(sub)r(ject)h(to)f(resource)i(errors.)158 933 y(All)e(MPI)h(calls)f (return)i(an)e(error)i(parameter)e(that)h(indicates)g(successful)h (completion)d(of)h(the)i(op)q(eration,)75 983 y(or)g(the)g(error)h(condition) e(that)g(o)q(ccurred,)j(otherwise.)24 b(Ho)o(w)o(ev)o(er,)17 b(MPI)f(do)q(es)g(not)g(require)g(the)h(user)f(to)g(c)o(hec)o(k)75 1033 y(error)h(parameters.)23 b(It)16 b(is)g(recommended)e(to)i(pro)o(vide)f (an)h(implemen)o(tatio)o(n)d(where)k(the)f(en)o(vironmen)o(t)f(for)g(an)75 1082 y(MPI)f(execution)g(can)g(b)q(e)g(set)h(so)e(that)h(an)o(y)f(error)h (encoun)o(tered)i(in)d(an)h(MPI)f(call)g(will)f(cause)j(a)e(program)f(to)h (fail)75 1132 y(with)h(a)h(suitable)f(error)i(message)e(returned)j(to)d(the)i (user.)21 b(The)15 b(en)o(vironmen)o(t)f(for)g(an)h(MPI)g(execution)g(can)g (b)q(e)75 1182 y(also)d(set)j(so)e(that)g(\(nonfatal\))f(errors)i(do)f(not)g (cause)h(a)f(program)f(to)h(fail,)e(and)i(the)h(user)g(can)f(pro)o(vide)g (his)g(or)g(her)75 1232 y(o)o(wn)g(error)i(reco)o(v)o(ery)g(co)q(de.)158 1282 y(MPI)g(calls)g(ma)o(y)e(initiate)h(op)q(erations)h(that)g(con)o(tin)o (ue)g(async)o(hronously)g(after)g(the)h(call)e(returned.)23 b(Th)o(us,)75 1332 y(the)d(op)q(eration)e(ma)o(y)f(return)k(with)d(a)h(co)q (de)h(indicating)d(successful)k(completion,)d(y)o(et)h(later)g(cause)h(an)f (error)75 1381 y(exception)d(to)f(b)q(e)g(raised.)22 b(If)15 b(there)h(is)f(a)g(subsequen)o(t)i(call)d(that)h(relates)h(to)f(the)h(same)e (op)q(eration)h(\(e.g.,)f(a)h(call)75 1431 y(that)i(v)o(eri\014es)i(that)e (an)g(async)o(hronous)i(op)q(eration)e(has)h(completed\))f(then)h(the)g (error)g(parameter)f(asso)q(ciated)75 1481 y(with)e(this)g(call)g(will)f(b)q (e)i(used)g(to)f(indicate)h(the)g(nature)g(of)e(the)i(error.)24 b(In)15 b(a)g(few)h(cases,)g(the)g(error)g(ma)o(y)e(o)q(ccur)75 1531 y(after)h(all)f(calls)h(that)g(relate)h(to)e(the)i(op)q(eration)f(ha)o (v)o(e)g(completed,)f(so)h(that)h(no)e(error)i(parameter)f(can)g(b)q(e)h (used)75 1581 y(to)e(indicate)h(the)g(nature)g(of)f(the)h(error)g(\(e.g.,)f (an)g(error)h(in)f(a)g(send)i(with)e(the)h(ready)g(mo)q(de\).)k(Suc)o(h)c (error)g(m)o(ust)75 1630 y(b)q(e)g(treated)g(as)f(fatal,)e(since)j (information)10 b(cannot)15 b(b)q(e)f(returned)i(for)d(the)i(user)g(to)e (reco)o(v)o(er)i(from)e(it.)158 1763 y Fg(Discussion:)38 b Ff(A)13 b(p)q(ossible)j(mec)o(hanism)f(is)g(to)e(ha)o(v)o(e)i(MPI)e(calls)i (generate)g(a)f(signal)h(that)f(can)g(b)q(e)g(ignored)i(when)75 1813 y(an)e(error)f(is)h(encoun)o(tered.)20 b(A)13 b(more)g(mo)q(dular)i(mec) o(hanism)g(that)f(allo)o(ws)g(to)g(handle)h(an)e(exception)j(within)e(its)g (con)o(text)75 1863 y(is)g(preferred.)158 1995 y Fi(The)h(b)q(eha)o(vior)g (of)f(erroneous)j(programs)c(is)i(not)g(de\014ned)h(b)o(y)e(this)h(do)q (cumen)o(t.)21 b(The)15 b(desired)h(b)q(eha)o(vior)f(is)75 2045 y(that)d(a)f(relev)n(an)o(t)g(error)i(co)q(de)f(b)q(e)g(returned;)h(ho)o (w)o(ev)o(er,)f(e\016ciency)h(consideration)e(ma)o(y)f(dictate)i(that)f(the)h (system)75 2095 y(do)q(es)j(not)e(detect)j(certain)e(t)o(yp)q(es)h(of)e (errors.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Fri Jul 23 18:46:05 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23637; Fri, 23 Jul 93 18:46:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16247; Fri, 23 Jul 93 18:46:36 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Jul 1993 18:46:35 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16239; Fri, 23 Jul 93 18:46:34 -0400 Message-Id: <9307232246.AA16239@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 2779; Fri, 23 Jul 93 18:46:35 EDT Date: Fri, 23 Jul 93 18:44:22 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: term and definitions chapter Reply-To: SNIR@watson.ibm.com Smaller postscript file contains the sections of the point-to-point chapter that are to become part of the terms and definitions chapter (owned by Steve Otto). I put it out, because the pt2pt chapter depends on it (and ont the context chapter). Comments to Steve Otto and myself. From owner-mpi-pt2pt@CS.UTK.EDU Fri Jul 23 18:48:28 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23709; Fri, 23 Jul 93 18:48:28 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16389; Fri, 23 Jul 93 18:48:48 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 23 Jul 1993 18:48:46 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16381; Fri, 23 Jul 93 18:48:45 -0400 Message-Id: <9307232248.AA16381@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 2793; Fri, 23 Jul 93 18:48:46 EDT Date: Fri, 23 Jul 93 18:47:51 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: sent new version of point to point chapter Reply-To: SNIR@watson.ibm.com It has the new proposal for datatypes (significant change!), and a few other minor changes. Please read it and comment, as we shall need to finalize this section at coming meeting. From owner-mpi-pt2pt@CS.UTK.EDU Sun Jul 25 00:10:03 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA00530; Sun, 25 Jul 93 00:10:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21352; Sun, 25 Jul 93 00:10:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 25 Jul 1993 00:10:22 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21337; Sun, 25 Jul 93 00:10:20 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA22197; Sat, 24 Jul 93 23:10:19 CDT Date: Sat, 24 Jul 93 23:10:19 CDT From: Tony Skjellum Message-Id: <9307250410.AA22197@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu Subject: data conversion I wanted to point out that data conversion routines (in those that build buffer descriptors) need to include communicator arguments, in order to limit the scope of the conversion operators to a specific communicator. Without this, all groups will use the same conversion techniques, prohibiting optimizations (via cacheing) in the groups and/or communicators. One can argue that the argument could be the group definition instead of the communicator. But, as in Zipcode, this argument needs to be there to limit scope. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Wed Jul 28 10:05:36 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10433; Wed, 28 Jul 93 10:05:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06535; Wed, 28 Jul 93 10:03:13 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 28 Jul 1993 10:03:12 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06525; Wed, 28 Jul 93 10:03:02 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA21852 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Wed, 28 Jul 1993 15:02:55 +0100 Date: Wed, 28 Jul 1993 15:02:55 +0100 From: James Cownie Message-Id: <199307281402.AA21852@hub> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00262; Wed, 28 Jul 93 15:02:47 BST To: mpi-pt2pt@cs.utk.edu Subject: Characters in Fortran Content-Length: 1775 People, I was speaking to someone the other day who is a prime MPI candidate. They have a large real (people pay money for it !) application which is already sitting on a proprietary message passing library. One of the requirements they have is to be able reasonably to transfer character data in Fortran. (They have found that the formatting load is greater than the I/O load, and therefore do format conversion locally before shipping the formatted data to one node to do the actual writes). Can we do this in MPI ? (Of course we should be able to !) The problem is not in expressing that we want to do it, rather it is in what an implementation should actually do, given that character strings in Fortran argument passing are "funny", possibly being passed by descriptor, or inducing additional secret arguments to appear. So, Do we expect that this code should work ? character * 80 line call mpi_send( line, len(line), MPI_CHARACTER, ...) I guess we do, it should be up to the implementation to handle the peculiarities of the Fortran calling conventions. Anyone else have any thoughts on this ? Examples of machines in which this is impossible ? (For instance if the calling convention added an extra argument immediately after the character string you'd be stuck ! [I've a feeling IBM 370 did something like this, anyone know ?]) If we expect this to work I think we should probably say so explicitly, because otherwise people (implementors in particular) may not make it so. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Jul 28 10:39:51 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10628; Wed, 28 Jul 93 10:39:51 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09003; Wed, 28 Jul 93 10:38:28 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 28 Jul 1993 10:38:26 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08993; Wed, 28 Jul 93 10:38:25 -0400 Message-Id: <9307281438.AA08993@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 4135; Wed, 28 Jul 93 10:38:26 EDT Date: Wed, 28 Jul 93 10:21:54 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: jim@meiko.co.uk Cc: mpi-pt2pt@cs.utk.edu Reply-To: SNIR@watson.ibm.com Subject: Characters in Fortran Reference: Your note of Wed, 28 Jul 1993 15:02:55 +0100 Quote from IBM XLFortran User Guide: "When you pass a FORTRAN character data item as parameter, the address of the beginning of the character is passed along with a parameter that is the length of the character string. The parameter is added to the end of the declared parameter list." Therefore, your example of sending character string messages will work in Fortran on a RISC/6000. Now, for the general case: we can either introduce an additional char*n datatype in MPI (is this standard F77?) -- this is a problem because of the variable size. Or, we can just leave the document as is, and flash the possible portability issue: For compilers that do not pass address of first byte when a string parameter is used, a mechanim need be provided to find this address. In order to make a decision, it would be useful to know how many compilers DO NOT pass the address of the first byte in a char*n argument, on systems that are potential targets of MPI. From owner-mpi-pt2pt@CS.UTK.EDU Wed Jul 28 10:49:02 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10696; Wed, 28 Jul 93 10:49:02 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09633; Wed, 28 Jul 93 10:47:54 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 28 Jul 1993 10:47:53 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares9.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09624; Wed, 28 Jul 93 10:47:41 -0400 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03264 (5.65c/IDA-1.4.4 for ); Wed, 28 Jul 1993 09:47:24 -0500 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12390; Wed, 28 Jul 93 09:47:20 CDT Date: Wed, 28 Jul 93 09:47:20 CDT Message-Id: <9307281447.AA12390@godzilla.mcs.anl.gov> To: jim@meiko.co.uk Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: James Cownie's message of Wed, 28 Jul 1993 15:02:55 +0100 <199307281402.AA21852@hub> Subject: Characters in Fortran I'm glad to see someone else bring this up. Many Unix implementation do indeed add an additional argument, containing the length of the string, somewhere (often at the end) in the argument list. However, some systems use descriptors. The fact remains that passing different data types to the same routine in the same argument position is illegal in Fortran, and a good Fortran environment can detect this without assuming any interprocedural analysis. For example, consider: subroutine foo call mpi_foobar( 1 ) call mpi_foobar( 1.0 ) call mpi_foobar( '1.0' ) end This is an illegal Fortran subroutine. I have seen some Fortran systems (correctly) detect this error. Because of common usage, this restriction is usually not enforced, except sometimes for character data types. One proposal could be that for routines that can reasonably take both character and non-character values, the Fortran binding include separate routines. MPI could explicitly say that passing a character argument to the non-character version has implementation-defined behavior; thus (a) we don't have to detect it and (b) users are free to write non-portable but working code that ignores this, depending on the MPI implementation that they have. By the way, it is interesting to note that PVM 3 DOES use separate routines for each C type (where it is unnecessary because there is an explicit cast operation) but not for Fortran (where it is required). Perhaps someone familar with the rationale for the PVM design can explain why the separate routines were selected for C and not for Fortran? Bill From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 08:01:39 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20354; Thu, 29 Jul 93 08:01:39 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01650; Thu, 29 Jul 93 07:59:52 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 07:59:51 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01642; Thu, 29 Jul 93 07:59:36 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA27701 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Thu, 29 Jul 1993 12:59:32 +0100 Date: Thu, 29 Jul 1993 12:59:32 +0100 From: James Cownie Message-Id: <199307291159.AA27701@hub> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00349; Thu, 29 Jul 93 12:59:22 BST To: mpi-pt2pt@cs.utk.edu In-Reply-To: <9307281447.AA12390@godzilla.mcs.anl.gov> (message from William Gropp on Wed, 28 Jul 93 09:47:20 CDT) Subject: Re: Characters in Fortran Content-Length: 3502 > The fact remains that passing different data types to the same routine in the > same argument position is illegal in Fortran, and a good Fortran environment > can detect this without assuming any interprocedural analysis. For example, > consider: > > subroutine foo > call mpi_foobar( 1 ) > call mpi_foobar( 1.0 ) > call mpi_foobar( '1.0' ) > end > > This is an illegal Fortran subroutine. I have seen some Fortran systems > (correctly) detect this error. Indeed I wrote such a compiler, and had to remove this checking on request from my customers ! However MPI has already accepted that we are going to accept "Common Practice" here and violate the Fortran standard. > Because of common usage, this restriction is usually not enforced, except > sometimes for character data types. One proposal could be that for routines > that can reasonably take both character and non-character values, the Fortran > binding include separate routines. MPI could explicitly say that passing a > character argument to the non-character version has implementation-defined > behavior; thus (a) we don't have to detect it and (b) users are free to write > non-portable but working code that ignores this, depending on the MPI > implementation that they have. There are three half reasonable choices 1) Obey the Fortran standard, and multiply the number of routines by (character, integer, logical, float, double, complex) = 6 Unfortunately even if we do this we don't actually conform top common practice on many machines which also want INTEGER*1, INTEGER*2, COMPLEX*16 etc 2) This is Bill's suggestion... (An unpleasant hack) Recognise that the only real problem arises with character, all the rest normally having the same implementation, and have separate routines ONLY for character. 3) (Where we are now) Ignore the problem and agree to violate the Fortran standard for all datatypes. I certainly want (as I said previously) to be able to send character data. So the question is whether we have enough information in the right places in the call to be able to make it work the way it is as the moment ? Certainly it works for compilers like those on most Unix machines which pass an additional length argument at the end of the list. It works for compilers which pass by descriptor, since the code inside the library call knows from the type being MPI_CHAR that it should interpret the descriptor. It won't work for compilers which either 1) pass the length immediately after the character argument (because you don't know where to look for the type) 2) pass the length before the character argument, or at the start of the argument list (for the same reasons). Jon Flower points out that we can make the first of these cases work by (the bodge of) making the type argument occur before the data pointer... Personally I think it's probably reasonable to ignore the problem and assert that it has to work. (I certainly have no problem in implementing what we have now on my machines !) It may be worth adopting Jon's suggestion as well, consistently, so that the type always comes before the data pointer, since it is a trivial change and may help someone. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 08:11:50 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20410; Thu, 29 Jul 93 08:11:50 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02256; Thu, 29 Jul 93 08:10:41 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 08:10:39 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from lutetia.ensmp.fr by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02242; Thu, 29 Jul 93 08:10:36 -0400 Received: by lutetia.ensmp.fr id AA29560 (5.65c8/IDA-1.5 for mpi-pt2pt@cs.utk.edu); Thu, 29 Jul 1993 14:12:07 +0200 Date: Thu, 29 Jul 1993 14:12:07 +0200 From: Ronan KERYELL Message-Id: <199307291212.AA29560@lutetia.ensmp.fr> To: jim@meiko.co.uk Subject: Re: Characters in Fortran Cc: mpi-pt2pt@cs.utk.edu > > subroutine foo > > call mpi_foobar( 1 ) > > call mpi_foobar( 1.0 ) > > call mpi_foobar( '1.0' ) > > end > > However MPI has already accepted that we are going to accept "Common > Practice" here and violate the Fortran standard. There is another problem : the caracter ``_'' is not allowed in Fortran 77. Hopefully, you can write : call mpi foobar (1) :-) Ronan KERYELL Centre de Recherche en Informatique de l'E'cole Nationale Supe'rieure des Mines de Paris (CRI-ENSMP) 35, Rue Saint-Honore' 77305 FONTAINEBLEAU CEDEX FRANCE Tel: (+33 1) 64.69.48.44 Fax: (+33 1) 64.69.47.09 E-mail: keryell@cri.ensmp.fr From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 09:21:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA21323; Thu, 29 Jul 93 09:21:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06091; Thu, 29 Jul 93 09:20:22 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 09:20:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06082; Thu, 29 Jul 93 09:20:19 -0400 Received: by msr.EPM.ORNL.GOV (4.1/1.34) id AA20321; Thu, 29 Jul 93 09:20:11 EDT Date: Thu, 29 Jul 93 09:20:11 EDT From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9307291320.AA20321@msr.EPM.ORNL.GOV> To: jim@meiko.co.uk, mpi-pt2pt@cs.utk.edu Subject: Re: Characters in Fortran >Personally I think it's probably reasonable to ignore the problem and >assert that it has to work. (I certainly have no problem in >implementing what we have now on my machines !) I agree with Jim here. From working with PVM I have feedback and experience on making "common practice" work on many architectures and Fortran compilers and so far we have had no problems doing so. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 11:06:59 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22329; Thu, 29 Jul 93 11:06:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12363; Thu, 29 Jul 93 11:05:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 11:05:07 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12348; Thu, 29 Jul 93 11:05:06 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13349; Thu, 29 Jul 93 10:04:40 CDT Date: Thu, 29 Jul 93 10:04:40 CDT From: Tony Skjellum Message-Id: <9307291504.AA13349@Aurora.CS.MsState.Edu> To: jim@meiko.co.uk, Ronan.Keryell@chailly.ensmp.fr Subject: Re: Characters in Fortran Cc: mpi-pt2pt@cs.utk.edu That is a language-binding issue. PVM changes all _'s to 'f' characters, for instance; that would be one choice language binding cmte could do. ----- Begin Included Message ----- From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 07:18:35 1993 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 08:10:39 EDT Date: Thu, 29 Jul 1993 14:12:07 +0200 From: Ronan KERYELL To: jim@meiko.co.uk Subject: Re: Characters in Fortran Cc: mpi-pt2pt@cs.utk.edu Content-Length: 783 > > subroutine foo > > call mpi_foobar( 1 ) > > call mpi_foobar( 1.0 ) > > call mpi_foobar( '1.0' ) > > end > > However MPI has already accepted that we are going to accept "Common > Practice" here and violate the Fortran standard. There is another problem : the caracter ``_'' is not allowed in Fortran 77. Hopefully, you can write : call mpi foobar (1) :-) Ronan KERYELL Centre de Recherche en Informatique de l'E'cole Nationale Supe'rieure des Mines de Paris (CRI-ENSMP) 35, Rue Saint-Honore' 77305 FONTAINEBLEAU CEDEX FRANCE Tel: (+33 1) 64.69.48.44 Fax: (+33 1) 64.69.47.09 E-mail: keryell@cri.ensmp.fr ----- End Included Message ----- From owner-mpi-pt2pt@CS.UTK.EDU Thu Jul 29 11:10:45 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22352; Thu, 29 Jul 93 11:10:45 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12592; Thu, 29 Jul 93 11:09:12 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 29 Jul 1993 11:09:10 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12574; Thu, 29 Jul 93 11:09:02 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub with SMTP id AA28859 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Thu, 29 Jul 1993 16:08:58 +0100 Date: Thu, 29 Jul 1993 16:08:58 +0100 From: James Cownie Message-Id: <199307291508.AA28859@hub> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00383; Thu, 29 Jul 93 16:08:47 BST To: mpi-pt2pt@cs.utk.edu In-Reply-To: <9307291504.AA13349@Aurora.CS.MsState.Edu> (message from Tony Skjellum on Thu, 29 Jul 93 10:04:40 CDT) Subject: Re: Characters in Fortran Content-Length: 698 > That is a language-binding issue. PVM changes all _'s to 'f' characters, > for instance; that would be one choice language binding cmte could do. Yup, or just remove them entirely and replace with nothing. The in f77 you can do as Ronan suggests and write CALL MPI SEND(...) In free format (or F90) you probably have to write call mpiSend(...) which I think is fine too. (assuming case is ignored, of course). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Aug 2 15:33:30 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA08236; Mon, 2 Aug 93 15:33:30 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05896; Mon, 2 Aug 93 15:31:12 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 2 Aug 1993 15:31:07 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05883; Mon, 2 Aug 93 15:30:50 -0400 Message-Id: <9308021930.AA05883@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8183; Mon, 02 Aug 93 15:30:45 EDT Date: Mon, 2 Aug 93 15:30:41 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V6.DVI.* %%Pages: 40 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 1 106 df<1204A21200A4123812FC12D81218A212321233123E123C080F7E8E0B>105 D E /Fc 2 81 df<1470EB01F0EB03E0EB0780EB0F00131E5B5BA25BB3AD485AA25B1203485A90 C7FC120E123C12705A1270123C120E7E7F6C7E12017FA26C7EB3AD1378A27F7F7FEB0780EB03E0 EB01F0EB007014637B811F>26 D80 D E /Fd 1 1 df0 D E /Fe 2 111 df<123CA2120C5AA31360EA31F0EA3330EA3E70EA3C60EA7F00EA6380EA6198 13B0EAC1F0EAC0E00D117E9010>107 D110 D E /Ff 2 50 df<121FEA3F80EA60C0EA4040EAC060A8EA 60C0A2EA3F80EA1F000B107F8F0F>48 D<1218127812F812981218AAEAFF80A209107E8F0F>I E /Fg 4 111 df<126012F0A2126004047D830B>58 D<126012F0A212701230A31260A212C012 40040B7D830B>I<127EA2120EA25AA45AA2EA3BC0EA3FE0EA7C701278EA7030A2EAE070A4EAC0 E013C012E1EAE380EA7F00123C0C1A7E9910>98 D110 D E /Fh 5 116 df<120313801300C7FCA4121C123E126612C612CC120CA25AEA19801231EA33 00123F121C091480930C>105 D<1360137013601300A4EA0380EA07C0EA0CE0EA18C0A21200A2 EA0180A4EA0300A4126612EE12FC12780C1A81930E>I110 D114 D<1207EA1F80EA19C0EA3180EA3000121E121F EA0380124112E1EAC30012FE127C0A0D7E8C10>I E /Fi 3 62 df48 D<1206121E12FE12EE120EADEAFFE0A20B137D9211>I<387FFFE0B512F0C8FCA6B512F06C13E0 140A7E8B19>61 D E /Fj 23 123 df<127012F8A3127005057D840C>58 D<127012F012F8A212781218A31230A2127012601240050D7D840C>I<12E01278121EEA0780EA 01E0EA0078131EEB0780EB01E0EB0078141EEC0780A2EC1E001478EB01E0EB0780011EC7FC1378 EA01E0EA0780001EC8FC127812E019187D9520>62 D<48B512804814E039003C01F0EC0078153C 151C5B151EA35BA44848133CA3153848481378157015F0EC01E039078003C0EC0780EC0F00147C B55A14C01F1C7E9B22>68 D<001FB512F0A2383C07800038146012701260EB0F0012C0A3D8001E 1300A45BA45BA45BA4387FFF80A21C1C7F9B18>84 D97 D<123F5A120F120EA45AA4EA39E0EA3FF8EA3E38EA3C1C1278EA701EA3EAE03CA3133813781370 13E0EA71C0EA3F80EA1E000F1D7E9C12>I100 D<13F8EA03FEEA0F06121E123C1238EA781CEA7FF8EAFFE0EAF000A4EA7004130EEA 383CEA1FF0EA0FC00F127F9113>II<13E0 1201A2EA00C01300A6120EEA1F8012331263124312C3EA0700A2120EA3EA1C201360A2EA38E013 C0EA1F80EA0F000B1C7F9B0E>105 D<1307130FA213061300A613F0EA01FCEA031C1206A2120C 1200A21338A41370A413E0A4EA01C01261EAF380EAF70012FE127C1024809B11>III<381C0F80383F3FC0386770E013C0EAC780A2EA0700 A2380E01C0A3EB0384001C138CA2EB071C1418383803F0381801E016127F9119>110 D<13FCEA03FF38078780380E03C0EA1C0112381278A2EAF003A3EB0780A2EB0F00EA701E6C5AEA 1FF0EA0FC012127F9114>I<380707C0380FCFE03819D870EBF038EA31E013C00001133CA23803 8078A31470380700F0EB80E0EB81C0EBC380380EFF00137C90C7FCA25AA4B47EA2161A819115> I115 D<13C01201A3EA0380A4EAFFF0A2EA0700A2 120EA45AA31310EA3830A2136013C0EA1F80EA0F000C1A80990F>I<000E1360381F80E0123312 63384381C012C3EA0701A2380E0380A31488EB0718A2130FEB1F303807F3F03803E1E015127F91 18>I120 D<000E1360381F80E012331263384381C0 12C3EA0701A2380E0380A4EB0700A25B5BEA07FEEA03EEEA000E5B1230EA78381330EA70E0EA3F C06CC7FC131A7F9114>I<3801E180EA07F1380FFF00EA0C1EEA180CC65A5B5B5B485A48C7FCEA 0601485A1218EA3F0EEA7FFCEA61F8EAC0F011127F9113>I E /Fk 2 4 df0 D<12041206A2EAC660EAF7E0EA3F80EA1F00EA3F80EAF7E0EAC660 EA0600A212040B0D7E8D11>3 D E /Fl 18 118 df70 D<133F3801FFE0487F3807C0F8380F807C381E001E003E131F003C7F48EB0780 A348EB03C0A86C130700781480A2007C130F003C1400003E5B6C133E6C6C5A6C6C5A6CB45A6C5B D8003FC7FC1A1F7E9D1F>79 DI82 D97 D99 D<130EABEA0F8EEA1FEEEA3FFEEA7C3E EA700EA212E0A612F0EA701EEA7C3EEA3FFEEA1FEEEA0F8E0F1D7E9C15>II<3803C3C0EA0FFF5A381C3800487EA56C5AEA1FF85BEA3BC00038C7FCA2EA1FFC13FF481380 EA700738E001C0A3EAF003387C0F80383FFF006C5AEA07F8121B7F9115>103 D<12F0A41200A71270B2041D7E9C0A>105 D<12E0B3AB031D7D9C0A>108 D<38E3F03F39EFF8FF80D8FFFD13C039F81F81E038F00F00EAE00EAD1B127D9124>III114 DI<121CA6EAFFE0A3EA 1C00AB1320EA1FF0120FEA07C00C187F970F>II E /Fm 49 122 df<1238127C127EA2123E121EA2127C12F81260070A7A8414>44 D<127012F8A312700505798414>46 D50 D<13F8120113B81203EA0738A2120F120E121C123C1238127812F0B51280A338003800A43803FF 80A311177F9614>52 D<127012F8A312701200A6127012F8A312700510798F14>58 D61 D65 DIIIIII<38FE3F80A338380E00A7EA3FFEA3EA 380EA738FE3F80A311177F9614>II76 D<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38 E3A21303A538FE0FE0A31317809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213 EE136EA4EAFE3EA311177F9614>III82 DII<38FF8FF8A3381C 01C0AEEA1E03000E1380EA0F073807FF006C5AEA00F81517819614>I<38FE3F80A338380E00A2 6C5AA56C5AA4EA0630EA0770A3EA0360A213E0A26C5A11177F9614>I88 D<38FE3F80A3383C1E00EA1C1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A3 11177F9614>I91 D93 D97 D<12FCA3121CA413FCEA1FFE7FEB0780EA1E03001C13C0 1301A4EA1E03EB0780EA1F0FEBFF005BEA1CF81217809614>II<137EA3130EA4EA07CE EA1FFE123FEA7C3EEA781EEAF00E12E0A4EAF01EEA703EEA787E383FFFC0EA1FEFEA078F12177F 9614>II<133C13FE1201EA03DE138C1380A2EAFFFEA3EA0380AAEA7FFC A30F177F9614>II<12FCA3121CA413F8EA1FFE7F130FEA1E07121CA838FF9FE0A31317809614>I<1203 EA0780A2EA0300C7FCA4EAFF80A31203AAEAFFFEA30F187D9714>I<136013F0A213601300A4EA 1FF0A3EA0070B0EA60F0EAF1E012FF1380EA3E000C207E9714>I108 D110 DII<38FF1F80EB7FC013FF3807E080EBC0005BA290 C7FCA6EAFFFCA31210808F14>114 DI<1207A5EAFFFCA3EA0700A613 0EA3139E13FCEA03F8EA00E00F157F9414>II<38FE7F80A3380E7800EA07705B12036C5A12037FEA0770EA0E78133838FF7F80A31110 7F8F14>120 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C5AA35BA3EA7B8012 7F90C7FC127E123C11187F8F14>I E /Fn 7 104 df0 D<127012F8A3127005057D8C0C>I3 D15 D<00C01306B36C130E0060130C00 70131C6C1338381F01F03807FFC03800FE00171A7E981C>91 D<131F137F13F0EA01E013C0ADEA 0380EA070012FE12F812FE1207EA0380EA01C0AD13E0EA00F0137F131F10297E9E15>102 D<12F8B4FCEA0F80EA03C01201ADEA00E013F0133F131F133F13F013E0EA01C0AD1203EA0F80EA FF0012F810297E9E15>I E /Fo 36 122 df<127012F812FCA3127C120CA21218A21230127012 20060D7D850C>44 D<127812FCA4127806067D850C>46 D<127812FCA412781200A5127812FCA4 127806117D900C>58 D<1303497EA3497EA2497EA3EB33F0A2EB63F81361EBE1FC13C0A248487E 13FF487FEB003FA20006EB1F80A2000E14C039FFC0FFFCA21E1A7F9921>65 D<90381FE0209038FFF8E03803F80F3807C003381F800190C7FC123E007E1460127C12FC1500A6 1560127C127E003E14C07E90388001803907C003003803F80E3800FFFCEB1FE01B1A7E9920>67 DIII< EAFFFCA2EA0FC0B3A4EAFFFCA20E1A7F9911>73 D76 DI<39FFE00FFCA2390FF000C07FEA0DFCEA0CFE137E7F1480EB1FC0EB0FE013 07EB03F014F8EB01FCEB00FE147E143FA2141F140F14071403A238FFC00114001E1A7E9923>I< B512E014F8380FC07E80801580A515005C147EEBFFF814E001C0C7FCA9EAFFFCA2191A7E991E> 80 D<007FB51280A238783F07007013030060130100E014C000C01300A300001400AF380FFFFC A21A1A7E991F>84 D97 DIIII<137E48B4FC3803CF80EA078F120FEB87001380A3EAFFF0A2EA0F80ADEA7F F0A2111A80990E>I<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C3CEA3FF8EA 37E00070C7FCA2EA3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF803807FC001319 7F9016>II<121E12 3FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>I108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7FE7FE0A22311 7F9026>IIII114 DI<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03 E00D187F9711>I<38FF1FE0A2EA1F03AB1307130F380FFFFCEA03F316117F9019>I<38FF83F0A2 381F0180A21383000F130013C7EA07C6A2EA03ECA2EA01F8A36C5AA2136014117F9017>I<39FF 9FF1F8A2393F07C0E0001F14C014E0EB8DE1000F148014F13907D8F30014FBEBF8FF3803F07EA2 3801E03CA33800C0181D117F9020>I<38FF8FF0A2381F0700EA0F8EEA07CC13D8EA03F86C5A6C 7EA2487EEA033E487E120E381C0F8038FE1FF0A214117F9017>I<38FF83F0A2381F0180A21383 000F130013C7EA07C6A2EA03ECA2EA01F8A36C5AA21360A2EA70C012F8EAC18012C3007FC7FC12 3C14187F9017>I E /Fp 18 122 df45 D97 D<123F5A120F120EA45AA4EA39C0EA3FE0EA3E70EA3830EA78381270A3EAE070A3136013E013C0 12E1EA6380EA7F00123C0D1D7B9C13>I101 DII<13C0EA01E013C01380C7 FCA6120E123FEA33801263EAC700A21207120EA35A1340EA38C0A3EA3980EA3F00121E0B1C7D9B 0D>105 D107 DI<391E1F 07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380E0A3EC81C2391C0701 C6A2EC038CA239380E01F839180600F01F127D9122>III114 DI<13C01201A3EA0380A4EAFFE0A2EA0700A2 120EA45AA31320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I118 D120 D<380F0180EA1F8312331263EB870012C3EA0707A2EA0E0EA4 EA0C1C121C120CEA0E3CEA0FF81203EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7FC111A 7D9114>I E /Fq 46 122 df45 D<12F0A404047B830E>I<13C0120112 0712FFA212FB1203B3A7B5FCA310227CA118>49 DII<131FA2132FA2136F13EF13CF 1201A2EA038FA21207130F120F120E121E123CA21278A212F0B512F8A338000F00A915217FA018 >I<383FFF80A3003CC7FCA913F8EA3FFE7FEB0F80EA3E07383C03C01238380001E0A7384003C0 A212E038700780387C1F00EA3FFEEA0FFCEA03F013227EA018>I<137E48B4FC5AEA07C148C7FC 121EA25AA25AA213F8EAF3FEEAF7FF38FF0F80EAFC0738F803C0A2130100F013E0A51270A21278 EB03C01238003C1380EA1C07381F0F00EA0FFE6C5AEA01F013237EA118>IIII<131FA2497EA2133BEB7BC013731371EBF1E0A213E0 00017FA213C000031378A2138000077FA21300487FA2EBFFFE487FA2381C000F003CEB0780A212 380078EB03C0A2127000F0EB01E0A25A1B237EA220>65 DIIII<00FC147E6C14FEA3 00F7EB01DEA3EB800300F3149EA2EBC007A200F1141EA2EBE00F00F0130EA2EBF01EA2EB701CEB 783CA2EB3838EB3C78A2EB1C70A2EB1EF0EB0EE0A2130FEB07C0A390C7FC1F237BA22A>77 D<00FC131E7EA312F7A2EAF380A213C012F1A213E012F013F0A21378A21338133CA2131EA2130E 130F1307A2149E1303A2EB01DEA2EB00FEA3147E17237BA222>III82 D<13FF000313C04813F0EA0F83EA 1E0048132014005AA5127C123C123FEA1FC0EA0FFC6CB4FC0001138038003FC0EB07E0EB01F013 0014F81478A5124000E013F07E38F801E0387F07C0383FFF80000F1300EA01FC15257EA31B>I< 00F0133CB3A900781378A36C13F0381E01E0EA1F87380FFFC000031300EA00FC16247BA221>85 D97 D<12F0ADEAF1F0EAF7FCB47EEAFC1F487E38F00780A2EB 03C0A8EB0780A238F80F00EAFC3EEAFFFEEAF7F8EAF3F012237CA219>IIII<133F13FF5A EA03C113801207A8EAFFF8A3EA0780B3102380A20F>I<3801F0F8EA07FF5A381F1F80381E0F00 383C0780A5381E0F00EA1F1F13FE5BEA39F00038C7FCA2123CEA3FFE381FFFC014E04813F0EA78 0138F800F8481378A4007813F0383E03E0381FFFC06C13803801FC0015217F9518>I<12F0ADEA F1F8EAF7FCEAFFFEEAFE1EEAF80FA212F0B010237CA219>I<12F0A41200A912F0B3A404237DA2 0B>I<13F0A41300A913F0B3A91280EAE3E012FFEA7FC0EA1F000C2D83A20D>I<12F0ADEB1F80EB 3F00133E5B5BEAF1F0EAF3E0EAF7C012FFA27FA2EAFDF0EAF8F8EAF078137C7F131E131F7F1480 EB07C012237CA218>I<12F0B3B104237DA20B>I<39F0F807C039F3FE1FF0B5EA7FF839FE0FF078 00FCEBE07C39F807C03CA200F01380AF1E167C9529>IIII114 DI<120FA6EAFFF8A3EA0F00AE1308131C EA07FC13F8EA03E00E1C7F9B12>II<38 F001E0A2387803C0A3383C0780A3001E13005BA2EA0F0E131EA2EA071C139C13BCEA03B8A213F8 6C5AA213167F9516>I<387801F0007C13E0383E03C0EA1E07000F138038078F0013DEEA03FC12 016C5A137013F8487EEA03DC139EEA078F380F0780121EEB03C0383C01E0387800F000F813F815 16809516>120 D<38F001E0A2387803C0A2127C383C0780A2121E14005B120F130EEA071E139E EA039CA3EA019813D8EA00F85BA35BA212015BA2EA4380127F90C7FC127E13207F9516>I E /Fr 13 86 df<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C7012 00A213E0A6EA01C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<14301470A214F0801301A2EB 0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206120F39 7FC0FFF012FF1C1D7F9C1F>65 D<903807F030EB1FFC90387C0E603901F003E0EA03C038078001 EA0F00000E1300001E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F 5B3807C0E06CB45AD8007EC7FC1C1E7C9C1E>67 D<000FB512E0A23800F003EC01C01400A33801 E060A31500EBE1E013FF485B13C113C0A215C0ECC18038078001A2EC0300A25C140E380F003EB5 12FE5C1B1C7E9B1C>69 D<380FFF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F 9B0F>73 D77 D<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387 A2EB83C1390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C 7E9B1F>II<380FFFFC14FF3900F00F80140315C01401A23801E003A31580 1407EC0F003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I83 D<001FB512F05A383C07810038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6137C 381FFFE0A21C1C7C9B1E>I<39FFF8FFE0A2390F001E00140CA4001E5BA6485BA6485BA300385B A2383C0180D81C03C7FCEA0E0EEA07FCEA01F01B1D7A9B1F>I E /Fs 10 58 df<121EEA7F801261EAC0C0A7EA6180127FEA1E000A0D7E8C0E>48 D<121812F8A21218A812 FFA2080D7D8C0E>I<123EEA7F80EAC1C012C01200A2EA0180EA0300120EEA18C01270EAFF80A2 0A0D7E8C0E>I<123E127FEA618012011203EA1F00A2EA0180EA00C012C012C1EA7F80EA3E000A 0D7E8C0E>I<12075AA2121B1233127312E3EAFFC0A2EA0300A2EA1FC0A20A0D7E8C0E>II<120FEA1F8012 31EA60005AEAFF80A2EAC0C0A31261EA3F80EA1E000A0D7E8C0E>I<1260EA7FE0A2EAC0C0EAC1 80EA03001206A35AA50B0E7E8D0E>I<121E123FEA6180A21273EA3F00A2EA6380EAC0C0A212E1 EA7F80EA3F000A0D7E8C0E>I<121E127FEAE180EAC0C0A3127FA21200EA01801263EA7F00123C 0A0D7E8C0E>I E /Ft 71 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4 B5FCA2EA0E07AC387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07 AC387F9FE0A2131A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E1381903803 80C01500A4B612E0A2380E0380AC397F8FE3FCA21E1A809920>I34 D<126012F012F812781218A31230127012 E01240050B7D990B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0 AA1260A21270A212301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E 121C120C7E1207A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A 267E9B0F>I<126012F0A212701230A31260A212C01240040B7D830B>44 DI<126012F0A2126004047D830B>I<130CA2131C1318A213381330A213 701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C1218A212381230A21270 1260A212E05AA20E257E9B13>II<12035AB4FCA21207B3EAFFF0A20C19 7D9813>III<1370A213F0A21201120313701206A212 0C121C12181230A2126012E0B5FCA2EA0070A5EA07FFA210197F9813>I<1260EA7FFEA213FCEA 600CEAC0181330A2EA006013C0A21201A2EA0380A41207A8EA03000F1A7E9813>55 D<126012F0A212601200A8126012F0A2126004107D8F0B>58 D<126012F0A212601200A8126012 F0A212701230A31260A212C0124004177D8F0B>I61 D63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47E A2130000061378A3487F121E39FF81FFC0A21A1B7F9A1D>65 DIIIIII<39FFF3FFC0A2390F003C00A9EBFFFCA2 EB003CAB39FFF3FFC0A21A1A7F991D>II76 D<39FF8001FF6D5A000F14 F0A2380DE006A3380CF00CA3EB7818A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEB C30F201A7F9923>II<13 7F3801FFC03807C1F0380F0078001E7F001C131C487F0078130FA200707F00F01480A80078EB0F 00A36C131E001C131C001E133C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II82 DI<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FC AF3803FFF0A2191A7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A3801E0E038 007FC0011FC7FC1A1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC 5DD807801306EB819EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE 13F0EBFC03017C5BA2EB7801A201385BEB3000241B7F9927>87 D<38FFE1FFA2380F80783807C0 60000313E06D5A3801E180EA00F301FBC7FC137F137E133E131E131F5BEB7780EB67C013E3EBC3 E0380181F0EA0380497E00061378000F7F39FFC1FFC0A21A1A7F991D>I<39FFE01FF0A2390F80 0F00000713066C6C5A13E000015BEBF038000013306D5A137CEB3CC0133F6D5AA26DC7FCA9EBFF F0A21C1A80991D>I92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A238 1F0F00EA1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A612 70131EEA3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA03BCEA073CEA0E1813 00A5EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870 EA3FE0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F 9013>I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA2 1218C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A212 00B0126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B 121DEA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80 A2091A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D10 7F8F20>IIIIIII<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>II<38FF3F80A2381C0E00130CA26C5AA21338EA 0730A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E 5B13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA 03B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80 A2381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE1278 11177F8F14>III E /Fu 78 126 df<126012F0AF12601200A4126012F0A212600419779816>33 D38 D<13E01201EA07C013005A121E5A1238127812 70A312F05AA77E1270A312781238123C7E7E7E13C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7120113C0A3120313801207EA0F 00121E121C127C12F05A0B217C9C16>III<1238127C127EA2123E120E121E123C127C12F81260070B798416>II<127012F8A312700505788416>IIII II<137C13FC13DC1201EA039CA2EA071C120F12 0E121E123C1238127812F0B512E0A338001C00A53801FFC0A313197F9816>II<13F8EA03FEEA0FFFEA1F0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2 EAF80F38F00780A2EAE00312F0A21270EA7807EB0F006C5AEA1FFEEA0FF8EA03E011197E9816> I<12E0B51280A338E00F00131EC65A13381378137013F05B12015B12035BA3120790C7FCA7111A 7E9916>I57 D<127012F8A312701200A8127012F8A312700512789116>I<1238127CA312381200A81238127C A3123C121C123C123812F812F012600618799116>III<13E0487EA213B0A2EA03B8A31318 EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816>65 DI<3801F180EA07FF5AEA1F0FEA3C071278130312 7000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7 FC14E0A5EA7FFFB5FC7E13197F9816>II I<387F1FC038FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013 197F9816>II<387F0FE038FF8FF0387F0FE0381C07 80EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E0 38FF8FF0387F07E01419809816>75 D I<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313 197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377 A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>IIIIII<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816> I<387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519 809816>I<38FE0FE0EAFF1FEAFE0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813 B8A3EA01B013F0A26C5A13197F9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803 A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F 131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207131CA2EA0E0EA2 487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<387FFF80B5FCA238E007 005B131E131CEA003C5B137013F0485A5B1203485A90C7FC5A381E0380121C123C12781270B5FC A311197E9816>II93 D95 D97 D<127E12FE127E120EA4133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E013 81EB83C013FF000E1300EA067C1419809816>II<133F5B7F1307A4EA03E7EA0F FF123FEA3C1F487E1270EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E01419 7F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FC A2EA01C0ACEA3FFE487E6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380E A3EA3C1E6C5AEA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000 A3EAF001387C07C0383FFF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0F FF148013871303A2120EA9387FC7F038FFE7F8387FC7F01519809816>II<13301378A213301300A4EA1FF8A3EA 0038B3EA6078EAF0F0EAFFE0EA7FC0EA3F800D237E9916>I<127E12FE127E120EA4EB7FE0A3EB 0F00131E5B5B5BEA0FF8A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816> II<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870 AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713 873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C 13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4 EA03B8A3EA01F0A26C5A13127F9116>I<38FF1FE013BF131F38380380A413E33819F300A213B3 EA1DB7A4EA0F1EA313127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F012016C5A 12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<387F1FC038FF 9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279 127BEA7F806CC7FC123C131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A13F8485A48 5A485A48C7FC381E01C0123C1278B5FCA312127F9116>II<12FCB4FC13C01203 1201A97F6CB4FCEB7F80A2EBFF00EA01E05BA9120312FF90C7FC12FC11207E9C16>125 D E /Fv 30 122 df<90381FE1F890B512FC3903F03F3E3807C07EEB807C000F141C1500A5B612 C0A2390F807C00AE397FE1FFC0A21F1D809C1C>11 D45 D63 D65 D<90381FE0209038FFF8E03803F80F3807E007380FC001EA1F 8048C7FCA2007E1460A212FE1500A7007E1460A27E15C06C7E390FC001803907E003003803F80E 3800FFFCEB1FE01B1C7D9B22>67 D70 D<3807F860381FFEE0EA3C07EA7801 EA700012F01460A26C130012FEEAFFE0EA7FFE6C7E1480000F13C06C13E0EA007FEB03F0130113 0012C0A214E07E38F001C0EAFC0338EFFF00EAC3FC141C7D9B1B>83 D97 DIIII<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E1FEA1FFCEA33F0 0030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F0383E03E0381FFFC0 3803FE00151B7F9118>103 DI<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>I107 DI<39FF1FC0FE90387FE3FF3A 1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE0381F E1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0 EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFFE0 A2171A7F911B>I<3803E060380FF8E0EA1F0DEA3E07EA7E03127C12FCA6127C127EEA3E07EA3F 0FEA0FFBEA07E3EA0003A6EB1FFCA2161A7E9119>III<1203A35AA25AA2123F EAFFFCA2EA1F00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA 0F03EBFEFFEA03F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A238 03E300A2EA01F6A213FE6C5AA21378A2133016127F9119>I<39FF8FF8FEA2391F03E030A20183 1370000FEBF0601386D807C613C0EBCEF8EBEC790003EB7D80EBF83D0001EB3F00A2497E000013 1EEBE00EA21F127F9122>I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA01FC5B1200137C 13FEEA01DFEA039F38070F80380607C0380C03E038FF07FCA216127F9119>I<38FFC1FCA2381F 0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5AA21378A21330A25B1270 EAF8E0EAC0C0EAE380007FC7FC123E161A7F9119>I E /Fw 44 122 df 45 D<12F8A505057A8411>I<137F3801FFC0000713F0487F13C1381F007C48137E003E133E003C 131E007C131FA300787F00F81480AF00781400007C5BA36C133EA2003F137E6C6C5A380FC1F813 FF6C5B000113C06C6CC7FC19297EA71E>48 D<1318133813F8120712FFA312F81200B3AA387FFF F0A414287CA71E>I<13FE3803FFC04813E04813F0381F03F8383C00FC0038137E12780070133E 00F0133FA20060131F1220C7FC143FA2143EA2147CA214F8EB01F0EB03E0130714C0EB0F80EB1E 005B5B5B485A485A485A48C7FC121E5A007FB5FCA418287EA71E>I<137F3801FFC0000713F048 7FEA1F81383E00FC003C137E5A0030133E1210C7127EA2147CA214FC495AEB07F0EBFFE05C8080 EB01F8EB007C147E80A2EC1F80A512400060EB3F0012F06C5B007C137E383F81FC6CB45A6C5B00 0313C0C66CC7FC19297EA71E>II<38 3FFFFCA4003EC7FCAA133FEBFFC0003F13E014F0EBC1F8EB80FCEB007C003E137EC7123E143FA7 12200060137E127000F813FCEA7C01383F03F8EBFFF0000F13E06C13803800FE0018287EA61E> III<137F3801FFC0000713F0487F381FC1FC383F007E003E133E007E133F007C7FA56C133EA26C 5B380FC1F86CB45A6C5BA2000F13F8381FC1FC383F007E003E133E487FA248EB0F80A6007CEB1F 00A2007E5B6C137E381FC1FC6CB45A6C5B6C5BC66CC7FC19297EA71E>I<137F3801FFC0487F00 0F7FEBC1F8381F007C123E487F141E141F5AA2801580A4141F127CA2007E133F003E137F003F13 FF381FFFEF6C13CF6C138F3903FE1F00C7FCA2141E143EA25CA25C381801F0381E07E0383FFFC0 6C5B6C48C7FCEA03F819297EA71E>I 66 D<49B4FC010F13E0013F13F85BEBFF013901F800304848131048481300485A485AA248C8FC A2127EA45AAA127EA47EA26C7EA26C7E6C7E6C6C13046C6C131C6CB413FCEB7FFF6D13F8010F13 E0010113001E2C7CAA26>II<12FCB3B3A6062A7BA911>73 D77 DI80 D83 DI<00FC147CB3AE007E14F8A26C130115F0391F 8003E0EBC00F390FF03FC06CB512800001EBFE006C6C5AEB1FE01E2B7BA929>I97 D<12F8AF133F38F9FFC000FB13E0B512F0 130738FC01F838F800FC147CA2147E143EA7147CA36C13F8130138FF07F0EBFFE000FB13C000F9 138038F87E00172A7BA91F>II<143EAF13FC EA03FF4813FE5AEA1FC1383F007E003E133E5AA212FC5AA77E127CA26C137E003F13FEEA1FC1EA 0FFF6C13BE6C133EEA00FC172A7EA91F>I<137E3803FF804813C04813E0381F83F0EA3F00003E 13785A14380078133CB512FCA400F8C7FCA31278127CA27E003F130C381FC07C380FFFFC7E0001 13F038007F80161B7E9A1B>II<9038F807803903FE7FC0000FB5FC5A903807C000383E03E0A2486C7EA66C485AA2 6C485A13FF485BD83BFEC7FCEA38F80078C8FCA27E383FFFC014F86C7F80487F387C007F48EB1F 80140FA36C131F007EEB3F00383F80FE13FF000F13F86C5BC613801A287E9A1E>I<12F8AF133F EBFF8000FB13C0B512E0130738FE03F0EAFC01A312F8B2142A7BA91F>I<12FCA61200A9127CB3 A9062A7DA90E>I<13FCA61300A9137CB3AFEA40FCEAE1F812FF13F0EA7FE0EA0F800E3683A910> I<12F8AFEB01F8EB03F0EB07E014C0EB0F80EB1F00133E5B5B12F9EAFBF012FF7F7FA2EAFE7E48 7E12F8EB1F80130F14C0EB07E0A2EB03F0130114F8EB00FC162A7BA91D>I<12F8B3B3A6052A7C A90E>III<137F3801FFC0000713F0487F381FC1FC383F007E003E133E487FA200787F00F8 1480A7007CEB1F00A2007E5B003E133E003F137E381FC1FC6CB45A6C5B000113C06C6CC7FC191B 7E9A1E>II114 DII<38F801F0B3A21303A2EAFC0FB5FCEA7F FDEA3FF9EA1FC1141B7B9A1F>I<00F8EB0F807E007CEB1F00A2127E003E133EA26C133C147CA2 6C6C5AA36C6C5A13C1A23803E1E013E300015BA213F33800F780A3017FC7FCA3191B7F9A1C>I< 00F8EB0F807E007CEB1F00127E003E133EA27E5CA2EA0F805CEA07C05C13C1EA03E15C120113E3 EBF3C012005C137313770137C7FC133F133E131EA25BA35BA25BA2EA7FE0A25B90C8FC19277F9A 1C>121 D E /Fx 79 124 df11 D<133FEBFF803803C1C0EA0703A2380E0180 90C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C17>II<90383F03F09038FFCFF83903C0FC1C390701F0 3CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA030012 06120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C12 0C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380 A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>II<1306ADB612E0A2D80006C7FCAD1B1C7E9720>I<127012F012F8A2127812 18A31230A2127012601240050D7D840C>II<127012F8A3127005057D84 0C>I<1303A213071306A2130E130C131C1318A213381330A213701360A213E013C0A212011380 12031300A25A1206A2120E120CA2121C1218A21238123012701260A212E05AA210297E9E15>I< EA03C0EA0FF0EA1C38EA381CA2EA700EA3EAF00FADEA700EA3EA381CA2EA1C38EA0FF0EA07E010 1D7E9B15>I<12035A123FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>III<131CA2133C137CA213DC1201139C1203EA071C1206120E120C 121812381230126012E0B512C0A238001C00A63801FFC0A2121C7F9B15>II<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA780012 70A2EAF3F0EAF7F8EAFC1CEAF81E130E12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0 101D7E9B15>I<1260387FFF80A21400EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA412 07A76CC7FC111D7E9B15>III<127012 F8A312701200A8127012F8A3127005127D910C>I<127012F8A312701200A8127012F012F8A212 781218A31230A2127012601240051A7D910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C 7E8F20>61 D<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003C A200067FA3001F131F39FFC0FFF0A21C1D7F9C1F>65 D I<90381F8080EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091 C7FCA70070EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E 9C1E>IIII< 90381F8080EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7 FCA5ECFFF0A20070EB07801278A27E121C121E7E3807800F3803F0393800FFF090381FC0001C1E 7E9C21>I<39FFF3FFC0A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>II76 DIIII82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0 EA3FF8EA1FFEEA07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFC EA83F8121E7E9C17>I<007FB512C0A238780F03007013010060130000E014E000C01460A40000 1400B03803FFFCA21B1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C0 3800E0E0EB7FC0011FC7FC1A1D7E9B1F>I<39FFE01FF0A2390F000780EC0300A238078006A2EB C00E0003130CA2EBE01C00011318A26C6C5AA3EB7860A36D5AA3EB1F80A36DC7FCA313061C1D7F 9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060A32603C0E7 13C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E137EA3013C13 3CA3011C133801181318281D7F9B2B>I<397FF0FFC0A23907C03E0000031338EBE03000011370 EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371EB61F0EBC0F800011378 EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I<12FEA212C0B3B312FEA207297C 9E0C>91 DI<12FEA21206B3B312FEA20729809E0C>I<1208121812301260A212C0A312F812FCA2127C12 38060D7E9C0C>96 DI<12FCA2121CA9137EEA1DFF381F8780381E 01C0001C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>I III<1378EA01FCEA039EEA071EEA0E0C1300 A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2 121CAB38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A209 1D7F9C0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C 131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E0 39FDFF9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307 123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2 EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380F FBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F8 6C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E 000713CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA03 8EEA01DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A215128091 16>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EA F180A200C7C7FC127E123C131A7F9116>III E /Fy 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815FE489039001F FF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F01 3FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB 01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF00101 07903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC0113C0 EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5BA212FF AA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB3F8090 3A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fz 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D83 D99 D<903801FFC0011F13F8017F13FE9038FFC1FF00039038007F80D807 FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03FCA212FFA290B6FCA301E0C8FCA5127F A27F003F153CA26C6C147C000F15786C6C14F86C6CEB01F06C6CEB07E06C9038E03FC0013FB512 00010F13FC010013E026267DA52D>101 D105 D<01FEEB7FC000FF903803FFF8 020F13FE91381F03FFEC3C010007017814800003497E6D4814C05CA25CA291C7FCB3A4B5D8FC3F 13FFA430267CA537>110 DI116 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 75 356 a Fz(Section)33 b(1)75 564 y Fy(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)884 780 y Fx(Marc)15 b(Snir)684 830 y(William)10 b(Gropp)j(and)h(Ewing)f(Lusk)75 967 y Fw(1.1)59 b(Intro)r(duction)75 1058 y Fx(This)10 b(c)o(hapter)i(is)e(a)g(draft)g(of)g(the)h(curren)o(t)h (prop)q(osal)e(for)g(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)14 b(Sending)c(and)g(receiv-)75 1108 y(ing)i(of)g(messages)h(b)o(y)g(pro)q (cesses)i(is)e(the)g(basic)g(MPI)g(comm)o(unication)d(mec)o(hanism.)15 b(All)d(other)h(comm)o(unication)75 1157 y(functions)18 b(can)g(b)q(e)h (implemen)o(ted)d(on)h(top)h(of)g(this)g(basic)g(comm)o(unication)d(la)o(y)o (er)i(\(although)g(a)h(more)f(direct)75 1207 y(implemen)o(tatio)o(n)11 b(ma)o(y)h(lead)i(to)g(greater)h(e\016ciency\).)158 1257 y(The)g(basic)g(p)q (oin)o(t)f(to)h(p)q(oin)o(t)f(comm)o(unication)d(op)q(erations)k(are)h Fv(send)d Fx(and)i Fv(receiv)o(e)p Fx(.)k(A)c Fv(send)e Fx(op)q(eration)75 1307 y(creates)19 b(and)e(sends)h(a)e(message.)27 b(The)18 b(op)q(eration)f(sp)q(eci\014es)h(a)f Fv(send)h(bu\013er)e Fx(in)g(the)i(sender)g(memory)d(from)75 1357 y(whic)o(h)f(the)i(message)e (data)g(is)g(tak)o(en.)20 b(In)15 b(addition,)e(the)i(send)h(op)q(eration)e (asso)q(ciates)i(an)e Fv(en)o(v)o(elop)q(e)e Fx(with)i(the)75 1407 y(message.)26 b(This)17 b(en)o(v)o(elop)q(e)g(sp)q(eci\014es)i(the)e (message)g(destination)f(and)h(con)o(tains)f(distinguishing)g(information)75 1456 y(that)e(can)g(b)q(e)h(used)f(b)o(y)g(the)g Fv(receiv)o(e)f Fx(op)q(eration)h(to)f(select)j(a)d(particular)h(message.)158 1506 y(A)g Fv(receiv)o(e)f Fx(op)q(eration)h(consumes)g(a)f(message.)18 b(The)d(message)f(to)f(b)q(e)i(receiv)o(ed)g(is)f(selected)i(according)e(to) 75 1556 y(the)g(v)n(alue)g(on)f(its)h(en)o(v)o(elop)q(e,)g(and)g(the)g (message)g(data)f(is)h(put)g(in)o(to)g(the)g Fv(receiv)o(e)h(bu\013er)p Fx(.)158 1606 y(The)g(next)h(sections)g(describ)q(e)h(the)f(basic)f(\(blo)q (c)o(king\))f(send)i(and)f(receiv)o(e)h(op)q(erations.)22 b(W)m(e)14 b(discuss)j(send,)75 1656 y(receiv)o(e,)k(basic)e(comm)o(unication)d(seman)o (tics,)j(t)o(yp)q(e)h(matc)o(hing)d(requiremen)o(ts,)j(t)o(yp)q(e)f(con)o(v)o (ersion)h(in)e(hetero-)75 1705 y(geneous)i(en)o(vironmen)o(ts,)g(and)f(more)f (general)i(comm)o(unicati)o(on)c(mo)q(des.)34 b(Non)o(blo)q(c)o(king)18 b(comm)o(unication)e(is)75 1755 y(addressed)h(next,)f(follo)o(w)o(ed)d(b)o(y) i(c)o(hannel-lik)o(e)g(constructs)i(and)e(send-receiv)o(e)i(op)q(erations.)22 b(W)m(e)15 b(then)h(consider)75 1805 y(general)g(datat)o(yp)q(es)g(that)g (allo)o(w)e(one)i(to)g(transfer)g(heterogeneous)i(and)e(noncon)o(tiguous)f (data,)h(and)f(conclude)75 1855 y(with)c(a)h(description)g(of)g(an)f (implemen)o(tation)d(of)k(MPI)g(p)q(oin)o(t)f(to)h(p)q(oin)o(t)f(comm)o (unicatio)o(n)e(using)j(a)f(small)f(n)o(um)o(b)q(er)75 1905 y(of)j(primitiv)o(es.)75 2042 y Fw(1.2)59 b(Basic)19 b(send)h(op)r(eration)75 2133 y Fx(The)14 b(syn)o(tax)g(of)f(the)i(simplest)e(send)i(op)q(eration)e (is)h(giv)o(en)g(b)q(elo)o(w.)75 2230 y Fu(MPI)p 144 2230 14 2 v 15 w(SEND)21 b(\(start,)f(count,)h(datatype,)f(dest,)h(tag,)g(comm\))117 2280 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 2325 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er) 12 b(of)f(elemen)o(ts)h(in)g(send)g(bu\013er)g(\(nonnegativ)o(e)i(in)o (teger\))117 2371 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 2417 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2462 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2508 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))-32 46 y Fs(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-39 554 y(10)-39 611 y(11)-39 667 y(12)-39 724 y(13)-39 780 y(14)-39 836 y(15)-39 893 y(16)-39 949 y(17)-39 1006 y(18)-39 1062 y(19)-39 1119 y(20)-39 1175 y(21)-39 1232 y(22)-39 1288 y(23)-39 1345 y(24)-39 1401 y(25)-39 1457 y(26)-39 1514 y(27)-39 1570 y(28)-39 1627 y(29)-39 1683 y(30)-39 1740 y(31)-39 1796 y(32)-39 1853 y(33)-39 1909 y(34)-39 1966 y(35)-39 2022 y(36)-39 2078 y(37)-39 2135 y(38)-39 2191 y(39)-39 2248 y(40)-39 2304 y(41)-39 2361 y(42)-39 2417 y(43)-39 2474 y(44)-39 2530 y(45)-39 2587 y(46)-39 2643 y(47)-39 2699 y(48)p eop %%Page: 2 2 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(2)75 45 y Fq(1.2.1)49 b(Message)16 b(data)75 122 y Fx(The)e(send)g(bu\013er)g(sp)q(eci\014ed)h(b)o(y)e(the)h Fu(MPI)p 736 122 14 2 v 15 w(SEND)e Fx(op)q(eration)h(consists)h(of)f Fu(count)f Fx(successiv)o(e)k(en)o(tries)e(of)f(the)g(t)o(yp)q(e)75 172 y(indicated)g(b)o(y)g Fu(datatype)p Fx(,)e(starting)i(with)g(the)h(en)o (try)g(at)f(address)h Fu(start)p Fx(.)j(Note)c(that)g(w)o(e)h(sp)q(ecify)g (the)f(message)75 221 y(length)k(in)f(terms)g(of)g(n)o(um)o(b)q(er)g(of)g Fp(elements)p Fx(,)h(not)f(n)o(um)o(b)q(er)g(of)g Fp(bytes)p Fx(.)26 b(The)17 b(former)f(is)g(mac)o(hine)g(indep)q(enden)o(t)75 271 y(and)e(closer)g(to)g(the)h(application)d(lev)o(el.)158 321 y(The)h(data)g(part)g(of)g(the)g(message)g(consists)h(of)e(a)h(sequence)i (of)e Fu(count)f Fx(v)n(alues,)g(eac)o(h)h(of)g(the)g(t)o(yp)q(e)h(indicated) 75 371 y(b)o(y)h Fu(datatype)p Fx(.)k Fu(count)14 b Fx(ma)o(y)f(b)q(e)j (zero,)f(in)g(whic)o(h)f(case)i(the)g(data)e(part)i(of)e(the)h(message)g(is)g (empt)o(y)m(.)k(The)d(basic)75 421 y(datat)o(yp)q(es)h(that)g(can)h(b)q(e)f (sp)q(eci\014ed)i(for)d(message)h(data)f(v)n(alues)h(corresp)q(ond)h(to)f (the)h(basic)f(datat)o(yp)q(es)g(of)f(the)75 470 y(host)h(language.)27 b(The)17 b(p)q(ossible`)g(v)n(alues)g(of)f(this)h(parameter)g(for)g(F)m (ortran)g(and)g(the)g(corresp)q(onding)h(F)m(ortran)75 520 y(t)o(yp)q(es)d(are)f(listed)g(b)q(elo)o(w)p 522 665 906 2 v 521 715 2 50 v 547 700 a(MPI)g(datat)o(yp)q(e)p 951 715 V 175 w(F)m(ortran)g(datat)o(yp)q(e)p 1427 715 V 522 716 906 2 v 521 766 2 50 v 547 751 a Fu(MPI)p 616 751 14 2 v 15 w(INTEGER)p 951 766 2 50 v 195 w Fx(INTEGER)p 1427 766 V 521 816 V 547 801 a Fu(MPI)p 616 801 14 2 v 15 w(REAL)p 951 816 2 50 v 261 w Fx(REAL)p 1427 816 V 521 866 V 547 851 a Fu(MPI)p 616 851 14 2 v 15 w(DOUBLE)p 951 866 2 50 v 217 w Fx(DOUBLE)g(PRECISION)p 1427 866 V 521 916 V 547 901 a Fu(MPI)p 616 901 14 2 v 15 w(COMPLEX)p 951 916 2 50 v 195 w Fx(COMPLEX)p 1427 916 V 521 966 V 547 951 a Fu(MPI)p 616 951 14 2 v 15 w(DOUBLE)p 763 951 V 15 w(COMPLEX)p 951 966 2 50 v 48 w Fx(DOUBLE)g(COMPLEX)p 1427 966 V 521 1015 V 547 1000 a Fu(MPI)p 616 1000 14 2 v 15 w(LOGICAL)p 951 1015 2 50 v 195 w Fx(LOGICAL)p 1427 1015 V 521 1065 V 547 1050 a Fu(MPI)p 616 1050 14 2 v 15 w(CHARACTER)p 951 1065 2 50 v 151 w Fx(CHARA)o(CTER)p 1427 1065 V 521 1115 V 547 1100 a Fu(MPI)p 616 1100 14 2 v 15 w(BYTE)p 951 1115 2 50 v 1427 1115 V 522 1117 906 2 v 75 1301 a Fx(The)g(p)q(ossible)g(v)n(alues)g(for)g(this)g (parameter)f(for)h(C)f(and)h(the)h(corresp)q(onding)f(C)g(t)o(yp)q(es)h(are)f (listed)g(b)q(elo)o(w)p 695 1480 560 2 v 694 1529 2 50 v 720 1515 a(MPI)g(datat)o(yp)q(e)p 1000 1529 V 51 w(C)f(datat)o(yp)q(e)p 1254 1529 V 695 1531 560 2 v 694 1581 2 50 v 720 1566 a Fu(MPI)p 789 1566 14 2 v 15 w(SHORT)p 1000 1581 2 50 v 115 w Fx(short)p 1254 1581 V 694 1631 V 720 1616 a Fu(MPI)p 789 1616 14 2 v 15 w(INT)p 1000 1631 2 50 v 159 w Fx(in)o(t)p 1254 1631 V 694 1681 V 720 1666 a Fu(MPI)p 789 1666 14 2 v 15 w(LONG)p 1000 1681 2 50 v 137 w Fx(long)p 1254 1681 V 694 1730 V 720 1715 a Fu(MPI)p 789 1715 14 2 v 15 w(UNSIGNED)p 1000 1730 2 50 v 49 w Fx(unsigned)p 1254 1730 V 694 1780 V 720 1765 a Fu(MPI)p 789 1765 14 2 v 15 w(FLOAT)p 1000 1780 2 50 v 115 w Fx(\015oat)p 1254 1780 V 694 1830 V 720 1815 a Fu(MPI)p 789 1815 14 2 v 15 w(DOUBLE)p 1000 1830 2 50 v 93 w Fx(double)p 1254 1830 V 694 1880 V 720 1865 a Fu(MPI)p 789 1865 14 2 v 15 w(CHAR)p 1000 1880 2 50 v 137 w Fx(c)o(har)p 1254 1880 V 694 1930 V 720 1915 a Fu(MPI)p 789 1915 14 2 v 15 w(BYTE)p 1000 1930 2 50 v 1254 1930 V 695 1931 560 2 v 158 2087 a Fx(The)h(datat)o(yp)q(e)g Fu(MPI)p 486 2087 14 2 v 15 w(BYTE)f Fx(do)q(es)h(not)g(corresp)q(ond)h(to)e (a)g(F)m(ortran)h(or)f(C)h(datat)o(yp)q(e.)k(A)c(v)n(alue)f(of)g(t)o(yp)q(e)h Fu(BYTE)75 2137 y Fx(consists)20 b(of)e(8)g(binary)g(digits.)32 b(A)19 b(b)o(yte)g(is)g(unin)o(terpreted)h(and)f(is)f(di\013eren)o(t)i(from)d (a)h(c)o(haracter.)34 b(Di\013eren)o(t)75 2186 y(mac)o(hines)19 b(ma)o(y)g(ha)o(v)o(e)h(di\013eren)o(t)h(represen)o(tations)i(for)d(c)o (haracters,)j(or)d(ma)o(y)f(use)i(more)e(than)i(one)f(b)o(yte)h(to)75 2236 y(represen)o(t)16 b(c)o(haracters.)k(On)14 b(the)h(other)f(hand,)f(a)h (b)o(yte)g(has)g(the)h(same)e(binary)g(v)n(alue)g(on)h(all)f(mac)o(hines.)158 2365 y Fo(Discussion:)158 2410 y Ft(Need)g(to)g(decide)h(whether)g(w)o(e)e(w) o(an)o(t)h(\(for)g(C\))158 2456 y(unsigned)i(c)o(har,)e(unsigned)i(short,)f (unsigned)h(long)158 2502 y(long)f(double)158 2547 y(DOUBLE)f(COMPLEX)g(ma)o (y)g(not)g(b)q(e)g(standard)i(F)m(ortran)e(77)158 2593 y(DOUBLE)g(is)h(no)o (w)f(used)g(b)q(oth)h(for)f(F)m(ortran)g(and)h(C.)e(Should)j(w)o(e)e(use)g (di\013eren)o(t)i(names?)p eop %%Page: 3 3 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(3)75 45 y Fq(1.2.2)49 b(Message)16 b(envelop)q(e)75 122 y Fx(In)c(addition)g(to)g(the)h(data)f(part,)g(messages)h(con)o(tain)f (information)d(that)k(can)f(b)q(e)h(used)h(to)e(distinguish)g(messages)75 172 y(and)h(selectiv)o(ely)h(receiv)o(e)h(them.)i(This)c(information)d(is)k (con)o(tained)f(in)g(a)g(\014xed)h(n)o(um)o(b)q(er)e(of)h(\014xed-format)f (\014elds,)75 221 y(whic)o(h)i(w)o(e)g(collectiv)o(ely)f(call)g(the)i Fv(message)g(en)o(v)o(elop)q(e)p Fx(.)h(These)f(\014elds)f(are)75 304 y Fv(source)75 387 y(destinatio)o(n)75 470 y(tag)75 553 y(con)o(text)158 636 y Fx(The)h(message)g(source)h(is)e(implicitly)e (determined)j(b)o(y)f(the)i(iden)o(tit)o(y)e(of)g(the)h(message)g(sender.)22 b(The)15 b(other)75 686 y(\014elds)f(are)h(sp)q(eci\014ed)g(b)o(y)f (parameters)g(in)f(the)i(send)f(op)q(eration.)158 736 y(The)g(in)o(teger-v)n (alued)g(message)f(tag)g(is)h(sp)q(eci\014ed)h(b)o(y)f(the)g Fu(tag)f Fx(parameter.)18 b(This)13 b(in)o(teger)h(can)g(b)q(e)h(used)f(b)o (y)75 786 y(the)f(program)e(to)h(distinguish)g(di\013eren)o(t)i(t)o(yp)q(es)f (of)f(messages.)18 b(The)13 b(range)g(of)e(v)n(alid)g(tag)i(v)n(alues)f(is)g (implemen)o(ta-)75 836 y(tion)g(dep)q(enden)o(t)h(and)f(can)h(b)q(e)f (queried)h(using)f(the)g Fu(TagRange)f Fx(en)o(vironmen)o(tal)f(inquiry)h (function,)h(as)g(describ)q(ed)75 886 y(in)h(Chapter)i Fv(??)p Fx(.)158 935 y(The)i(con)o(text)g(of)f(the)h(message)f(sen)o(t)i(is)e(sp)q (eci\014ed)i(b)o(y)e(the)h Fu(comm)f Fx(parameter.)26 b(the)17 b(message)f(carries)h(the)75 985 y(con)o(text)e(asso)q(ciated)f(with)g(this)g (comm)o(unicator)d(\(see)k(Chapter)g Fv(??)p Fx(\).)158 1035 y(The)f(message)e(destination)h(is)g(sp)q(eci\014ed)i(b)o(y)e(the)h Fu(dest)e Fx(parameter)h(as)g(a)g(rank)g(within)f(the)i(pro)q(cess)h(group)75 1085 y(asso)q(ciated)h(with)f(that)g(same)g(comm)o(unicator.)k(The)d(range)g (of)e(v)n(alid)g(v)n(alues)h(is)g Fu(0,)22 b(...)42 b(,)22 b(n-1)p Fx(,)15 b(where)h Fu(n)f Fx(is)75 1135 y(the)i(n)o(um)o(b)q(er)f(of)h (pro)q(cesses)i(in)d(this)h(group.)27 b(Th)o(us,)17 b(p)q(oin)o(t-to-p)q(oin) o(t)e(comm)o(unications)f(do)j(not)f(use)i(absolute)75 1184 y(addresses,)e(but)e(only)f(relativ)o(e)g(ranks)i(within)e(a)g(group.)18 b(This)c(pro)o(vides)g(imp)q(ortan)o(t)e(mo)q(dularit)o(y)m(.)158 1234 y(The)i(message)g(en)o(v)o(elop)q(e)g(w)o(ould)f(normally)e(b)q(e)j (enco)q(ded)h(b)o(y)e(a)h(\014xed-length)g(message)f(header.)19 b(Ho)o(w)o(ev)o(er,)75 1284 y(the)d(actual)f(mec)o(hanism)e(used)j(to)f(asso) q(ciate)i(an)e(en)o(v)o(elop)q(e)g(with)h(a)f(message)g(is)g(implemen)o (tatio)o(n)e(dep)q(enden)o(t:)75 1334 y(some)k(of)f(the)j(information)14 b(\(e.g.,)j(source)i(or)f(destination\))f(ma)o(y)e(b)q(e)k(implicit,)c(and)i (need)h(not)g(b)q(e)g(explicitly)75 1384 y(carried)d(b)o(y)e(messages;)h(pro) q(cesses)i(ma)o(y)c(b)q(e)j(iden)o(ti\014ed)f(b)o(y)f(relativ)o(e)h(ranks,)g (or)g(absolute)g(ids;)f(etc.)75 1521 y Fw(1.3)59 b(Basic)19 b(receive)g(op)r(eration)75 1612 y Fx(The)14 b(syn)o(tax)g(of)f(the)i (simplest)e(receiv)o(e)i(op)q(eration)f(is)g(giv)o(en)f(b)q(elo)o(w.)75 1709 y Fu(MPI)p 144 1709 14 2 v 15 w(RECV)21 b(\(start,)f(count,)h(datatype,) f(source,)g(tag,)h(comm,)g(status\))117 1759 y Ft(out)14 b Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i(bu\013er)g(\(c)o (hoice\))117 1804 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(receiv)o(e)g(bu\013er)g(\(in)o(teger\))117 1850 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(receiv)o (e)h(bu\013er)g(elemen)o(t)g(\(state\))117 1896 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 1941 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1987 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))117 2033 y(out)g Fu(status)629 b Ft(status)13 b(ob)r(ject)158 2109 y Fx(The)18 b(receiv)o(e)h(bu\013er)g(consists)g(of)e(the)h(storage)g(con)o (taining)f Fu(count)f Fx(consecutiv)o(e)j(elemen)o(ts)f(of)f(the)h(t)o(yp)q (e)75 2159 y(sp)q(eci\014ed)e(b)o(y)e Fu(datatype)p Fx(,)f(starting)h(at)h (address)g Fu(start)p Fx(.)k(The)c(length)g(of)f(the)h(receiv)o(ed)g(message) g(m)o(ust)e(b)q(e)i(less)75 2209 y(then)e(or)f(equal)g(to)g(the)g(length)g (of)g(the)h(receiv)o(e)g(bu\013er.)18 b(I.e.,)12 b(all)f(incoming)f(data)h(m) o(ust)g(\014t,)i(without)e(truncation,)75 2258 y(in)o(to)18 b(the)h(receiv)o(e)g(bu\013er.)33 b(The)19 b Fu(MPI)p 688 2258 V 15 w(PROBE)e Fx(function)h(describ)q(ed)i(in)e(Section)h(1.9)f(can)g(b)q(e) h(used)g(to)g(receiv)o(e)75 2308 y(messages)14 b(of)f(unkno)o(wn)h(length.) 158 2358 y(The)f(selection)g(of)f(a)g(message)h(b)o(y)f(a)g(receiv)o(e)i(op)q (eration)e(is)h(done)g(uniquely)e(according)i(to)f(the)i(v)n(alue)d(of)h(the) 75 2408 y(message)g(en)o(v)o(elop)q(e.)18 b(The)13 b(receiv)o(e)h(op)q (eration)e(sp)q(eci\014es)i(an)e Fv(en)o(v)o(elop)q(e)g(pattern)p Fx(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o(ed)75 2458 y(b)o(y)j(that)g (receiv)o(e)h(op)q(eration)f(only)f(if)g(its)h(en)o(v)o(elop)q(e)g(matc)o (hes)f(that)h(pattern.)25 b(A)16 b(pattern)h(sp)q(eci\014es)h(v)n(alues)d (for)75 2508 y(the)h Fu(source)p Fx(,)e Fu(tag)g Fx(and)h Fu(context)f Fx(\014elds)h(of)g(the)h(message)f(en)o(v)o(elop)q(e.)22 b(The)16 b(receiv)o(er)g(ma)o(y)e(sp)q(ecify)h(a)g(wildcard)75 2557 y Fu(MPI)p 144 2557 V 15 w(ANY)p 225 2557 V 15 w(SOURCE)c Fx(v)n(alue)g(for)g Fu(source)p Fx(,)g(and/or)g(a)h(wildcard)f Fu(MPI)p 1105 2557 V 15 w(ANY)p 1186 2557 V 15 w(TAG)g Fx(v)n(alue)g(for)h Fu(tag)p Fx(,)f(indicating)g(that)g(an)o(y)75 2607 y(source)k(and/or)f(tag)g(are)g (acceptable.)19 b(It)c(cannot)f(sp)q(ecify)g(a)g(wildcard)g(v)n(alue)f(for)h Fu(context)p Fx(.)j(Th)o(us,)d(a)f(message)75 2657 y(can)f(b)q(e)g(receiv)o (ed)i(b)o(y)d(a)h(receiv)o(e)h(op)q(eration)f(only)f(if)g(it)g(is)h (addressed)h(to)f(the)h(receiving)f(pro)q(cess,)h(has)f(a)g(matc)o(hing)p eop %%Page: 4 4 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(4)75 45 y(con)o(text,)13 b(has)f(matc)o(hing)e(source)j (unless)g(source=)p Fu(MPI)p 943 45 14 2 v 17 w(ANY)p 1026 45 V 15 w(SOURCE)e Fx(in)g(the)i(pattern,)f(and)g(has)g(a)g(matc)o(hing)e (tag)75 95 y(unless)15 b(tag=)p Fu(MPI)p 357 95 V 15 w(ANY)p 438 95 V 15 w(TAG)e Fx(in)g(the)i(pattern.)158 145 y(The)20 b(message)g(tag)g(is)f(sp)q(eci\014ed)j(b)o(y)d(the)i Fu(tag)e Fx(parameter)h(of)f(the)i(receiv)o(e)g(op)q(eration.)36 b(The)20 b(message)75 195 y Fu(context)e Fx(is)h(the)h(con)o(text)g(attac)o(hed)g (with)f(the)h(comm)o(unicator)d(sp)q(eci\014ed)k(b)o(y)e(the)h(parameter)f Fu(comm)p Fx(.)34 b(The)75 244 y(message)17 b(source,)h(if)e(di\013eren)o(t)i (from)d Fu(MPI)p 758 244 V 15 w(ANY)p 839 244 V 16 w(SOURCE)p Fx(,)g(is)i(sp)q(eci\014ed)h(as)f(a)g(rank)g(within)f(the)h(pro)q(cess)i (group)75 294 y(asso)q(ciated)14 b(with)f(that)g(same)f(comm)o(unicator.)j (Th)o(us,)e(the)h(range)f(of)g(v)n(alid)e(v)n(alues)i(for)g(the)g Fu(source)f Fx(parameter)75 344 y(is)i Fn(f)21 b Fu(0,)h(...)43 b(,)21 b(n-1)g Fn(g)h([f)f Fu(MPI)p 646 344 V 15 w(ANY)p 727 344 V 15 w(SOURCE)g Fn(g)p Fx(,)13 b(where)i Fu(n)f Fx(is)f(the)i(n)o(um)o(b) q(er)e(of)g(pro)q(cesses)k(in)c(this)h(group.)158 394 y(Note)20 b(the)f(asymmetry)e(b)q(et)o(w)o(een)k(send)f(and)f(receiv)o(e)h(op)q (erations:)29 b(A)19 b(receiv)o(e)i(op)q(eration)e(ma)o(y)e(accept)75 444 y(messages)i(from)f(an)h(arbitrary)g(sender;)24 b(on)19 b(the)h(other)g(hand,)g(a)f(send)h(op)q(eration)g(m)o(ust)e(sp)q(ecify)i(a)f (unique)75 493 y(receiv)o(er.)i(This)15 b(matc)o(hes)f(a)g(\\push")h(comm)o (unicatio)o(n)d(mec)o(hanism,)g(where)j(data)g(transfer)g(is)f(e\013ected)j (b)o(y)d(the)75 543 y(sender)h(\(rather)g(than)f(a)g(\\pull")e(mec)o(hanism,) f(where)16 b(data)d(transfer)i(is)f(e\013ected)i(b)o(y)d(the)i(receiv)o (er\).)158 593 y(Source)g(=)f(destination)g(is)f(allo)o(w)o(ed:)k(a)d(pro)q (cess)i(can)e(send)g(a)g(message)g(to)f(itself.)75 709 y Fq(1.3.1)49 b(Return)16 b(status)75 869 y Fo(Discussion:)i(This)d(is)g(a)g(c)o(hange)g (from)e(previous)j(draft)158 1001 y Fx(The)d(source)h(or)f(the)g(tag)f(of)h (a)f(receiv)o(ed)i(message)e(ma)o(y)f(not)i(b)q(e)g(kno)o(wn)f(if)g(wildcard) h(v)n(alues)f(where)i(used)f(in)75 1051 y(the)g(receiv)o(e)g(op)q(eration.)k (Also,)12 b(the)g(actual)g(length)g(of)f(the)i(message)f(receiv)o(ed)h(ma)o (y)d(not)i(b)q(e)h(kno)o(wn.)k(Th)o(us,)12 b(this)75 1101 y(information)e (needs)k(to)f(b)q(e)h(returned)h(b)o(y)d(the)i(receiv)o(e)g(op)q(eration.)k (F)m(or)12 b(reasons)i(that)g(will)d(b)q(ecome)i(clear)g(later,)75 1150 y(it)i(is)h(preferable)g(to)f(return)i(this)f(information)c(in)j(a)h (separate)g Fu(out)f Fx(v)n(ariable,)g(rather)h(then)g(using)f(the)i Fu(count)p Fx(,)75 1200 y Fu(source)12 b Fx(and)i Fu(tag)e Fx(parameters)i(b)q(oth)g(for)f(input)g(and)h(output.)k(The)c(information)c (is)k(returned)h(b)o(y)e(the)h Fu(status)75 1250 y Fx(parameter)f(of)g(the)h Fu(MPI)p 460 1250 V 15 w(RECV)e Fx(function.)18 b(This)13 b(is)h(a)f (parameter)g(of)g(a)g(sp)q(ecial)g(MPI-de\014ned)i(t)o(yp)q(e.)j(The)c (status)75 1300 y(v)n(ariable)g(can)h(b)q(e)h(\\deco)q(ded")g(to)f(retriev)o (e)i(the)f Fu(count)p Fx(,)e Fu(source)f Fx(and)i Fu(tag)g Fx(\014elds,)g(using)g(the)h(query)g(functions)75 1350 y(listed)e(b)q(elo)o (w.)75 1447 y Fu(MPI)p 144 1447 V 15 w(GET)p 225 1447 V 15 w(SOURCE\(status,)19 b(source\))117 1497 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i(op)q(eration)h(\(Status\))117 1542 y(out)f Fu(source)629 b Ft(source)13 b(rank)h(\(in)o(teger\))158 1619 y Fx(Returns)i(the)g(rank)g(of)f(the)h(message)f(source)h(\(in)g(the)g (group)f(asso)q(ciated)h(with)f(the)h(comm)o(unicator)d(that)75 1668 y(w)o(as)h(used)h(to)e(receiv)o(e\).)75 1765 y Fu(MPI)p 144 1765 V 15 w(GET)p 225 1765 V 15 w(TAG\(status,)20 b(tag\))117 1815 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i (op)q(eration)h(\(Status\))117 1861 y(out)f Fu(tag)695 b Ft(message)13 b(tag)h(\(in)o(teger\))158 1937 y Fx(Returns)h(the)f(tag)g(of)f(receiv)o(ed)i (message.)75 2034 y Fu(MPI)p 144 2034 V 15 w(GET)p 225 2034 V 15 w(COUNT\(status,)k(count\))117 2084 y Ft(in)14 b Fu(status)653 b Ft(return)13 b(status)h(of)e(receiv)o(e)i(op)q(eration)h(\(Status\))117 2130 y(out)f Fu(count)651 b Ft(n)o(um)o(b)q(er)14 b(of)e(receiv)o(ed)j (elemen)o(ts)f(\(in)o(teger\))158 2206 y Fx(Returns)h(the)f(n)o(um)o(b)q(er)f (of)h(elemen)o(ts)f(receiv)o(ed.)20 b(\(Here,)15 b(again,)d(w)o(e)i(coun)o(t) g Fp(elements)p Fx(,)f(not)h Fp(bytes)p Fx(.\))158 2256 y(The)f(information)e (returned)j(b)o(y)f(these)h(query)g(functions)f(is)g(the)g(information)d (last)j(stored)h(in)f(the)g Fu(status)75 2306 y Fx(v)n(ariable)i(b)o(y)h(a)g (receiv)o(e)h(function.)24 b(It)16 b(is)g(erroneous)i(to)e(call)f(these)j (query)e(functions)g(if)g(the)g Fu(status)f Fx(v)n(ariable)75 2355 y(w)o(as)f(nev)o(er)h(set)f(b)o(y)g(a)g(receiv)o(e.)158 2405 y(Note)h(that)g(it)f(is)g(not)g(mandatory)f(to)i(query)g(the)g(return)h (status)f(after)g(a)f(receiv)o(e.)21 b(The)15 b(receiv)o(er)h(will)d(use)75 2455 y(these)i(calls,)e(or)h(some)f(of)g(them,)g(only)g(when)i(the)f (information)d(they)j(return)h(is)f(needed.)158 2505 y(information)d (returned)16 b(in)d(it)h(b)o(y)f(the)i(last)e(receiv)o(e)i(op)q(eration)f (that)g(up)q(dated)158 2633 y Fo(Discussion:)p eop %%Page: 5 5 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(5)158 45 y Ft(The)13 b(use)g(of)g(a)g(separate)h(status)f (parameter)h(prev)o(en)o(ts)g(errors)f(that)g(are)g(often)g(attac)o(hed)h (with)f Fm(inout)e Ft(parameters)75 91 y(\(e.g.,)h(passing)k(the)d Fm(MPI)p 435 91 12 2 v 13 w(ANY)p 508 91 V 13 w(TAG)f Ft(constan)o(t)i(as)g (the)f(actual)i(argumen)o(t)f(for)f Fm(tag)p Ft(\).)k(The)c(use)h(of)f(an)h (explicit)i(user)e(ob)r(ject)75 136 y(b)q(ecomes)j(imp)q(ortan)o(t)g(for)e (non)o(blo)q(c)o(king)k(comm)o(unication)g(as)d(it)g(allo)o(ws)h(the)f (receiv)o(e)g(op)q(eration)i(to)e(b)q(e)g(stateless)g(and,)75 182 y(hence,)d(reen)o(tran)o(t.)18 b(This)c(prev)o(en)o(ts)f(confusions)i(in) f(the)f(case)h(where)f(m)o(ultiple)i(receiv)o(es)f(can)g(b)q(e)f(p)q(osted)h (b)o(y)f(a)g(pro)q(cess.)158 393 y Fo(Implemen)o(tation)g(note:)158 443 y Ft(One)g(exp)q(ects)h(deco)q(de)g(functions)g(to)f(b)q(e)h(in-lined)h (in)f(man)o(y)g(implemen)o(tations.)158 575 y Fx(All)f(send)h(and)g(receiv)o (e)h(op)q(erations)e(use)i(the)f Fu(start)p Fx(,)e Fu(count)p Fx(,)g Fu(datatype)p Fx(,)g Fu(source)p Fx(,)g Fu(dest)p Fx(,)g Fu(tag)p Fx(,)g Fu(comm)h Fx(and)75 625 y Fu(status)i Fx(parameters)i(in)g (the)g(same)f(w)o(a)o(y)g(as)h(the)g(basic)g Fu(MPI)p 1064 625 14 2 v 15 w(SEND)f Fx(and)h Fu(MPI)p 1333 625 V 15 w(RECV)f Fx(op)q(erations)h(describ)q(ed)h(in)75 675 y(this)c(section.)75 812 y Fw(1.4)59 b(Semantic)o(s)17 b(of)j(p)r(oint-to-p)r(oint)e(comm)m (unicat)o(ion)75 903 y Fx(A)12 b(v)n(alid)e(MPI)i(implem)o(en)o(tation)d (guaran)o(tees)j(certain)h(general)e(prop)q(erties)j(of)d(p)q(oin)o(t-to-p)q (oin)o(t)f(comm)o(unicatio)o(n,)75 953 y(whic)o(h)k(are)g(describ)q(ed)i(in)d (this)h(section.)75 1061 y Fl(Order)42 b Fx(Messages)13 b(are)f Fp(non-overtaking)p Fx(,)g(within)f(eac)o(h)h(con)o(text:)18 b(if)10 b(t)o(w)o(o)h(messages)h(are)f(successiv)o(ely)i(sen)o(t)g(from)75 1111 y(the)h(same)f(source,)i(to)f(the)g(same)f(destination,)g(with)h(the)g (same)f(con)o(text,)h(then)h(the)f(messages)g(are)g(receiv)o(ed)h(in)75 1161 y(the)h(order)h(they)f(w)o(ere)h(sen)o(t.)25 b(I.e.,)15 b(if)g(the)i(receiv)o(er)g(p)q(osts)g(t)o(w)o(o)e(successiv)o(e)j(receiv)o (es)g(that)d(b)q(oth)h(matc)o(h)f(either)75 1210 y(message,)k(then)h(the)f (\014rst)g(receiv)o(e)h(will)e(receiv)o(e)i(the)f(\014rst)h(message,)f(and)f (the)i(second)f(receiv)o(e)h(will)e(receiv)o(e)75 1260 y(the)e(second)g (message.)21 b(This)14 b(requiremen)o(t)h(facilitates)g(matc)o(hing)e(of)h (sends)i(to)f(receiv)o(es.)23 b(It)15 b(guaran)o(tees)h(that)75 1310 y(message-passing)11 b(co)q(de)i(is)e(deterministic,)g(if)g(pro)q (cesses)k(are)d(single-threaded)g(and)g(wildcard)f Fu(MPI)p 1651 1310 V 15 w(ANY)p 1732 1310 V 15 w(SOURCE)75 1360 y Fx(is)j(not)g(used)g (in)g(receiv)o(es.)158 1410 y(If)i(a)h(pro)q(cess)h(has)f(a)f(single)h (thread)g(of)f(execution,)i(then)f(an)o(y)f(t)o(w)o(o)g(comm)o(unications)e (executed)k(b)o(y)f(this)75 1459 y(pro)q(cess)j(are)f(ordered.)32 b(On)19 b(the)g(other)g(hand,)g(if)e(the)i(pro)q(cess)h(is)e(m)o (ulti-threaded,)g(then)h(the)g(seman)o(tics)f(of)75 1509 y(thread)i (execution)g(ma)o(y)e(not)h(de\014ne)i(a)e(relativ)o(e)g(order)i(b)q(et)o(w)o (een)g(t)o(w)o(o)e(send)h(op)q(erations)g(executed)h(b)o(y)e(t)o(w)o(o)75 1559 y(distinct)e(threads:)24 b(the)17 b(op)q(erations)g(are)g(logically)d (concurren)o(t,)k(ev)o(en)f(though)f(one)h(ph)o(ysically)e(precedes)k(the)75 1609 y(other.)i(In)14 b(suc)o(h)i(a)e(case,)h(the)h(t)o(w)o(o)e(messages)g (sen)o(t)i(can)f(b)q(e)g(receiv)o(ed)h(in)e(an)o(y)g(order.)21 b(Similarly)l(,)12 b(if)i(t)o(w)o(o)g(receiv)o(e)75 1659 y(op)q(erations)c (that)g(are)g(logically)e(concurren)o(t)j(receiv)o(e)h(t)o(w)o(o)d(successiv) o(ely)j(sen)o(t)e(messages,)g(then)h(the)f(t)o(w)o(o)g(messages)75 1709 y(can)k(matc)o(h)f(the)h(t)o(w)o(o)g(receiv)o(es)h(in)f(either)g(order.) 75 1816 y Fl(Progress)41 b Fx(If)12 b(a)h(pair)f(of)g(matc)o(hing)e(send)k (and)e(receiv)o(es)i(ha)o(v)o(e)f(b)q(een)g(initiated)f(on)g(t)o(w)o(o)g(pro) q(cesses,)j(then)e(at)g(least)75 1866 y(one)j(of)f(these)j(t)o(w)o(o)d(op)q (erations)h(will)f(complete,)g(indep)q(enden)o(tly)i(of)e(other)i(actions)f (in)f(the)i(system:)22 b(the)16 b(send)75 1916 y(op)q(eration)11 b(will)e(complete,)h(unless)i(the)f(receiv)o(e)h(is)f(satis\014ed)g(b)o(y)g (another)g(message,)g(and)f(completes;)h(the)h(receiv)o(e)75 1966 y(op)q(eration)h(will)f(complete,)g(unless)i(the)g(message)f(sen)o(t)i (is)e(consumed)g(b)o(y)g(another)h(matc)o(hing)d(receiv)o(e)k(that)e(w)o(as) 75 2016 y(p)q(osted)i(at)f(the)g(same)f(destination)h(pro)q(cess.)75 2124 y Fl(F)o(airness)42 b Fx(MPI)16 b(mak)o(es)f(no)h(guaran)o(tee)h(of)e Fp(fairness)h Fx(in)g(the)g(handling)f(of)h(comm)o(unicati)o(on.)22 b(Supp)q(ose)17 b(that)f(a)75 2173 y(send)c(w)o(as)f(p)q(osted.)17 b(Then)12 b(it)e(is)h(p)q(ossible)g(that)g(the)g(destination)g(rep)q(eatedly) h(p)q(osts)g(a)e(receiv)o(e)j(that)d(matc)o(hes)h(this)75 2223 y(send,)k(y)o(et)f(the)h(message)e(is)h(nev)o(er)h(receiv)o(ed,)g(b)q(ecause) h(it)e(is)g(eac)o(h)g(time)f(o)o(v)o(ertak)o(en)h(b)o(y)g(another)h(message,) e(sen)o(t)75 2273 y(from)d(another)j(source.)19 b(Similarl)o(y)m(,)9 b(supp)q(ose)k(that)f(a)g(receiv)o(e)i(w)o(as)e(p)q(osted.)18 b(Then)13 b(it)e(is)h(p)q(ossible)h(that)f(messages)75 2323 y(that)20 b(matc)o(h)g(this)g(receiv)o(e)i(are)e(rep)q(eatedly)i(receiv)o (ed,)h(y)o(et)e(the)g(receiv)o(e)g(is)f(nev)o(er)i(satis\014ed,)g(b)q(ecause) g(it)e(is)75 2373 y(o)o(v)o(ertak)o(en)12 b(b)o(y)g(other)g(receiv)o(es)i(p)q (osted)f(at)f(this)g(no)q(de)g(\(b)o(y)g(other)h(executing)f(threads\).)19 b(It)12 b(is)g(the)g(programmer's)75 2422 y(resp)q(onsibilit)o(y)h(to)h(prev) o(en)o(t)h(starv)n(ation)e(in)h(suc)o(h)g(situations.)158 2551 y Fo(Discussion:)158 2596 y(Need)g(to)h(b)q(e)g(discussed)g(b)o(y)g(MPIF)158 2642 y Ft(Alternativ)o(e)g(p)q(ositions:)p eop %%Page: 6 6 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(6)158 45 y Ft(1.)25 b(Sends)17 b(are)f(handled)i(fairly;)g (e.g.)25 b(a)16 b(p)q(osted)g(send)h(cannot)f(b)q(e)g(rep)q(eatedly)i(o)o(v)o (ertak)o(en)f(b)o(y)f(other)g(sends.)26 b(An)75 91 y(example)16 b(of)f(an)g(implemen)o(tation)k(that)c(violates)i(this:)k(Assume)16 b(that)f(arriving)i(messages)f(are)f(k)o(ept)g(in)h(a)f(hash)h(table)75 136 y(organized)d(b)o(y)e(source,)h(and)f(that)h(a)f(receiv)o(e)h(with)f (source=don)o(tcare)i(is)e(handled)i(b)o(y)e(searc)o(hing)i(this)f(table)g (sequen)o(tially)n(.)75 182 y(Then)j(a)f(message)h(p)q(osted)h(at)e(the)h (end)g(of)f(this)h(table)h(ma)o(y)e(b)q(e)h(rep)q(eatedly)h(o)o(v)o(ertak)o (en)g(b)o(y)f(messages)g(that)f(get)h(p)q(osted)75 228 y(at)e(the)g(head)h (of)f(the)g(table.)18 b(\(Easy)13 b(\014x:)k(rotate)d(starting)g(p)q(oin)o(t) g(of)f(searc)o(h.\))158 273 y(2.)18 b(Receiv)o(es)d(are)e(handled)j(fairly;)e (e.g.)k(a)13 b(p)q(osted)i(receiv)o(e)f(cannot)g(b)q(e)g(rep)q(eatedly)h(o)o (v)o(ertak)o(en)g(b)o(y)e(other)h(receiv)o(es.)75 319 y(An)j(example)i(of)d (an)i(implemen)o(tation)i(that)e(violates)h(this:)26 b(assume)18 b(that)f(there)g(are)g(t)o(w)o(o)g(di\013eren)o(t)i(p)q(osting)g(tables)75 365 y(for)e(receiv)o(es)h(with)f(a)g(sp)q(eci\014c)i(source)f(and)f(for)g (receiv)o(es)h(with)g(source=don)o(tcare;)i(assume)d(further)h(that)f(these)g (t)o(w)o(o)75 410 y(structures)i(are)f(searc)o(hed)h(in)g(a)f(\014xed)h (order)g(when)f(a)h(message)f(arriv)o(es.)34 b(Then)18 b(a)g(receiv)o(e)i (with)e(source=don)o(tcare)75 456 y(ma)o(y)e(b)q(e)f(rep)q(eatedly)i(o)o(v)o (ertak)o(en)g(b)o(y)e(receiv)o(es)i(with)f(source)g(sp)q(eci\014ed.)26 b(\(Fix:)c(alternate)17 b(searc)o(h)f(order;)g(but)g(this)g(has)75 502 y(p)q(erformance)e(problems,)g(if)f(w)o(e)g(assume)h(that)f(one)g(t)o(yp) q(e)h(of)e(receiv)o(e)i(is)g(m)o(uc)o(h)f(more)g(frequen)o(t)h(than)g(the)f (other)g(t)o(yp)q(e.\))158 547 y(3.)k(Both)c(send)h(and)g(receiv)o(es)g(are)f (handled)i(fairly)m(.)158 597 y(Opinions?)75 786 y Fl(Resource)20 b(limi)o(tations)39 b Fx(The)19 b(curren)o(t)g(practice)h(for)e(man)o(y)e (commercial)g(message-passing)i(libraries)g(is)g(that)75 836 y(\(short\))g(messages)g(are)g(bu\013ered)h(b)o(y)f(the)g(system,)g(th)o(us)g (allo)o(wing)d(blo)q(c)o(king)i(send)h(op)q(erations)g(to)g(complete)75 886 y(ahead)h(of)f(the)h(matc)o(hing)e(receiv)o(es.)34 b(It)18 b(is)h(exp)q(ected)h(that)f(man)o(y)e(MPI)i(implem)o(en)o(tations)d(will)h (follo)o(w)g(this)75 935 y(practice,)22 b(and)d(pro)o(vide)h(the)g(same)f (lev)o(el)g(of)g(bu\013ering)h(that)g(is)g(a)o(v)n(ailable)d(on)j(the)g (libraries)f(they)i(replace.)75 985 y(Ho)o(w)o(ev)o(er,)e(message)g (bu\013ering)f(is)h(not)f(a)g(univ)o(ersal)g(practice.)33 b(Ev)o(en)18 b(on)h(systems)f(where)i(bu\013ering)e(o)q(ccur,)75 1035 y(the)h(amoun)o(t)e (of)h(bu\013er)h(space)h(a)o(v)n(ailable)c(and)i(the)i(w)o(a)o(y)d(it)i(is)f (allo)q(cated)g(is)g(b)q(ound)h(to)f(b)q(e)h(implemen)o(tation)75 1085 y(dep)q(enden)o(t.)158 1135 y(Therefore,)12 b(message)f(bu\013ering)g (is)f(not)h(mandated)f(b)o(y)g(MPI)h(and)g(is)g(seen)h(as)e(a)h(qualit)o(y)e (of)h(implemen)o(tation)75 1184 y(issue.)20 b(A)14 b(v)n(alid)e(MPI)j (implemen)o(tatio)o(n)d(of)h Fu(MPI)p 846 1184 14 2 v 15 w(SEND)g Fx(is)h(to)g(blo)q(c)o(k)g(the)h(sender)h(un)o(til)d(a)h(matc)o(hing)e (receiv)o(e)k(has)75 1234 y(b)q(een)h(initiated.)24 b(In)16 b(general,)g(the)h(programmer)c(can)k(mak)o(e)d(no)i(assumptions)f(on)h(the)g (a)o(v)n(ailabilit)o(y)d(of)i(bu\013er)75 1284 y(space,)21 b(and)e(ho)o(w)f(this)h(space)h(is)f(allo)q(cated.)33 b(Th)o(us,)20 b(p)q(ortable)f(\(safe\))g(MPI)g(co)q(de)h(should)f(w)o(ork)g(under)g(the)75 1334 y(assumption)14 b(that)i(an)g(arbitrary)g(subset)h(of)e(the)h(send)h(op) q(erations)f(are)g(going)f(to)h(return)h(b)q(efore)f(a)g(matc)o(hing)75 1384 y(receiv)o(e)f(is)f(p)q(osted,)g(and)g(the)g(rest)i(will)c(blo)q(c)o(k)i (un)o(til)f(a)g(matc)o(hing)f(receiv)o(e)j(is)f(p)q(osted.)158 1434 y(MPI)i(implemen)o(tatio)o(n)d(will)h(pro)o(vide)h(information)e(on)i (the)i(amoun)o(t)c(of)i(a)o(v)n(ailable)f(bu\013er)i(space)h(and)e(on)75 1483 y(the)f(bu\013ering)h(p)q(olicy)e(via)g(the)h(en)o(vironmen)o(tal)f (enquiries)h(describ)q(ed)i(in)d(Chapter)i Fv(??)p Fx(.)158 1533 y(Examples)e(\(in)o(v)o(olving)f(t)o(w)o(o)h(pro)q(cessors)j(with)e (ranks)g(0)f(and)h(1\))158 1583 y(The)g(follo)o(wing)e(program)g(is)i(safe,)f (and)h(should)g(alw)o(a)o(ys)f(succeed.)75 1707 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1757 y(IF)g(\(rank.EQ.0\))119 1807 y(THEN)162 1857 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)j(MPI_REAL,)f(1,)h (tag,)g(comm,)g(ierr\))162 1907 y(CALL)g(MPI_RECV\(recvbuf,)d(count,)j (MPI_REAL,)f(1,)h(tag,)g(comm,)g(ierr\))119 1957 y(ELSE)86 b(!)22 b(rank.EQ.1)162 2006 y(CALL)f(MPI_RECV\(recvbuf,)d(count,)j(MPI_REAL,) f(0,)h(tag,)g(comm,)g(ierr\))162 2056 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)j (MPI_REAL,)f(0,)h(tag,)g(comm,)g(context\))75 2106 y(END)g(IF)158 2181 y Fx(The)14 b(follo)o(wing)e(program)g(is)i(erroneous,)g(and)g(should)g (alw)o(a)o(ys)f(fail.)75 2305 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,) i(ierr\))75 2355 y(IF)g(\(rank.EQ.0\))119 2405 y(THEN)162 2455 y(CALL)g(MPI_RECV\(recvbuf,)d(count,)j(MPI_REAL,)f(1,)h(tag,)g(comm,)g (ierr\))162 2504 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)j(MPI_REAL,)f(1,)h (tag,)g(comm,)g(ierr\))119 2554 y(ELSE)86 b(!)22 b(rank.EQ.1)162 2604 y(CALL)f(MPI_RECV\(recvbuf,)d(count,)j(MPI_REAL,)f(0,)h(tag,)g(comm,)g (ierr\))162 2654 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)j(MPI_REAL,)f(0,)h (tag,)g(comm,)g(ierr\))75 2704 y(END)g(IF)p eop %%Page: 7 7 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(7)158 45 y(The)13 b(receiv)o(e)h(op)q(eration)f(of)f(the)i (\014rst)f(pro)q(cess)i(m)o(ust)d(complete)g(b)q(efore)i(its)e(send,)i(and)f (can)g(complete)f(only)75 95 y(if)g(the)h(matc)o(hing)d(send)k(of)e(the)h (second)g(pro)q(cessor)i(is)d(executed;)i(the)f(receiv)o(e)h(op)q(eration)f (of)e(the)j(second)f(pro)q(cess)75 145 y(m)o(ust)j(complete)h(b)q(efore)h (its)g(send)g(and)f(can)h(complete)e(only)h(if)f(the)i(matc)o(hing)e(send)i (of)f(the)h(\014rst)g(pro)q(cess)h(is)75 195 y(executed.)h(This)14 b(program)e(will)g(deadlo)q(c)o(k.)158 244 y(The)i(follo)o(wing)e(program)g (is)i(unsafe,)f(and)h(ma)o(y)e(succeed)k(or)e(fail,)e(dep)q(ending)i(on)g (implemen)o(tatio)o(n.)75 374 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,) i(ierr\))75 424 y(IF)g(\(rank.EQ.0\))119 474 y(THEN)162 524 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)j(MPI_REAL,)f(1,)h(tag,)g(comm,)g (ierr\))162 574 y(CALL)g(MPI_RECV\(recvbuf,)d(count,)j(MPI_REAL,)f(1,)h(tag,) g(comm,)g(ierr\))119 623 y(ELSE)86 b(!)22 b(rank.EQ.1)162 673 y(CALL)f(MPI_SEND\(sendbuf,)d(count,)j(MPI_REAL,)f(0,)h(tag,)g(comm,)g (ierr\))162 723 y(CALL)g(MPI_RECV\(recvbuf,)d(count,)j(MPI_REAL,)f(0,)h(tag,) g(comm,)g(ierr\))75 773 y(END)g(IF)158 903 y Fx(The)15 b(message)g(sen)o(t)h (b)o(y)f(eac)o(h)g(pro)q(cess)i(has)e(to)g(b)q(e)h(copied)f(out)g(b)q(efore)h (the)g(send)g(op)q(eration)f(returns)h(and)75 952 y(the)c(receiv)o(e)h(op)q (eration)e(starts.)19 b(F)m(or)11 b(the)h(program)e(to)h(complete,)g(it)h(is) f(necessary)j(that)d(at)h(least)f(one)h(of)f(the)h(t)o(w)o(o)75 1002 y(messages)k(sen)o(t)i(is)e(bu\013ered.)27 b(Th)o(us,)17 b(this)f(program)f(can)i(succeed)h(only)e(if)g(the)h(comm)o(unicati)o(on)d (system)i(has)75 1052 y(su\016cien)o(t)d(bu\013er)g(space)g(to)g(bu\013er)g Fu(count)e Fx(w)o(ords)i(of)e(data,)h(and)g(the)h(MPI)g(implemen)o(tatio)o(n) d(bu\013ers)j(messages.)75 1187 y Fw(1.5)59 b(Data)19 b(T)-5 b(yp)r(e)19 b(Matching)75 1278 y Fx(One)c(can)f(think)f(of)h(message)f (transmission)g(as)h(consisting)g(of)f(three)i(phases:)126 1352 y(1.)20 b(Data)13 b(is)h(pulled)f(out)h(of)f(the)i(send)g(bu\013er)f (and)g(a)g(message)f(is)h(assem)o(bled)126 1431 y(2.)20 b(A)14 b(message)f(is)h(transferred)i(from)c(sender)j(to)f(receiv)o(er)126 1509 y(3.)20 b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming)c(message)j(and) g(disassem)o(bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 1583 y(T)o(yp)q(e)g(matc)o(hing)e(has)i(to)f(b)q(e)i(observ)o(ed)f(at)g(eac)o (h)g(of)f(these)i(three)g(phases:)21 b(The)15 b(t)o(yp)q(e)h(of)e(eac)o(h)h (v)n(ariable)e(in)75 1633 y(the)18 b(sender)h(bu\013er)g(has)e(to)h(matc)o(h) e(the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(for)e(that)h(en)o(try)g(b)o(y)f(the)h (send)h(op)q(eration;)f(the)g(t)o(yp)q(e)75 1683 y(sp)q(eci\014ed)g(b)o(y)e (the)h(send)g(op)q(eration)f(has)g(to)g(matc)o(h)g(the)g(t)o(yp)q(e)h(sp)q (eci\014ed)h(b)o(y)e(the)h(receiv)o(e)g(op)q(eration;)g(and)f(the)75 1733 y(t)o(yp)q(e)j(of)e(eac)o(h)i(v)n(ariable)e(in)g(the)i(receiv)o(e)g (bu\013er)g(has)f(to)g(matc)o(h)f(the)i(t)o(yp)q(e)f(sp)q(eci\014ed)i(for)e (that)g(en)o(try)h(b)o(y)e(the)75 1783 y(receiv)o(e)e(op)q(eration.)j(A)c (program)e(that)i(fails)f(to)h(observ)o(e)h(these)g(three)g(rules)g(is)e (erroneous.)158 1833 y(T)m(o)j(de\014ne)j(t)o(yp)q(e)e(matc)o(hing)f(more)g (precisely)m(,)i(w)o(e)g(need)g(to)f(deal)g(with)g(t)o(w)o(o)f(issues:)26 b(matc)o(hing)15 b(of)i(t)o(yp)q(es)75 1882 y(of)f(the)h(host)g(language)f (with)g(t)o(yp)q(es)h(sp)q(eci\014ed)i(in)d(comm)o(unicati)o(on)e(op)q (erations;)k(and)e(matc)o(hing)f(of)h(t)o(yp)q(es)h(at)75 1932 y(sender)e(and)f(receiv)o(er.)158 1982 y(A)k(t)o(yp)q(e)h(sp)q(eci\014ed)h (for)d(an)h(en)o(try)h(b)o(y)f(a)g(send)h(op)q(eration)f(matc)o(hes)f(the)i (t)o(yp)q(e)g(sp)q(eci\014ed)g(for)f(that)g(en)o(try)75 2032 y(b)o(y)d(a)f(receiv)o(e)i(op)q(eration)f(if)f(b)q(oth)h(op)q(erations)g(use) h(iden)o(tical)e(names:)19 b Fu(MPI)p 1289 2032 14 2 v 15 w(INTEGER)13 b Fx(matc)o(hes)i Fu(MPI)p 1699 2032 V 15 w(INTEGER)p Fx(,)75 2082 y Fu(MPI)p 144 2082 V 15 w(REAL)e Fx(matc)o(hes)h Fu(MPI)p 487 2082 V 15 w(REAL)p Fx(,)e(and)i(so)g(on.)158 2131 y(The)19 b(t)o(yp)q(e)g(of)g(a)f(v)n(ariable)g(in)g(a)g(host)h(program)f(matc)o(hes)g (the)h(t)o(yp)q(e)h(sp)q(eci\014ed)g(in)e(the)h(comm)o(unication)75 2181 y(op)q(eration)e(if)f(the)i(datat)o(yp)q(e)f(name)e(used)j(b)o(y)f(that) g(op)q(eration)g(corresp)q(onds)i(to)e(the)g(basic)g(t)o(yp)q(e)h(of)e(the)i (host)75 2231 y(program)8 b(v)n(ariable:)15 b(an)9 b(en)o(try)i(with)e(t)o (yp)q(e)h(name)f Fu(MPI)p 920 2231 V 15 w(INTEGER)f Fx(matc)o(hes)h(a)h(F)m (ortran)f(v)n(ariable)g(of)g(t)o(yp)q(e)h Fu(INTEGER)p Fx(,)75 2281 y(an)i(en)o(try)h(with)e(t)o(yp)q(e)i(name)e Fu(MPI)p 600 2281 V 15 w(REAL)g Fx(matc)o(hes)h(a)g(F)m(ortran)f(v)n(ariable)g(of)h(t) o(yp)q(e)h Fu(REAL)p Fx(,)d(and)i(so)g(on.)18 b(There)13 b(is)f(one)75 2331 y(exception)17 b(to)f(this)g(last)f(rule:)23 b(An)16 b(en)o(try)g(with)g (t)o(yp)q(e)h(name)d Fu(MPI)p 1137 2331 V 15 w(BYTE)i Fx(can)g(b)q(e)g(used)h (to)f(matc)o(h)f(an)o(y)g(b)o(yte)i(of)75 2380 y(storage)f(\(on)f(a)g(b)o (yte-addressable)h(mac)o(hine\),)e(irresp)q(ectiv)o(e)j(of)d(the)i(datat)o (yp)q(e)f(of)g(the)h(v)n(ariable)e(that)h(con)o(tains)75 2430 y(this)j(b)o(yte.)30 b(The)18 b(v)n(alue)g(of)f(the)h(message)g(en)o(try)g (will)e(b)q(e)j(the)f(binary)g(v)n(alue)f(of)g(the)h(corresp)q(onding)h(b)o (yte)f(in)75 2480 y(memory)m(.)158 2530 y(W)m(e)c(th)o(us)g(ha)o(v)o(e)g(t)o (w)o(o)f(cases:)137 2604 y Fn(\017)21 b Fx(Comm)o(uni)o(cation)d(of)j(t)o(yp) q(ed)h(v)n(alues)g(\(e.g.,)g(with)f(datat)o(yp)q(e)h(di\013eren)o(t)g(from)e Fu(MPI)p 1541 2604 V 15 w(BYTE)p Fx(\),)h(where)h(the)179 2654 y(datat)o(yp)q(es)15 b(of)g(the)g(corresp)q(onding)h(en)o(tries)g(in)e(the)i (sender)g(program,)d(in)i(the)g(send)h(call,)e(in)g(the)i(receiv)o(e)179 2704 y(call)d(and)h(in)f(the)i(receiv)o(er)g(program)d(should)i(all)f(matc)o (h.)p eop %%Page: 8 8 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(8)137 45 y Fn(\017)21 b Fx(Comm)o(uni)o(cation)15 b(of)i(un)o(t)o(yp)q(ed)i(v)n(alues)f(\(e.g.,)g(of)f(datat)o(yp)q(e)i Fu(MPI)p 1237 45 14 2 v 15 w(BYTE)p Fx(\),)d(where)k(b)q(oth)e(sender)i(and)d (re-)179 95 y(ceiv)o(er)e(use)h(the)f(datat)o(yp)q(e)g Fu(MPI)p 686 95 V 15 w(BYTE)p Fx(.)e(In)h(this)h(case,)g(there)h(are)f(no)g (requiremen)o(ts)f(on)h(the)g(t)o(yp)q(es)g(of)f(the)179 145 y(corresp)q(onding)h(en)o(tries)g(in)e(the)i(sender)h(and)e(the)g(receiv)o (er)i(programs,)c(nor)i(is)g(it)g(required)h(that)f(they)g(b)q(e)179 195 y(the)g(same.)158 275 y(Examples:)158 325 y(1st)g(program:)75 406 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 456 y(IF\(rank.EQ.0\))119 506 y(THEN)162 555 y(CALL)g(MPI_SEND\(a\(1\),)e (10,)i(MPI_REAL,)f(1,)i(tag,)e(comm,)h(ierr\))119 605 y(ELSE)162 655 y(CALL)g(MPI_RECV\(a\(1\),)e(15,)i(MPI_REAL,)f(0,)i(tag,)e(comm,)h (ierr\))75 705 y(END)g(IF)158 786 y Fx(This)12 b(co)q(de)i(is)e(correct)i(if) e(b)q(oth)g(sender)j(and)d(receiv)o(er)i(programs)d(ha)o(v)o(e)h(allo)q (cated)g(consecutiv)o(e)i(storage)f(for)75 835 y(ten)h(real)g(n)o(um)o(b)q (ers,)f(starting)h(from)e Fu(a\(1\))p Fx(.)75 885 y(2nd)i(program:)75 966 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1016 y(IF\(rank.EQ.0\))119 1066 y(THEN)162 1116 y(CALL)g(MPI_SEND\(a\(1\),)e (10,)i(MPI_REAL,)f(1,)i(tag,)e(comm,)h(ierr\))119 1165 y(ELSE)162 1215 y(CALL)g(MPI_RECV\(a\(1\),)e(40,)i(MPI_BYTE,)f(0,)i(tag,)e(comm,)h (ierr\))75 1265 y(END)g(IF)158 1346 y Fx(This)12 b(co)q(de)h(is)e(erroneous,) j(since)e(sender)i(and)d(receiv)o(er)j(do)e(not)f(pro)o(vide)h(matc)o(hing)e (datat)o(yp)q(e)i(parameters.)75 1396 y(3rd)i(program:)75 1476 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1526 y(IF\(rank.EQ.0\))119 1576 y(THEN)162 1626 y(CALL)g(MPI_SEND\(a\(1\),)e (40,)i(MPI_BYTE,)f(1,)i(tag,)e(comm,)h(ierr\))119 1676 y(ELSE)162 1725 y(CALL)g(MPI_RECV\(a\(1\),)e(60,)i(MPI_BYTE,)f(0,)i(tag,)e(comm,)h (ierr\))75 1775 y(END)g(IF)158 1856 y Fx(This)14 b(co)q(de)h(is)e(correct,)i (irresp)q(ectiv)o(e)h(of)d(the)i(t)o(yp)q(e)f(of)g Fu(a\(1\))f Fx(and)g(follo)o(wing)f(v)n(ariables)h(in)g(store.)158 1984 y Fo(Discussion:)158 2030 y Ft(The)k(b)q(eha)o(vior)i(of)e(erroneous)h(co)q (de)g(is)g(implemen)o(tation-dep)r(end)q(en)o(t.)32 b(It)17 b(is)g(desirable)j(but)d(not)h(required)h(that)75 2076 y(errors)13 b(are)g(rep)q(orted)h(at)f(run-time.)k(An)c(MPI)g(implemen)o(tation)j(ma)o(y) d(decide)h(to)f(a)o(v)o(oid)h(t)o(yp)q(e-c)o(hec)o(king)h(for)e(the)g(sak)o (e)g(of)75 2121 y(e\016ciency)m(,)i(in)g(whic)o(h)g(case)f(none)h(of)e(the)h (three)h(programs)g(will)g(rep)q(ort)f(an)h(error.)20 b(On)14 b(a)g(system)g(where)g(real)h(n)o(um)o(b)q(ers)75 2167 y(o)q(ccup)o(y)g(four) f(b)o(ytes)g(and)g(b)q(oth)g(sender)h(and)f(receiv)o(er)h(execute)f(in)h(the) f(same)g(en)o(vironmen)o(t,)h(it)f(is)h(lik)o(ely)h(that)e(all)h(three)75 2213 y(co)q(des)f(will)g(ha)o(v)o(e)g(the)f(same)g(b)q(eha)o(vior.)75 2432 y Fw(1.6)59 b(Data)19 b(conversion)75 2523 y Fx(One)g(of)e(the)h(goals)f (of)g(MPI)h(is)g(to)g(supp)q(ort)g(parallel)f(computations)g(across)h (heterogeneous)i(en)o(vironmen)o(ts.)75 2573 y(Comm)o(uni)o(cation)11 b(in)i(a)h(heterogeneous)i(en)o(vironmen)o(t)d(ma)o(y)f(require)i(data)g(con) o(v)o(ersions.)158 2623 y(W)m(e)g(use)g(the)h(follo)o(wing)c(terminology:)75 2704 y Fv(t)o(yp)q(e)k(con)o(v)o(ersion)j Fx(c)o(hanges)d(the)f(datat)o(yp)q (e)g(of)f(a)h(v)n(alue,)f(e.g.)18 b(b)o(y)13 b(rounding)h(a)f Fu(REAL)g Fx(to)h(an)g Fu(INTEGER)p Fx(.)p eop %%Page: 9 9 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)753 b Fx(9)75 45 y Fv(represen)o(tati)o(on)12 b(con)o(v)o(ersion)18 b Fx(c)o(hanges)e(the)g(binary)f(represen)o(tation)i(of)e(a)g(v)n(alue,)f (e.g.)23 b(from)13 b(Hex)j(\015oating)179 95 y(p)q(oin)o(t)d(to)h(IEEE)g (\015oating)f(p)q(oin)o(t.)158 178 y(Represen)o(tation)20 b(con)o(v)o(ersion) f(is)g(needed)i(when)e(data)g(is)g(mo)o(v)o(ed)f(across)h(di\013eren)o(t)i (en)o(vironmen)o(ts,)e(e.g.)75 228 y(di\013eren)o(t)h(mac)o(hines)d(that)i (use)g(di\013eren)o(t)h(binary)e(enco)q(dings)h(for)f(the)h(same)f(datat)o (yp)q(e,)h(or)g(co)q(des)h(compiled)75 278 y(with)d(di\013eren)o(t)h (compiler)e(options)h(for)g(datat)o(yp)q(e)h(represen)o(tation.)30 b(Represen)o(tation)18 b(con)o(v)o(ersion)g(preserv)o(es,)75 327 y(to)d(the)g(greatest)h(p)q(ossible)f(exten)o(t,)h(the)f Fv(v)m(alue)g Fx(represen)o(ted.)23 b(Ho)o(w)o(ev)o(er,)15 b(rounding)g(errors)h(and)f(o)o(v)o(er\015o)o(w)f(and)75 377 y(under\015o)o(w)i(exceptions)h(ma)o(y)d(o)q(ccur)j(during)e(\015oating)g(p)q (oin)o(t)g(con)o(v)o(ersions;)i(con)o(v)o(ersion)f(of)f(in)o(tegers)i(ma)o(y) d(also)75 427 y(lead)i(to)h(exceptions)g(when)g(a)f(v)n(alue)g(that)h(can)f (represen)o(ted)j(in)d(one)h(system)f(cannot)h(b)q(e)g(represen)o(ted)i(in)d (the)75 477 y(other)f(system.)i(MPI)d(do)q(es)h(not)f(sp)q(ecify)g(rules)h (for)e(represen)o(tation)j(con)o(v)o(ersion.)158 527 y(The)j(t)o(yp)q(e)f (matc)o(hing)e(rules)j(imply)d(that)i(MPI)g(comm)o(unication)d(nev)o(er)k(en) o(tails)e(t)o(yp)q(e)i(con)o(v)o(ersion.)31 b(On)75 576 y(the)15 b(other)g(hand,)f(MPI)g(requires)i(that)e(a)g(represen)o(tation)i(con)o(v)o (ersion)e(b)q(e)h(p)q(erformed)f(when)h(a)f(t)o(yp)q(ed)g(v)n(alue)g(is)75 626 y(transferred)g(across)g(en)o(vironmen)o(ts)d(that)i(use)g(di\013eren)o (t)g(represen)o(tations)i(for)d(the)h(datat)o(yp)q(e)g(of)e(this)i(v)n(alue.) k(An)75 676 y(exception)f(o)q(ccurring)g(during)f(represen)o(tation)i(con)o (v)o(ersion)f(results)g(in)f(a)h(failure)e(of)h(the)h(comm)o(unicatio)o(n;)d (an)75 726 y(error)i(o)q(ccurs)g(either)g(in)e(the)i(send)g(op)q(eration,)e (or)h(the)g(receiv)o(e)h(op)q(eration,)f(or)g(b)q(oth.)158 776 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o(t)o(yp)q(ed) i(\(i.e.,)e(of)g(t)o(yp)q(e)i Fu(MPI)p 1180 776 14 2 v 15 w(BYTE)p Fx(\),)e(then)h(the)h(binary)e(represen)o(tation)75 826 y(of)h(the)g(b)o(yte) h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o(tical)f(to)h(the)h(binary)f (represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded)e(at)h(the)75 875 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender)g(and)e(receiv) o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o(vironmen)o(ts;)75 925 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g(required.)158 975 y(Note)f(that)g(no)g(con)o(v)o(ersion)g(ev)o(er)h(o)q(ccurs)g(when)g(an)e (MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f(system,)75 1025 y(where)j(all)e(pro)q(cesses)j(run)f(in)e(the)i(same)e(en)o(vironmen)o (t.)k(Also)c(note)i(the)f(di\013eren)o(t)h(b)q(eha)o(vior)f(of)f Fu(MPI)p 1695 1025 V 15 w(BYTE)g Fx(and)75 1075 y(of)18 b Fu(MPI)p 196 1075 V 15 w(CHARACTER)p Fx(.)f(A)h(bu\013er)i(descriptor)g(en)o(try)f (with)g(datat)o(yp)q(e)g(of)f Fu(MPI)p 1320 1075 V 15 w(CHARACTER)f Fx(can)i(only)f(matc)o(h)f(a)75 1124 y(F)m(ortran)e(v)n(ariable)f(of)h(t)o (yp)q(e)g Fu(CHARACTER)p Fx(;)e(and)i(represen)o(tation)i(con)o(v)o(ersion)e (ma)o(y)e(o)q(ccur)j(when)g(v)n(alues)f(of)f(t)o(yp)q(e)75 1174 y Fu(MPI)p 144 1174 V 15 w(CHARACTER)e Fx(are)i(transferred.,)h(e.g.,)e (from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q(ding.)158 1224 y(Consider)g(the)h(previous)f(three)h(examples.)158 1274 y(The)h(\014rst)h(program)d(is)i(correct,)h(assuming)d(b)q(oth)i(sender)i (and)d(receiv)o(er)j(declared)e(ten)h(consecutiv)o(e)g(real)75 1324 y(v)n(ariables)e(in)g(storage)g(starting)h(at)f Fu(a\(1\))p Fx(.)22 b(If)15 b(the)h(sender)h(and)e(receiv)o(er)i(execute)g(in)e (di\013eren)o(t)h(en)o(vironmen)o(ts,)75 1373 y(then)d(the)f(ten)h(real)f(v)n (alues)g(that)g(are)h(fetc)o(hed)g(from)d(the)j(send)g(bu\013er)g(will)e(b)q (e)h(con)o(v)o(erted)i(to)e(the)g(represen)o(tation)75 1423 y(for)j(reals)g(on)f(the)i(receiv)o(er)g(site)g(b)q(efore)f(they)h(are)f (stored)h(in)e(the)i(receiv)o(e)g(bu\013er.)22 b(While)14 b(the)i(n)o(um)o(b) q(er)e(of)g(real)75 1473 y(elemen)o(ts)e(fetc)o(hed)i(from)c(the)j(send)h (bu\013er)f(equal)f(the)h(n)o(um)o(b)q(er)f(of)g(real)g(elemen)o(ts)g(stored) i(in)e(the)h(receiv)o(e)g(bu\013er,)75 1523 y(the)j(n)o(um)o(b)q(er)e(of)h(b) o(ytes)h(stored)g(need)g(not)f(equal)g(the)g(n)o(um)o(b)q(er)g(of)f(b)o(ytes) i(loaded:)k(e.g.)i(the)15 b(sender)i(ma)o(y)c(use)j(a)75 1573 y(four)11 b(b)o(yte)h(represen)o(tation)h(and)e(the)i(receiv)o(er)g(an)e (eigh)o(t)g(b)o(yte)h(represen)o(tation)h(for)e(reals.)18 b(If)11 b(the)h(send)g(or)g(receiv)o(e)75 1623 y(bu\013er)i(do)e(not)g(con)o(tain)g (ten)i(consecutiv)o(e)g(real)e(v)n(ariables,)g(then)h(the)g(program)e(is)i (erroneous,)g(and)g(its)f(b)q(eha)o(vior)75 1672 y(is)i(unde\014ned.)158 1722 y(The)g(second)h(program)e(is)g(erroneous,)i(and)f(its)g(b)q(eha)o(vior) f(is)h(unde\014ned.)158 1772 y(The)e(third)g(program)e(is)h(correct.)19 b(The)12 b(exact)g(same)f(sequence)j(of)d(fort)o(y)g(b)o(ytes)h(that)g(w)o (ere)g(loaded)g(from)d(the)75 1822 y(send)k(bu\013er)h(will)d(b)q(e)i(stored) g(in)f(the)h(receiv)o(e)h(bu\013er,)f(ev)o(en)g(if)f(sender)i(and)e(receiv)o (er)i(run)f(in)f(a)g(di\013eren)o(t)h(en)o(viron-)75 1872 y(men)o(t.)k(The)d (message)f(sen)o(t)h(has)g(exactly)g(the)g(same)e(length)i(\(in)f(b)o(ytes\)) h(and)g(the)g(same)e(binary)h(represen)o(tation)75 1921 y(as)h(the)h(message) f(receiv)o(ed.)20 b(If)13 b(the)i(v)n(ariables)f(in)f(the)i(send)g(bu\013er)g (are)g(of)e(di\013eren)o(t)i(t)o(yp)q(es)g(as)f(the)h(v)n(ariables)e(in)75 1971 y(the)g(receiv)o(e)i(bu\013er,)e(or)g(they)g(are)g(of)g(the)g(same)f(t)o (yp)q(e)h(but)g(di\013eren)o(t)h(data)e(represen)o(tations)j(are)e(used,)h (then)f(the)75 2021 y(bits)i(stored)g(in)g(the)g(receiv)o(e)h(bu\013er)f(ma)o (y)e(enco)q(de)j(v)n(alues)e(that)h(are)g(di\013eren)o(t)h(from)d(the)i(v)n (alues)f(they)i(enco)q(ded)75 2071 y(in)d(the)i(send)g(bu\013er.)158 2121 y(Data)h(represen)o(tation)h(con)o(v)o(ersion)g(also)e(applies)h(to)g (the)h(en)o(v)o(elop)q(e)g(of)e(a)h(message:)23 b(source,)17 b(destination)75 2170 y(and)d(tag)f(are)i(all)d(in)o(tegers)j(that)f(ma)o(y)e (need)j(to)e(b)q(e)i(con)o(v)o(erted.)158 2220 y(The)h(curren)o(t)h(draft)e (do)q(es)h(not)g(pro)o(vide)f(a)g(mec)o(hanism)e(for)i(in)o(ter-language)g (comm)o(unication:)j(messages)75 2270 y(sen)o(t)g(b)o(y)f(F)m(ortran)f(calls) h(should)g(b)q(e)g(receiv)o(ed)i(b)o(y)d(F)m(ortran)h(calls)g(and)g(messages) g(sen)o(t)g(b)o(y)g(C)g(calls)g(should)f(b)q(e)75 2320 y(receiv)o(ed)g(b)o(y) f(C)f(calls)h(\(this)g(follo)o(ws)e(from)g(the)i(requiremen)o(ts)g(of)f(t)o (yp)q(e)i(matc)o(hing)d(and)h(the)i(fact)e(that)h(F)m(ortran)75 2370 y(and)f(C)g(datat)o(yp)q(es)g(are)h(distinct\).)k(If)13 b(in)o(ter-language)h(comm)o(unicati)o(on)d(is)j(needed)i(then)e(one)h(needs) g(to)f(in)o(v)o(ok)o(e)75 2420 y(C)j(comm)o(unication)d(routines)k(from)e(a)h (F)m(ortran)g(program)e(or)j(F)m(ortran)f(comm)o(unicatio)o(n)e(routines)i (from)f(a)h(C)75 2469 y(program.)158 2598 y Fo(Implemen)o(tation)c(note:)158 2643 y Ft(The)j(curren)o(t)g(de\014nition)i(do)q(es)e(not)g(require)h (messages)f(to)g(carry)g(data)g(t)o(yp)q(e)g(information.)27 b(A)15 b(message)i(can)f(b)q(e)75 2689 y(comp)q(osed)h(and)f(sen)o(t)f(using) i(only)g(the)e(information)j(pro)o(vided)f(b)o(y)f(the)g(send)g(call,)h(and)f (can)g(b)q(e)f(receiv)o(ed)i(and)f(stored)p eop %%Page: 10 10 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(10)75 45 y Ft(using)12 b(only)g(the)f(information)i(pro)o (vided)g(b)o(y)e(the)g(receiv)o(e)h(call.)18 b(If)10 b(messages)h(are)g(sen)o (t)g(b)q(et)o(w)o(een)h(di\013eren)o(t)g(mac)o(hines)g(then)75 91 y(one)i(can)h(either)f(use)h(a)f(\\univ)o(ersal")i(data)f(enco)q(ding)h (for)d(messages,)i(use)f(kno)o(wledge)i(of)d(the)h(receiv)o(er)h(en)o (vironmen)o(t)h(in)75 136 y(order)f(to)g(con)o(v)o(ert)h(data)f(at)g(the)g (sender,)g(or)g(use)h(kno)o(wledge)g(of)f(the)g(sender)g(en)o(vironmen)o(t)i (in)f(order)f(to)g(con)o(v)o(ert)g(data)75 182 y(at)g(the)g(receiv)o(er.)23 b(In)15 b(either)h(case)f(the)g(datat)o(yp)q(e)h(parameter)f(in)h(the)f(lo)q (cal)i(call)f(can)f(b)q(e)g(used)h(to)f(deriv)o(e)h(the)f(t)o(yp)q(es)g(of)75 228 y(the)g(v)n(alues)g(transferred.)22 b(Ho)o(w)o(ev)o(er,)14 b(additional)k(t)o(yp)q(e)d(information)h(carried)f(b)o(y)g(messages)g(can)g (pro)o(vide)h(b)q(etter)f(error)75 273 y(detection.)158 484 y Fo(Discussion:)158 530 y Ft(MPI)e(do)q(es)h(not)f(handle)i(in)o (ter-language)h(comm)o(unication)g(b)q(ecause)e(there)f(seem)h(to)f(b)q(e)g (no)h(agreed)f(standards)i(for)75 576 y(in)o(ter-language)g(t)o(yp)q(e)d(con) o(v)o(ersions.)19 b(The)12 b(curren)o(t)h(\(non\)solution)i(means)d(that)h (in)o(ter-language)i(comm)o(unication)f(relies)75 621 y(on)f(the)g(v)o(endor) h(sp)q(eci\014c)h(rules)f(for)f(in)o(ter-language)j(calls)e(at)f(eac)o(h)g (site.)158 667 y(It)f(ma)o(y)h(b)q(e)g(desireable)h(to)f(pro)o(vide)h (information)g(on)f(the)g(corresp)q(ondance)i(b)q(et)o(w)o(een)d(MPI)h(C)f (datat)o(yp)q(es)i(\(suc)o(h)f(as)75 713 y Fm(MPI)p 137 713 12 2 v 13 w(FLOAT)p Ft(\))e(and)i(MPI)g(F)m(ortran)h(datat)o(yp)q(es)g(\(suc) o(h)g(as)f Fm(MPI)p 963 713 V 13 w(REAL)p Ft(\))e(with)i(suitable)i(en)o (vironmen)o(t)g(query)f(functions.)75 933 y Fw(1.7)59 b(Comm)n(unicat)o(ion) 17 b(Mo)r(des)75 1024 y Fx(The)12 b(basic)g(send)h(op)q(eration)f(describ)q (ed)i(in)d(Section)h(1.2)f(used)i(the)g Fv(standard)d Fx(comm)o(uni)o(cation) f(mo)q(de.)16 b(In)c(suc)o(h)75 1073 y(a)18 b(comm)o(unicati)o(on)d(mo)q(de,) j(a)f(send)i(op)q(eration)f(can)g(b)q(e)h(started)g(whether)g(or)f(not)g(a)g (matc)o(hing)e(receiv)o(e)j(w)o(as)75 1123 y(p)q(osted.)f(The)13 b(completion)d(of)i(the)g(send)h(op)q(eration)f(indicates)g(that)g(the)h (message)f(and)f(its)i(en)o(v)o(elop)q(e)f(ha)o(v)o(e)g(b)q(een)75 1173 y(safely)17 b(stored)h(a)o(w)o(a)o(y)d(and)i(that)g(the)h(sender)g(is)f (free)h(to)e(access)j(and)e(mo)q(dify)e(the)i(sender)i(bu\013er.)28 b(Th)o(us,)17 b(the)75 1223 y(op)q(eration)f(is)g Fv(blo)q(c)o(king)p Fx(:)k(it)c(do)q(es)g(not)g(return)h(un)o(til)f(the)g(send)h(op)q(eration)f (has)g(completed)g Fv(lo)q(cally)p Fx(,)f(on)g(the)75 1273 y(sender)g(side.)158 1322 y(The)h(completion)e(of)i(a)f(send)i(op)q(eration)e (giv)o(es)h(no)g(indication)e(that)i(the)g(message)g(w)o(as)g(receiv)o(ed)h (on)e(the)75 1372 y(receiv)o(er)k(side.)29 b(A)17 b(blo)q(c)o(king)g(send)h (ma)o(y)d(b)q(e)j(implemen)o(ted)d(so)j(that)f(a)g(blo)q(c)o(king)g(send)h (returns)h(only)e(after)g(a)75 1422 y(matc)o(hing)9 b(receiv)o(e)k(has)e(b)q (een)i(executed)g(on)e(the)g(receiv)o(er)i(side.)18 b(This)11 b(a)o(v)o(oids)f(the)i(need)g(to)f(bu\013er)h(message)f(data)75 1472 y(out)i(of)f(sender)i(or)e(receiv)o(er)j(memory)m(.)f(In)f(this)g(case)g (the)g(send)h(op)q(eration)e(completes)h(only)f(after)h(the)g(matc)o(hing)75 1522 y(receiv)o(e)18 b(has)g(started)g(executing.)28 b(On)17 b(the)h(other)g(hand,)f(it)g(is)g(also)f(p)q(ossible)h(for)g(MPI)g(to)g (bu\013er)h(messages,)75 1572 y(so)d(as)h(to)f(allo)o(w)f(the)i(sender)h(to)e (pro)q(ceed)i(ahead)f(of)e(the)i(receiv)o(er.)24 b(In)16 b(suc)o(h)g(a)f (case)h(the)g(send)h(op)q(eration)e(ma)o(y)75 1621 y(complete)e(successfully) i(b)q(efore)g(the)g(message)e(w)o(as)h(receiv)o(ed.)20 b(Th)o(us,)13 b(the)i(basic)f(send)h(op)q(eration)f(describ)q(ed)h(in)75 1671 y(the)h(previous)g(section)g(is)g Fv(async)o(hronous)p Fx(,)c(since)17 b(its)e(return)i(do)q(es)f(not)f(imply)f(a)h(sync)o (hronization)g(with)g(the)75 1721 y(\(remote\))f(receiv)o(e)h(op)q(eration,)e (and)h(do)q(es)h(not)e(imply)f Fv(global)g Fx(termination)g(of)h(the)i(comm)o (uni)o(cation.)158 1771 y(There)g(are)f(t)o(w)o(o)g(additional)e(comm)o (unicatio)o(n)f(mo)q(des:)158 1821 y(A)j(send)h(that)f(uses)h(the)f Fv(ready)g Fx(comm)o(uni)o(cation)d(mo)q(de)i(ma)o(y)f(b)q(e)i(started)h (only)e(if)g(a)h(matc)o(hing)e(receiv)o(e)j(is)75 1870 y(already)d(p)q (osted;)h(otherwise)g(the)g(op)q(eration)f(is)g(erroneous)i(and)e(its)g (outcome)g(is)g(unde\014ned.)19 b(In)12 b(some)f(systems,)75 1920 y(this)h(allo)o(ws)e(the)i(remo)o(v)n(al)d(of)i(a)g(hand-shak)o(e)h(op)q (eration)g(that)f(is)h(otherwise)g(required,)h(and)e(results)i(in)e(impro)o (v)o(ed)75 1970 y(p)q(erformance.)24 b(The)17 b(completion)d(of)i(the)g(send) h(op)q(eration)f(do)q(es)h(not)f(dep)q(end)h(on)f(the)h(status)g(of)e(a)h (matc)o(hing)75 2020 y(receiv)o(e,)i(and)e(merely)g(indicates)h(that)g(the)g (send)g(bu\013er)h(can)f(b)q(e)g(reused.)27 b(A)17 b(send)h(op)q(eration)e (that)h(uses)g(the)75 2070 y(ready)i(mo)q(de)e(has)i(the)g(same)e(seman)o (tics)h(as)h(a)f(standard)h(send)g(op)q(eration.)32 b(It)18 b(is)g(merely)g(that)g(the)h(sender)75 2120 y(pro)o(vides)14 b(additional)e(information)f(to)j(the)g(comm)o(unication)d(subsystem,)i(when) i(using)e(the)i(ready)f(mo)q(de.)158 2169 y(A)e(send)h(that)f(uses)h(the)g Fv(sync)o(hronous)c Fx(mo)q(de)i(can)i(b)q(e)f(started)h(whether)h(or)e(not)g (a)g(matc)o(hing)e(receiv)o(e)j(w)o(as)75 2219 y(p)q(osted.)23 b(Ho)o(w)o(ev)o(er,)16 b(the)g(send)g(will)e(complete)g(successfully)j(only)d (if)h(a)g(matc)o(hing)e(receiv)o(e)k(is)e(p)q(osted,)h(and)f(the)75 2269 y(receiv)o(e)g(op)q(eration)e(has)g(started)h(to)f(receiv)o(e)i(the)f (message)f(sen)o(t)h(b)o(y)f(the)h(sync)o(hronous)g(send.)k(\(I.e.,)13 b(the)h(receiv)o(e)75 2319 y(has)j(b)q(een)g(p)q(osted,)h(and)f(the)g (incoming)d(message)j(has)f(b)q(een)i(matc)o(hed)e(to)g(this)h(p)q(osted)h (receiv)o(e.\))27 b(Th)o(us,)17 b(the)75 2369 y(completion)e(of)h(a)g(sync)o (hronous)h(send)h(not)e(only)g(indicates)h(that)f(the)i(send)f(bu\013er)h (can)e(b)q(e)h(reused,)i(but)e(also)75 2418 y(indicates)f(that)f(the)h (receiv)o(er)h(has)f(reac)o(hed)h(a)e(certain)h(p)q(oin)o(t)f(in)g(its)h (execution,)g(namely)e(that)h(it)g(has)h(started)75 2468 y(executing)g(the)f (matc)o(hing)e(receiv)o(e.)23 b(If)14 b(b)q(oth)h(sends)i(and)d(receiv)o(es)j (are)e(blo)q(c)o(king)f(op)q(erations)i(then)f(the)h(use)g(of)75 2518 y(the)f(sync)o(hronous)h(mo)q(de)e(pro)o(vides)h(sync)o(hronous)g(comm)o (unication)d(seman)o(tics:)19 b(a)14 b(comm)o(unication)e(do)q(es)j(not)75 2568 y(complete)f(at)h(either)g(ends)h(b)q(efore)f(b)q(oth)g(pro)q(cesses)i (\\attend")e(to)g(the)g(comm)o(unicatio)o(n;)d(the)j(completion)e(of)h(a)75 2618 y(sync)o(hronous)h(send)g(is)e(a)h Fv(global)e Fx(ev)o(en)o(t.)158 2667 y(Tw)o(o)h(additional)f(send)j(functions)f(are)g(pro)o(vided)g(for)f (the)i(t)o(w)o(o)e(additional)f(comm)o(unicatio)o(n)f(mo)q(des.)18 b(The)p eop %%Page: 11 11 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(11)75 45 y(comm)o(unication)10 b(mo)q(de)j(is)h(indicated)g(b) o(y)g(a)f(one)h(letter)h(pre\014x:)k Fu(R)13 b Fx(for)h(ready)g(and)g Fu(S)g Fx(for)f(sync)o(hronous)158 145 y(Send)h(in)g(ready)g(mo)q(de)75 242 y Fu(MPI)p 144 242 14 2 v 15 w(RSEND)21 b(\(start,)f(count,)h(datatype,)f (dest,)g(tag,)h(comm\))117 292 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 337 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (send)g(bu\013er)g(\(in)o(teger\))117 383 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 429 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 474 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 520 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 696 y Fx(Send)g(in)g(sync)o(hronous)h(mo)q(de)75 793 y Fu(MPI)p 144 793 V 15 w(SSEND)21 b(\(start,)f(count,)h(datatype,)f(dest,)g(tag,)h (comm\))117 843 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e (send)i(bu\013er)g(\(c)o(hoice\))117 888 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(bu\013er)g(\(in)o (teger\))117 934 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 980 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1025 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1071 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 1147 y Fx(There)g(is)e(only)g(one)g(receiv)o(e)i(mo)q(de,)d(whic)o(h)h(can)h (matc)o(h)e(an)o(y)h(of)g(the)h(send)h(mo)q(des.)i(The)d(receiv)o(e)h(op)q (eration)75 1197 y(describ)q(ed)h(in)e(the)h(last)g(section)g(is)f Fv(blo)q(c)o(king)p Fx(:)j(it)d(returns)i(only)d(after)i(the)g(receiv)o(e)h (bu\013er)g(con)o(tains)e(the)h(newly)75 1247 y(receiv)o(ed)20 b(message.)34 b(It)19 b(is)g Fv(async)o(hronous)p Fx(:)25 b(the)20 b(completion)d(of)i(a)f(receiv)o(e)j(is)e(a)f(lo)q(cal)h(op)q(eration,)g(and) g(a)75 1297 y(receiv)o(e)c(can)f(complete)f(b)q(efore)i(the)f(matc)o(hing)e (send)i(has)g(completed)f(\(of)h(course,)g(it)g(can)g(complete)f(only)g (after)75 1347 y(the)h(matc)o(hing)f(send)h(has)g(started\).)158 1396 y(Comm)o(uni)o(cation)f(imp)q(oses)j(an)g(order)h(on)g(the)g(ev)o(en)o (ts)g(o)q(ccurring)g(at)f(the)h(comm)o(unicating)c(no)q(des.)27 b(It)16 b(is)75 1446 y(alw)o(a)o(ys)d(the)i(case)h(that)e(the)h(completion)d (of)i(a)g(receiv)o(e)i(o)q(ccurs)g(after)e(the)h(start)g(of)f(the)h(matc)o (hing)d(send.)20 b(If)14 b(the)75 1496 y(sync)o(hronous)i(send)g(mo)q(de)f (is)g(used,)h(then)g(it)f(is)g(also)g(the)h(case)g(that)f(the)h(completion)e (of)h(a)g(send)h(o)q(ccurs)h(after)75 1546 y(the)f(start)g(of)g(the)g(matc)o (hing)e(receiv)o(e.)24 b(Of)16 b(course,)h(on)e(eac)o(h)h(pro)q(cess,)i(a)d (comm)o(unicatio)o(n)e(completes)i(after)h(it)75 1596 y(is)h(started.)28 b(No)17 b(other)g(order)h(is)f(imp)q(osed)f(b)o(y)g(MPI.)h(E.g.,)f(if)g(the)i (standard)f(send)h(mo)q(de)e(is)g(used,)j(the)e(send)75 1645 y(op)q(eration)d(ma)o(y)e(complete)h(b)q(efore)i(the)f(matc)o(hing)e(receiv)o (e)j(w)o(as)f(started.)158 1695 y(In)d(a)f(m)o(ulti-threaded)f(implemen)o (tation)e(of)j(MPI,)h(a)f(blo)q(c)o(king)f(send)j(or)f(receiv)o(e)g(op)q (eration)g(ma)o(y)e(only)g(blo)q(c)o(k)75 1745 y(the)15 b(executing)h (thread,)f(not)g(the)g(en)o(tire)h(pro)q(cess.)22 b(Th)o(us,)15 b(while)g(a)f(thread)i(is)e(blo)q(c)o(k)o(ed)h(w)o(aiting)e(for)i(a)f(send)i (or)75 1795 y(receiv)o(e)i(to)f(complete,)g(other)g(threads)h(ma)o(y)d (execute)k(within)d(the)i(same)e(address)i(space.)28 b(In)17 b(suc)o(h)h(a)e(case)i(it)75 1845 y(the)d(user)h(resp)q(onsibilit)o(y)f(not)f (to)h(mo)q(dify)d(the)k(send)f(bu\013er)h(un)o(til)e(the)h(send)h(completes)f (and)f(not)h(to)f(access)j(or)75 1895 y(mo)q(dify)12 b(the)k(receiv)o(e)g (bu\013er)g(un)o(til)e(the)h(receiv)o(e)h(completes;)f(otherwise)h(the)f (outcome)f(of)h(the)g(computation)e(is)75 1944 y(unde\014ned.)158 2073 y Fo(Discussion:)158 2118 y Ft(Do)g(w)o(e)g(w)o(an)o(t)g(to)g(sa)o(y)g (that)g(a)g(blo)q(c)o(king)j(op)q(eration)f(MA)m(Y)d(or)h(MUST)g(only)h(blo)q (c)o(k)g(the)g(executing)g(thread?)158 2329 y Fo(Implemen)o(tation)f(note:)38 b Ft(A)13 b(ready)h(send)h(can)f(b)q(e)g(implemen)o(ted)h(as)f(a)g(standard)h (send;)f(in)h(suc)o(h)f(a)g(case)g(there)75 2375 y(will)h(b)q(e)e(no)g(p)q (erformance)h(adv)n(an)o(tage)g(\(or)f(disadv)n(an)o(tage\))j(for)d(the)g (use)g(of)g(ready)g(send.)158 2421 y(A)d(standard)h(send)g(can)g(b)q(e)f (implemen)o(ted)j(as)d(a)h(sync)o(hronous)h(send.)17 b(In)10 b(suc)o(h)h(a)f(case,)h(no)f(data)h(bu\013ering)h(is)f(needed.)158 2466 y(A)i(p)q(ossible)i(comm)o(unication)h(proto)q(col)e(for)f(the)g(v)n (arious)i(comm)o(unication)g(mo)q(des)f(is)g(outlined)h(b)q(elo)o(w:)158 2512 y Fm(ready)j(send)p Ft(:)d(The)e(message)g(is)h(sen)o(t)f(as)g(so)q(on)h (as)f(p)q(ossible.)158 2558 y Fm(synchronou)o(s)j(send:)25 b Ft(The)18 b(sender)g(sends)h(a)f(request-to-send)h(message.)31 b(The)18 b(receiv)o(er)h(stores)f(this)g(request.)75 2603 y(When)e(a)e(matc)o (hing)j(receiv)o(e)e(is)h(p)q(osted,)f(the)g(receiv)o(er)h(sends)f(bac)o(k)h (a)f(p)q(ermission-to-send)i(message,)f(and)f(the)g(sender)75 2649 y(no)o(w)e(sends)h(the)f(message.)158 2695 y Fm(standard)k(send:)d Ft(First)e(proto)q(col)g(ma)o(y)g(b)q(e)f(used)h(for)f(short)g(messages,)h (and)g(second)g(proto)q(col)h(for)e(long)h(messages.)p eop %%Page: 12 12 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(12)158 45 y Ft(Additional)17 b(con)o(trol)d(messages)h(migh)o (t)f(b)q(e)g(needed)h(for)f(\015o)o(w)f(con)o(trol)i(and)f(error)g(reco)o(v)o (ery)m(.)20 b(Of)13 b(course,)h(there)g(are)75 91 y(man)o(y)g(other)f(p)q (ossible)i(c)o(hoices.)75 311 y Fw(1.8)59 b(Nonblo)r(cking)20 b(comm)m(unicat)o(ion)75 402 y Fx(One)d(can)f(impro)o(v)o(e)f(p)q(erformance) h(on)g(man)o(y)e(systems)i(b)o(y)g(o)o(v)o(erlapping)f(comm)o(unication)e (and)j(computation.)75 451 y(This)11 b(is)g(esp)q(ecially)h(true)g(on)f (systems)h(where)g(comm)o(unication)c(can)k(b)q(e)g(executed)h(autonomously)c (b)o(y)i(an)h(in)o(telli-)75 501 y(gen)o(t)h(comm)o(unicatio)o(n)d(con)o (troller.)18 b(Ligh)o(t-w)o(eigh)o(t)11 b(threads)j(are)f(one)g(mec)o(hanism) d(for)j(ac)o(hieving)f(suc)o(h)h(o)o(v)o(erlap.)75 551 y(An)h(alternativ)o(e) g(mec)o(hanism)e(that)j(often)f(leads)g(to)g(b)q(etter)i(p)q(erformance)e(is) g(to)h(use)g Fv(non)o(blo)q(c)o(kin)o(g)e(comm)o(u-)75 601 y(nication)p Fx(.)i(A)f(non)o(blo)q(c)o(king)e(send)i(call)e(initiates)h(the) h(send)g(op)q(eration,)f(but)g(do)q(es)h(not)f(complete)g(it.)k(The)d(send)75 651 y(call)h(will)e(return)k(b)q(efore)f(the)g(message)f(w)o(as)g(copied)h (out)f(of)g(the)h(send)g(bu\013er.)24 b(A)15 b(separate)i(call)d(is)h(needed) i(to)75 701 y(complete)f(the)i(comm)o(unicati)o(on,)c(i.e.)27 b(to)16 b(v)o(erify)h(that)f(the)i(data)e(has)h(b)q(een)h(copied)f(out)g(of)f (the)i(send)f(bu\013er.)75 750 y(With)e(suitable)h(hardw)o(are,)g(the)g (transfer)h(of)e(data)g(out)h(of)f(the)h(sender)h(memory)d(ma)o(y)f(pro)q (ceed)18 b(concurren)o(tly)75 800 y(with)10 b(computations)g(done)h(at)f(the) h(sender)i(after)d(the)i(send)f(w)o(as)g(initiated)f(and)g(b)q(efore)i(it)e (completed.)16 b(Similarly)l(,)75 850 y(a)e(non)o(blo)q(c)o(king)f(receiv)o (e)i(initiates)f(the)h(receiv)o(e)g(op)q(eration,)f(but)g(do)q(es)h(not)f (complete)g(it.)k(The)d(call)e(will)g(return)75 900 y(b)q(efore)i(a)f (message)f(is)h(stored)h(in)o(to)e(the)i(receiv)o(e)g(bu\013er.)20 b(A)14 b(separate)h(call)e(is)h(needed)i(to)e(complete)f(the)i(receiv)o(e)75 950 y(op)q(eration)c(and)g(v)o(erify)g(that)h(the)f(data)g(has)h(b)q(een)g (receiv)o(ed)h(in)o(to)e(the)g(receiv)o(e)i(bu\013er.)18 b(With)11 b(suitable)g(hardw)o(are,)75 999 y(the)18 b(transfer)g(of)e(data)h(in)o(to)g (the)g(receiv)o(er)i(memory)c(ma)o(y)g(pro)q(ceed)k(concurren)o(tly)f(with)f (computations)f(done)75 1049 y(after)e(the)h(receiv)o(e)g(w)o(as)f(initiated) f(and)g(b)q(efore)i(it)f(completed.)158 1099 y(Non)o(blo)q(c)o(king)k(sends)j (can)e(use)h(the)g(same)f(three)h(mo)q(des)f(as)g(blo)q(c)o(king)f(sends:)30 b Fu(standard)p Fx(,)19 b Fu(ready)f Fx(and)75 1149 y Fu(synchronous)p Fx(.)e(These)e(carry)g(the)g(same)e(meaning.)k(The)e(initiation)e(and)h (completion)e(of)i(a)g(standard)h(send)g(do)75 1199 y(not)h(dep)q(end)i(on)e (the)h(status)h(of)d(a)i(matc)o(hing)d(receiv)o(e.)24 b(A)16 b(ready)g(send)g(can)g(b)q(e)g(initiated)e(only)h(if)g(a)g(matc)o(hing)75 1248 y(receiv)o(e)e(has)f(already)f(b)q(een)i(initiated,)e(otherwise)i(the)f (call)f(is)h(erroneous;)h(its)f(completion)e(do)q(es)j(not)e(dep)q(end)i(on) 75 1298 y(the)g(status)h(of)d(a)i(matc)o(hing)d(receiv)o(e.)19 b(A)13 b(sync)o(hronous)g(send)h(can)f(b)q(e)g(initiated)f(b)q(efore)h(a)f (matc)o(hing)f(receiv)o(e)j(has)75 1348 y(b)q(een)i(initiated,)f(but)g(will)f (complete)h(only)f(after)i(a)f(matc)o(hing)e(receiv)o(e)k(has)e(b)q(een)i (initiated,)d(and)h(has)h(started)75 1398 y(receiving)e(the)h(message)e (generated)j(b)o(y)d(the)i(send)f(op)q(eration.)158 1448 y(In)k(all)e(cases)j (the)f(op)q(eration)f(that)h(initiates)f(the)h(comm)o(unicati)o(on)d(\(send)j (or)g(receiv)o(e\))h(is)e(a)g(lo)q(cal,)g(non-)75 1498 y(blo)q(c)o(king)d (call)g(that)h(returns)h(as)f(so)q(on)g(as)g(p)q(ossible.)20 b(A)15 b(non)o(blo)q(c)o(king)f(comm)o(unicatio)o(n)e(call)i(ma)o(y)f(fail)g (b)q(ecause)75 1547 y(b)q(ecause)g(the)g(system)e(has)h(exhausted)h(a)o(v)n (ailable)d(resources)k(\(e.g.,)d(exceeded)j(the)e(limit)d(on)j(n)o(um)o(b)q (er)f(of)g(p)q(ending)75 1597 y(comm)o(unications)f(p)q(er)15 b(no)q(de\).)j(Go)q(o)q(d)13 b(qualit)o(y)f(implemen)o(tatio)o(ns)f(of)i(MPI) h(will)e(set)i(these)h(limits)c(high)i(enough)75 1647 y(so)h(that)g(a)f(non)o (blo)q(c)o(king)g(comm)o(unication)e(call)i(will)f(fail)g(only)h(in)h (\\pathological")d(cases.)158 1697 y(Non)o(blo)q(c)o(king)i(sends)i(can)f(b)q (e)g(matc)o(hed)g(with)f(blo)q(c)o(king)g(receiv)o(es,)i(and)f(vice-v)o (ersa.)75 1813 y Fq(1.8.1)49 b(Comm)o(unication)14 b(Objects)75 1890 y Fx(Non)o(blo)q(c)o(king)f(comm)o(unications)f(use)j(opaque)f(comm)o (unication)d(ob)r(jects)16 b(to)e(iden)o(tify)g(comm)o(unicatio)o(n)e(op)q (era-)75 1939 y(tions)h(and)g(matc)o(h)f(the)i(op)q(eration)f(that)g (initiates)g(the)h(comm)o(uni)o(cation)c(with)j(the)h(op)q(eration)f(that)g (terminates)75 1989 y(it.)32 b(These)20 b(are)f(system)f(ob)r(jects)i(that)f (are)g(accessed)i(via)d(a)g(handle.)33 b(An)18 b(opaque)h(comm)o(unicatio)o (n)d(ob)r(ject)75 2039 y(iden)o(ti\014es)e(v)n(arious)f(prop)q(erties)i(of)e (a)g(comm)o(unicati)o(on)e(op)q(eration,)i(suc)o(h)h(as)f(the)i(\(send)f(or)g (receiv)o(e\))g(bu\013er)h(that)75 2089 y(is)g(asso)q(ciated)h(with)f(it,)g (its)h(con)o(text,)g(the)g(tag)f(and)g(destination)g(parameters)g(to)h(b)q(e) g(used)g(for)f(a)g(send,)h(or)f(the)75 2139 y(tag)d(and)f(source)j (parameters)e(to)g(b)q(e)g(used)h(for)f(a)f(receiv)o(e.)19 b(In)12 b(addition,)f(this)h(ob)r(ject)h(stores)g(information)c(ab)q(out)75 2188 y(the)14 b(status)h(of)e(the)i(p)q(ending)f(comm)o(unicatio)o(n)d(op)q (eration)j(that)g(is)g(p)q(erformed)f(with)h(this)g(ob)r(ject.)75 2305 y Fq(1.8.2)49 b(Comm)o(unication)14 b(initiation)75 2381 y Fx(W)m(e)i(use)g(the)h(same)e(naming)e(con)o(v)o(en)o(tions)j(as)g(for)g (blo)q(c)o(king)f(comm)o(unicatio)o(n:)20 b(a)15 b(pre\014x)i(of)e Fu(R)h Fx(\()p Fu(S)p Fx(\))g(is)f(used)i(for)75 2431 y Fu(READY)g Fx(\()p Fu(SYNCHRONOUS)p Fx(\))f(mo)q(de.)29 b(In)18 b(addition)f(a)g (pre\014x)i(of)e Fu(I)h Fx(\(for)g Fu(IMMEDIATE)p Fx(\))e(indicates)i(that)g (the)h(call)e(is)75 2481 y(non)o(blo)q(c)o(king.)158 2581 y(Initiate)c(a)h (standard)g(mo)q(de)f(non)o(blo)q(c)o(king)g(comm)o(unicatio)o(n.)75 2678 y Fu(MPI)p 144 2678 14 2 v 15 w(ISEND\(handle,)19 b(start,)i(count,)f (datatype,)g(dest,)h(tag,)g(comm\))p eop %%Page: 13 13 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(13)117 45 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 91 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o(hoice\))117 136 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (send)g(bu\013er)g(\(in)o(teger\))117 182 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 228 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 273 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 319 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 495 y Fx(Initiate)f(a)h(ready)g(mo)q(de)f(non)o(blo)q(c)o(king)g(comm)o (unicatio)o(n.)75 592 y Fu(MPI)p 144 592 14 2 v 15 w(IRSEND\(handle,)19 b(start,)i(count,)f(datatype,)g(dest,)h(tag,)g(comm\))117 642 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 688 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 733 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(bu\013er)g(\(in)o(teger\))117 779 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h (bu\013er)g(elemen)o(t)117 825 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 870 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 916 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 1092 y Fx(Initiate)f(a)h(sync)o (hronous)h(mo)q(de)e(non)o(blo)q(c)o(king)f(comm)o(unication.)75 1189 y Fu(MPI)p 144 1189 V 15 w(ISSEND\(handle,)19 b(start,)i(count,)f (datatype,)g(dest,)h(tag,)g(comm\))117 1239 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1284 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(bu\013er)g(\(c)o (hoice\))117 1330 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(bu\013er)g(\(in)o(teger\))117 1376 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(send)h (bu\013er)g(elemen)o(t)117 1421 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1467 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1513 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 1689 y Fx(Initiate)f(a)h(non)o(blo)q (c)o(king)f(receiv)o(e.)75 1786 y Fu(MPI)p 144 1786 V 15 w(IRECV)21 b(\(handle,)f(start,)h(count,)f(datatype,)g(source,)g(tag,)h(comm\))117 1835 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob) r(ject)117 1881 y(out)e Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv) o(e)i(bu\013er)g(\(c)o(hoice\))117 1927 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(receiv)o(e)g(bu\013er)g (\(in)o(teger\))117 1972 y(in)g Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)117 2018 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 2064 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2109 y(in)g Fu(comm)697 b Ft(comm)o(unicator)14 b(\(handle\))158 2186 y Fx(The)i(call)f(allo)q(cates)g(a)g(comm)o(unication)d(ob)r(ject)17 b(and)e(asso)q(ciates)h(it)f(with)h(the)g(handle.)22 b(The)16 b(handle)g(can)75 2235 y(b)q(e)f(used)f(later)g(to)g(query)g(ab)q(out)g(the)h (status)f(of)g(the)g(comm)o(unication)d(or)i(w)o(ait)h(for)f(its)h (completion.)158 2285 y(A)d(sender)h(should)f(not)f(up)q(date)h(an)o(y)g (part)f(of)g(a)h(send)g(bu\013er)h(after)f(a)f(non)o(blo)q(c)o(king)g(send)h (op)q(eration)g(returns,)75 2335 y(un)o(til)h(the)i(send)g(completes.)k(A)13 b(receiv)o(er)i(should)e(not)g(access)i(or)e(up)q(date)h(an)o(y)e(part)i(of)e (a)h(receiv)o(e)i(bu\013er)f(after)f(a)75 2385 y(non)o(blo)q(c)o(king)g (receiv)o(e)i(op)q(eration)f(returns,)h(un)o(til)e(the)h(receiv)o(e)h (completes.)p eop %%Page: 14 14 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(14)75 45 y Fq(1.8.3)49 b(Comm)o(unication)14 b(Completion)75 122 y Fx(The)j(completion)e(of)h(a)h(send)g(op)q(eration)g (indicates)g(that)g(the)g(sender)i(is)d(no)o(w)h(free)g(to)g(up)q(date)g(the) g(lo)q(cations)75 172 y(in)g(the)h(send)g(bu\013er,)h(or)f(an)o(y)e(other)i (lo)q(cation)f(that)g(can)h(b)q(e)g(referenced)i(b)o(y)d(the)h(send)h(op)q (eration)e(\(the)h(send)75 221 y(op)q(eration)h(itself)g(lea)o(v)o(es)g(the)g (con)o(ten)o(t)h(of)f(the)g(send)h(bu\013er)g(unc)o(hanged\).)34 b(It)20 b(do)q(es)f(not)g(indicate)g(that)g(the)75 271 y(message)f(has)g(b)q (een)i(receiv)o(ed;)h(rather,)e(it)f(ma)o(y)f(ha)o(v)o(e)h(b)q(een)h (bu\013ered)h(b)o(y)e(the)g(comm)o(unication)d(subsystem.)75 321 y(Ho)o(w)o(ev)o(er,)d(if)f(a)h Fu(synchronous)d Fx(mo)q(de)i(send)i(w)o (as)e(used,)i(the)g(completion)d(of)h(the)h(send)h(op)q(eration)f(indicates)g (that)75 371 y(a)h(matc)o(hing)e(receiv)o(e)k(w)o(as)e(initiated,)f(and)h (that)h(the)f(message)g(will)f(ev)o(en)o(tually)h(b)q(e)h(receiv)o(ed)g(b)o (y)f(this)h(matc)o(hing)75 421 y(receiv)o(e.)158 470 y(The)k(completion)e(of) g(a)i(receiv)o(e)g(op)q(eration)g(indicates)f(that)h(the)g(receiv)o(e)h (bu\013er)f(con)o(tains)g(the)g(receiv)o(ed)75 520 y(message,)e(and)g(that)g (the)g(status)h(ob)r(ject)g(is)f(set;)h(the)g(receiv)o(er)g(is)f(no)o(w)g (free)g(to)g(access)i(the)f(receiv)o(e)g(bu\013er,)g(or)75 570 y(an)o(y)f(other)i(lo)q(cation)d(that)i(can)g(b)q(e)g(referenced)j(b)o(y) c(the)i(receiv)o(e)g(op)q(eration.)26 b(It)17 b(do)q(es)g(not)g(indicate)g (that)f(the)75 620 y(matc)o(hing)c(send)j(op)q(eration)f(has)g(completed)f (\(but)h(indicates,)g(of)f(course,)i(that)f(the)g(send)h(w)o(as)f (initiated\).)75 767 y Fu(MPI)p 144 767 14 2 v 15 w(WAIT)21 b(\(handle,)f(status\))117 817 y Ft(in)14 b Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 862 y(out)e Fu(status)629 b Ft(status)13 b(ob)r(ject)158 939 y Fx(A)k(call)e(to)i Fu(MPI)p 407 939 V 15 w(WAIT)e Fx(returns)j(when)f(the)g(send)h(op)q(eration) e(iden)o(ti\014ed)g(b)o(y)h Fu(handle)e Fx(is)h(complete.)25 b(If)16 b(the)75 988 y(comm)o(unication)9 b(ob)r(ject)j(asso)q(ciated)h(with) f(this)g(handle)g(w)o(as)g(created)i(b)o(y)d(a)h(non)o(blo)q(c)o(king)f(send) i(or)f(receiv)o(e)h(call,)75 1038 y(then)i(the)f(ob)r(ject)h(is)e(deallo)q (cated)h(b)o(y)g(the)h(call)e(to)g Fu(MPI)p 951 1038 V 15 w(WAIT)g Fx(and)h(the)h(handle)e(b)q(ecomes)i(n)o(ull.)158 1088 y(The)i(call)f (returns)j(in)d Fu(status)g Fx(information)d(on)k(the)g(completed)g(op)q (eration.)26 b(The)18 b(status)f(ob)r(ject)h(for)e(a)75 1138 y(receiv)o(e)c(op)q(eration)e(can)h(b)q(e)g(queried)g(using)f(the)h (functions)g(describ)q(ed)h(in)e(Section)h(1.3.1.)k(The)c(status)g(parameter) 75 1188 y(is)g(not)h(used)g(or)f(up)q(dated)h(for)f(a)h(send)g(op)q(eration,) f(and)h(an)f(arbitrary)g(v)n(alue)g(ma)o(y)e(b)q(e)j(passed)h(for)e(this)g (parameter)75 1237 y(in)i(the)i(call.)158 1366 y Fo(Discussion:)158 1411 y Ft(Do)e(w)o(e)g(agree)g(on)h(the)f(last)g(sen)o(tence?)19 b(\(this)13 b(implies)j(some)d(restriction)i(on)e(the)g(implemen)o(tation)q (\))75 1591 y Fu(MPI)p 144 1591 V 15 w(TEST)21 b(\(handle,)f(flag,)h (status\))117 1641 y Ft(in)14 b Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1687 y(out)36 b Fu(flag)651 b Ft(logical)117 1732 y(out)14 b Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1809 y Fx(A)18 b(call)f(to)g Fu(MPI)p 410 1809 V 15 w(TEST)g Fx(returns)i Fu(flag=true)d Fx(if)h(the)h(op)q(eration)f(iden)o (ti\014ed)h(b)o(y)f Fu(handle)g Fx(is)g(complete.)29 b(In)75 1858 y(suc)o(h)21 b(a)f(case,)i(the)e(status)h(ob)r(ject)g(is)f(set)h(to)f (con)o(tain)f(information)f(on)h(the)i(completed)f(op)q(eration;)i(if)d(the) 75 1908 y(comm)o(unication)11 b(ob)r(ject)k(w)o(as)g(created)h(b)o(y)e(a)g (non)o(blo)q(c)o(king)f(send)j(or)e(receiv)o(e,)i(then)f(it)f(is)g(deallo)q (cated)h(and)f(the)75 1958 y(handle)e(b)q(ecomes)f(n)o(ull.)17 b(The)12 b(call)f(returns)i Fu(flag=false)p Fx(,)c(otherwise.)19 b(In)11 b(suc)o(h)i(a)e(case,)i(the)f(v)n(alue)f(of)g(the)h(status)75 2008 y(ob)r(ject)j(is)e(unde\014ned.)158 2136 y Fo(Discussion:)158 2186 y Ft(W)m(e)g(could)i(disp)q(ose)f(of)f(the)g(\015ag)h(parameter)f(and)h (add)g(an)f(additional)j(query)e(function)h(for)d(status)i(ob)r(jects.)158 2318 y Fu(MPI)p 227 2318 V 15 w(TEST)c Fx(is)i(a)f(lo)q(cal,)f(non)o(blo)q(c) o(king)g(op)q(eration.)17 b(If)11 b(rep)q(eatedly)h(called)f(for)g(an)g(op)q (eration)h(that)f(is)g(enabled,)75 2368 y(it)j(m)o(ust)e(ev)o(en)o(tually)i (succeed.)158 2418 y(The)g(return)h(status)g(ob)r(ject)f(for)g(a)f(receiv)o (e)i(op)q(eration)f(carries)g(information)d(that)j(can)g(b)q(e)g(accessed)i (using)75 2468 y(the)i(functions)f(describ)q(ed)i(in)e(Section)h(1.3.1.)26 b(The)18 b(status)g(argumen)o(t)e(is)h(not)g(used)h(or)f(up)q(dated)h(for)f (a)g(send)75 2518 y(op)q(eration)d(and)f(an)h(arbitrary)g(v)n(alue)f(can)h(b) q(e)h(passed)g(for)e(this)h(parameter)g(in)f(the)i(call.)158 2568 y(In)h(m)o(ulti-threaded)e(en)o(vironmen)o(t,)h(the)h(use)h(of)e(a)g (blo)q(c)o(king)g(receiv)o(e)i(op)q(eration)f(\()p Fu(MPI)p 1559 2568 V 15 w(WAIT)p Fx(\))f(ma)o(y)f(allo)o(w)75 2617 y(the)e(op)q (erating)f(system)h(to)f(de-sc)o(hedule)i(the)f(blo)q(c)o(k)o(ed)f(thread)i (and)e(sc)o(hedule)h(another)g(thread)g(for)g(execution,)g(if)p eop %%Page: 15 15 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(15)75 45 y(suc)o(h)14 b(is)f(a)o(v)n(ailable.)i(The)e(use)h (of)f(a)f(non)o(blo)q(c)o(king)g(receiv)o(e)i(op)q(eration)f(\()p Fu(MPI)p 1265 45 14 2 v 15 w(TEST)p Fx(\))g(allo)o(ws)e(the)j(user)g(to)f(sc) o(hedule)75 95 y(alternativ)o(e)h(activities)f(within)g(a)h(single)g(thread)g (of)f(execution.)75 195 y(Example:)75 276 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o (m,)e(rank,)i(ierr\))75 325 y(IF\(rank.EQ.0\))119 375 y(THEN)162 425 y(CALL)g(MPI_ISEND\(handle,)d(a\(1\),)j(10,)g(MPI_REAL,)f(1,)i(tag,)f (comm,)f(ierr\))162 475 y(****)h(do)h(some)e(computation)g(to)h(mask)g (latency)g(****)162 525 y(CALL)g(MPI_WAIT\(handle,)e(status,)h(ierr\))119 574 y(ELSE)162 624 y(CALL)h(MPI_IRECV\(handle,)d(a\(1\),)j(15,)g(MPI_REAL,)f (0,)i(tag,)f(comm,)f(ierr\))162 674 y(****)h(do)h(some)e(computation)g(to)h (mask)g(latency)g(****)162 724 y(CALL)g(MPI_WAIT\(handle,)e(status,)h(ierr\)) 75 774 y(END)h(IF)158 855 y Fx(The)16 b(functions)f Fu(MPI)p 494 855 V 15 w(WAIT)g Fx(and)g Fu(MPI)p 760 855 V 15 w(TEST)f Fx(can)i(b)q(e)g(used)g(to)f(complete)g(b)q(oth)g(sends)i(and)e(receiv)o(es;) i(they)75 905 y(will)12 b(b)q(e)j(also)e(used)i(to)f(complete)f(an)o(y)g (other)i(non)o(blo)q(c)o(king)d(comm)o(unication)f(call)i(pro)o(vided)h(b)o (y)f(MPI.)158 954 y(Non)o(blo)q(c)o(king)20 b(comm)o(unications)e(cannot)k(b) q(e)g(replaced)g(b)o(y)f(blo)q(c)o(king)f(comm)o(unications,)g(ev)o(en)i(in)e (the)75 1004 y Fu(synchronous)12 b Fx(comm)o(uni)o(cation)f(mo)q(de.)17 b(Consider)d(the)h(follo)o(wing)c(example:)75 1085 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1135 y(IF\(rank.EQ.0\))119 1185 y(THEN)162 1235 y(CALL)g(MPI_ISEND\(handle1,)d(a\(1\),)j(count,)g (MPI_REAL,)e(1,)j(tag1,)f(comm,)f(ierr\))162 1285 y(CALL)h (MPI_ISEND\(handle2,)d(a\(2\),)j(count,)g(MPI_REAL,)e(1,)j(tag2,)f(comm,)f (ierr\))162 1334 y(CALL)h(MPI_WAIT\(handle1,)d(status,)j(ierr\))162 1384 y(CALL)g(MPI_WAIT\(handle2,)d(status,)j(ierr\))119 1434 y(ELSE)162 1484 y(CALL)g(MPI_IRECV\(handle2,)d(a\(2\),)j(count,)g(MPI_REAL,)e (0,)j(tag2,)f(comm,)f(ierr\))162 1534 y(CALL)h(MPI_IRECV\(handle1,)d(a\(1\),) j(count,)g(MPI_REAL,)e(0,)j(tag1,)f(comm,)f(ierr\))162 1583 y(CALL)h(MPI_WAIT\(handle2,)d(status,)j(ierr\))162 1633 y(CALL)g (MPI_WAIT\(handle1,)d(status,)j(ierr\))75 1683 y(END)g(IF)158 1764 y Fx(The)c(co)q(de)h(is)f(guaran)o(teed)g(to)g(execute)h(correctly)m(,)g (ev)o(en)g(in)e(an)h(implem)o(en)o(tation)d(that)j(do)q(es)g(not)g(bu\013er) 75 1814 y(messages)f(\(the)h(only)e(resource)k(requiremen)o(t)d(is)g(the)g (abilit)o(y)f(to)h(ha)o(v)o(e)g(t)o(w)o(o)g(p)q(ending)g(comm)o(unicatio)o (ns\).)23 b(On)75 1864 y(the)14 b(other)f(hand,)g(the)h(co)q(de)f(ma)o(y)f (deadlo)q(c)o(k)h(if)f(the)h(non)o(blo)q(c)o(king)f(op)q(erations)h(are)h (replaced)g(b)o(y)f(blo)q(c)o(king)f(ones.)75 1979 y Fq(1.8.4)49 b(Multiple)17 b(Completions)75 2056 y Fx(It)f(is)g(con)o(v)o(enien)o(t)h(to)f (b)q(e)h(able)e(to)h(w)o(ait)g(for)g(the)g(completion)f(of)g(an)o(y)h(or)g (all)f(the)i(op)q(erations)f(in)g(a)g(set,)h(rather)75 2106 y(than)e(ha)o(ving)g(to)g(w)o(ait)f(for)h(a)g(sp)q(eci\014c)i(message.)22 b(A)16 b(call)e(to)h Fu(MPI)p 1125 2106 V 15 w(WAITANY)f Fx(or)h Fu(MPI)p 1426 2106 V 16 w(TESTANY)e Fx(can)j(b)q(e)g(used)g(to)75 2156 y(w)o(ait)d(for)g(the)i(completion)d(of)h(one)h(out)f(of)h(sev)o(eral)g (op)q(erations;)f(a)h(call)f(to)g Fu(MPI)p 1336 2156 V 15 w(WAITALL)g Fx(or)g Fu(MPI)p 1634 2156 V 15 w(TESTALL)g Fx(can)75 2206 y(b)q(e)i(used)f(to)g(w)o(ait)f(for)h(all)e(p)q(ending)i(op)q(erations)g(in)g (a)f(list.)158 2334 y Fo(Discussion:)158 2384 y Fm(MPI)p 220 2384 12 2 v 13 w(TESTALL)f Fo(is)j(new.)75 2563 y Fu(MPI)p 144 2563 14 2 v 15 w(WAITANY)20 b(\(count,)h(array)p 618 2563 V 14 w(of)p 676 2563 V 16 w(handles,)f(index,)g(status\))117 2613 y Ft(in)14 b Fu(count)675 b Ft(list)14 b(length)g(\(in)o(teger\))117 2658 y(in)g Fu(array)p 275 2658 V 15 w(of)p 334 2658 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o(unication)i(ob)r(jects)117 2704 y(out)f Fu(index)651 b Ft(index)14 b(of)f(handle)i(for)d(op)q(eration)j (that)e(completed)i(\(in)o(teger\))p eop %%Page: 16 16 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(16)117 45 y Ft(out)14 b Fu(status)629 b Ft(return)13 b(status)h(ob)r(ject)158 121 y Fx(Blo)q(c)o(ks)g(un)o(til)f(one)h(of)f(the)h (op)q(erations)g(asso)q(ciated)h(with)e(the)i(comm)o(uni)o(cation)c(handles)j (in)f(the)h(arra)o(y)g(has)75 171 y(completed.)19 b(Returns)c(the)g(index)g (of)e(that)i(handle)f(in)g(the)h(arra)o(y)m(,)e(and)i(returns)h(the)f(status) g(of)f(that)g(op)q(eration)75 221 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f (with)g(the)g(status.)158 271 y(The)23 b(successful)h(execution)f(of)f Fu(MPI)p 768 271 14 2 v 15 w(WAITANY\(count,)d(array)p 1220 271 V 15 w(of)p 1279 271 V 15 w(handles,)h(index,)h(status\))g Fx(has)75 321 y(the)f(same)f(e\013ect)i(as)e(the)h(successful)h(execution)g (of)d Fu(MPI)p 1017 321 V 16 w(WAIT\(array)p 1253 321 V 13 w(of)p 1310 321 V 15 w(handles[i],)i(status\))p Fx(,)f(where)75 370 y Fu(i)12 b Fx(is)h(the)g(v)n(alue)f(returned)i(b)o(y)e Fu(index)p Fx(.)17 b(In)12 b(particular,)g(the)h(asso)q(ciated)h(comm)o (unicati)o(on)c(ob)r(ject)j(is)f(deallo)q(cated,)75 420 y(and)i(the)h(handle) f(to)g(it)f(in)h Fu(array)p 616 420 V 15 w(of)p 675 420 V 15 w(handles)f Fx(is)h(set)h(to)f(n)o(ull.)k(If)13 b(more)g(then)i(one)g(op)q (eration)f(is)g(enabled)g(and)75 470 y(can)f(terminate,)e(one)i(is)f (arbitrarily)f(c)o(hosen.)18 b(There)c(is)e(no)g(requiremen)o(t)h(that)f(the) h(c)o(hoice)g(satisfy)f(an)o(y)g(fairness)75 520 y(criterion.)158 648 y Fo(Discussion:)158 694 y Ft(If)g(statemen)o(t)i(of)f(fairness)h(is)f(c) o(hanged)i(in)e(Section)i(1.4)e(then)g(will)i(need)e(to)g(c)o(hange)h(here,)f (to)q(o.)75 874 y Fu(MPI)p 144 874 V 15 w(TESTANY\()20 b(count,)h(array)p 618 874 V 14 w(of)p 676 874 V 16 w(handles,)f(index,)g(status\))117 923 y Ft(in)14 b Fu(count)675 b Ft(list)14 b(length)g(\(in)o(teger\))117 968 y(in)g Fu(array)p 275 968 V 15 w(of)p 334 968 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o(unication)i(ob)r(jects)117 1013 y(out)f Fu(index)651 b Ft(index)18 b(of)f(handle)h(for)f(op)q(eration)h (that)f(completed,)i(or)e(-1)g(if)905 1063 y(none)d(completed)g(\(in)o (teger\))117 1107 y(out)g Fu(status)629 b Ft(return)13 b(status)h(ob)r(ject) 158 1184 y Fx(Causes)20 b(either)h(one)f(or)f(none)h(of)f(the)h(op)q (erations)g(asso)q(ciated)g(with)f(the)h(comm)o(unication)c(handles)k(to)75 1233 y(return.)i(In)15 b(the)h(former)e(case,)h(it)g(returns)h(the)g(index)f (of)f(that)h(handle)g(in)f(the)h(arra)o(y)m(,)f(and)h(returns)i(the)e(status) 75 1283 y(of)d(that)h(op)q(eration)f(in)g(the)h(ob)r(ject)h(asso)q(ciated)f (with)f(the)i(status.)k(The)13 b(comm)o(unicatio)o(n)d(ob)r(ject)j(is)g (deallo)q(cated)75 1333 y(and)k(the)h(handle)f(to)g(it)g(in)g Fu(array)p 635 1333 V 15 w(of)p 694 1333 V 15 w(handles)f Fx(is)h(set)h(to)f (n)o(ull.)27 b(In)17 b(the)h(latter)g(case,)g(it)f(returns)i(a)e(v)n(alue)g (of)75 1383 y(-1)e(in)f Fu(index)g Fx(and)h Fu(status)f Fx(is)g(unde\014ned.) 23 b(Lik)o(e)15 b Fu(MPI)p 938 1383 V 15 w(TEST)p Fx(,)e(this)i(is)g(a)g(non) o(blo)q(c)o(king)f(op)q(eration,)g(that)h(returns)75 1433 y(immediately;)f (and)i(if)g(one)g(busy-w)o(aits)h(with)f Fu(MPI)p 904 1433 V 15 w(TESTANY)p Fx(,)e(and)j(some)e(op)q(eration)i(on)f(the)h(list)f(is)g (enabled,)75 1482 y(then)f Fu(MPI)p 239 1482 V 15 w(TESTANY)d Fx(will)g(ev)o(en)o(tually)i(return)h Fu(flag)21 b(=)g(true)p Fx(.)158 1611 y Fo(Discussion:)158 1656 y Ft(Could)e(ha)o(v)o(e)f(a)f Fm(flag)f Ft(parameter,)j(rather)f(than)g(using)h(index)g(=)f(-1)f(for)g (unsuccessful)j(return.)31 b(This)19 b(is)f(more)75 1702 y(consisten)o(t)d (and)e(more)g(elegan)o(t,)h(but)g(requires)g(an)f(additional)k(parameter.)158 1752 y(If)12 b(w)o(e)h(stic)o(k)h(with)f(curren)o(t)h(v)o(ersion,)g(ma)o(y)f (replace)h(-1)f(b)o(y)g(an)h(MPI)f(constan)o(t.)75 1932 y Fu(MPI)p 144 1932 V 15 w(WAITALL\()20 b(count,)h(array)p 618 1932 V 14 w(of)p 676 1932 V 16 w(handles,)f(array)p 998 1932 V 14 w(of)p 1056 1932 V 16 w(status\))117 1981 y Ft(in)14 b Fu(count)675 b Ft(lists)14 b(length)g(\(in)o(teger\))117 2026 y(in)g Fu(array)p 275 2026 V 15 w(of)p 334 2026 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o(unication)i(ob)r(jects)117 2071 y(out)f Fu(array)p 299 2071 V 14 w(of)p 357 2071 V 16 w(status)445 b Ft(arra)o(y)13 b(of)g(status)g(ob)r(jects)158 2147 y Fx(Blo)q(c)o(ks)19 b(un)o(til)e(all)g(comm)o(unicati)o(on)e(op)q(erations)k(asso)q(ciated)g (with)f(handles)g(in)g(the)g(list)g(complete,)g(and)75 2197 y(return)f(the)f(status)g(of)f(all)f(these)j(op)q(erations.)22 b(Both)16 b(arra)o(ys)g(ha)o(v)o(e)f(the)h(same)f(n)o(um)o(b)q(er)f(of)h(v)n (alid)f(en)o(tries.)24 b(The)75 2247 y Fu(i)p Fx(-th)13 b(en)o(try)h(in)f Fu(array)p 432 2247 V 15 w(of)p 491 2247 V 15 w(status)g Fx(is)g(set)h(to)g (the)g(return)g(status)h(of)e(the)h Fu(i)p Fx(-th)f(op)q(eration.)18 b(All)12 b(comm)o(unication)75 2297 y(ob)r(jects)j(are)f(deallo)q(cated)g (and)g(all)f(handles)h(are)g(set)h(to)f(n)o(ull.)75 2394 y Fu(MPI)p 144 2394 V 15 w(TESTALL\(count,)19 b(array)p 596 2394 V 15 w(of)p 655 2394 V 15 w(handles,)h(flag,)h(array)p 1107 2394 V 14 w(of)p 1165 2394 V 16 w(status\))117 2443 y Ft(in)14 b Fu(count)675 b Ft(lists)14 b(length)g(\(in)o(teger\))117 2488 y(in)g Fu(array)p 275 2488 V 15 w(of)p 334 2488 V 15 w(handles)447 b Ft(arra)o(y)13 b(of)g(handles)i(to)e(comm)o(unication)i(ob)r(jects)117 2533 y(out)f Fu(flag)673 b Ft(\(logical\))117 2577 y(out)14 b Fu(array)p 299 2577 V 14 w(of)p 357 2577 V 16 w(status)445 b Ft(arra)o(y)13 b(of)g(status)g(ob)r(jects)158 2654 y Fx(Causes)h(either)g (all)e(or)h(none)h(of)e(the)i(op)q(erations)g(asso)q(ciated)g(with)e(the)i (comm)o(unication)c(ahndles)j(to)g(com-)75 2704 y(plete.)36 b(It)20 b(returns)h Fu(flag)g(=)g(true)e Fx(if)g(all)f(comm)o(unications)f (asso)q(ciated)j(with)f(handles)h(in)g(the)g(arra)o(y)f(ha)o(v)o(e)p eop %%Page: 17 17 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(17)75 45 y(completed.)18 b(In)c(this)h(case,)f(eac)o(h)h (status)g(en)o(try)g(is)f(set)h(to)f(the)h(status)g(of)e(the)i(corresp)q (onding)g(comm)o(unicatio)o(n.)75 95 y(All)10 b(comm)o(unicatio)o(n)e(ob)r (jects)k(are)f(deallo)q(cated,)g(and)f(all)f(handles)i(are)g(set)h(to)e(n)o (ull.)16 b(Otherwise,)c Fu(flag)21 b(=)h(false)75 145 y Fx(is)14 b(returned)h(and)f(the)h(v)n(alues)f(of)f(the)h(status)h(en)o(tries)g(are)g (unde\014ned.)k(This)14 b(is)g(a)f(non)o(blo)q(c)o(king)g(op)q(eration)h (that)75 195 y(returns)20 b(immediately;)e(if)f(one)i(busy)g(w)o(aits)g(with) f Fu(MPI)p 986 195 14 2 v 15 w(TESTALL)p Fx(,)f(and)i(all)e(op)q(erations)i (in)f(the)i(list)e(b)q(ecome)75 244 y(enabled,)c(then)g Fu(MPI)p 402 244 V 15 w(TESTALL)f Fx(will)f(ev)o(en)o(tually)h(return)j Fu(flag)21 b(=)g(true)p Fx(.)158 344 y(Example:)75 477 y Fu(CALL)g (MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 527 y(IF\(rank)f(<)i(2\))119 576 y(THEN)162 626 y(CALL)f(MPI_ISEND\(handle,)d(a,)k(n,)f(MPI_REAL,)f(2,)h (tag,)g(comm,)g(ierr\))162 676 y(****)g(do)h(some)e(computation)g(to)h(mask)g (latency)g(****)162 726 y(CALL)g(MPI_WAIT\(handle,)e(status,)h(ierr\))119 776 y(ELSE)86 b(!)22 b(rank.EQ.2)162 826 y(CALL)f(MPI_IRECV\(handle_l)o (ist\(0)o(\),)e(a,)i(n,)h(MPI_REAL,)d(0,)j(tag,)f(comm,)g(ierr\))162 875 y(CALL)g(MPI_IRECV\(handle_l)o(ist\(1)o(\),)e(b,)i(n,)h(MPI_REAL,)d(1,)j (tag,)f(comm,)g(ierr\))162 925 y(****)g(do)h(some)e(computation)g(to)h(mask)g (latency)g(****)162 975 y(CALL)g(MPI_WAITANY\(2,)e(handle_list,)g(index,)i (status,)f(ierr\))162 1025 y(IF\(index.EQ.0\))206 1075 y(THEN)249 1124 y(****)h(handle)g(message)f(from)h(process)g(0)g(****)249 1174 y(CALL)g(MPI_WAIT\(handle_lis)o(t\(1\),)d(status,)i(ierr\))249 1224 y(****)h(handle)g(message)f(from)h(process)g(1)g(****)206 1274 y(ELSE)249 1324 y(****)g(handle)g(message)f(from)h(process)g(1)g(****) 249 1373 y(CALL)g(MPI_WAIT\(handle_lis)o(t\(0\),)d(status,)i(ierr\))249 1423 y(****)h(handle)g(message)f(from)h(process)g(0)g(****)162 1473 y(END)g(IF)75 1523 y(END)g(IF)158 1606 y Fx(The)d(calls)f(in)o(tro)q (duced)h(in)f(this)h(subsection)h(can)e(b)q(e)i(used)f(to)f(w)o(ait)g(or)h (test)g(for)f(the)h(completion)e(of)h(an)75 1656 y(arbitrary)f(mix)f(of)h (non)o(blo)q(c)o(king)f(comm)o(unicatio)o(n)f(calls,)i(including)g(a)g(mix)e (sends)k(and)e(receiv)o(es)i(and)f(of)e(an)o(y)75 1706 y(additional)d(non)o (blo)q(c)o(king)h(comm)o(unicati)o(on)e(calls)j(pro)o(vided)f(b)o(y)h(MPI.) 158 1755 y(The)e Fu(status)f Fx(argumen)o(t)f(in)h Fu(MPI)p 682 1755 V 16 w(WAITANY)f Fx(and)h Fu(MPI)p 1006 1755 V 15 w(WAITALL)g Fx(and)g(the)h Fu(array)p 1443 1755 V 15 w(of)p 1502 1755 V 15 w(status)f Fx(argumen)o(t)f(in)75 1805 y Fu(MPI)p 144 1805 V 15 w(WAITALL)i Fx(and)h Fu(MPI)p 471 1805 V 15 w(TESTALL)f Fx(are)i(not)f(used)i(or)e(up)q(dated)h(if)f(the)h(completing)d(comm)o (unication)f(is)j(a)h(send.)75 1855 y(If)h(all)e(op)q(erations)i(in)g(the)g (list)g(pro)o(vided)g(b)o(y)f(the)i(argumen)o(t)e Fu(array)p 1175 1855 V 14 w(of)p 1233 1855 V 16 w(handles)f Fx(are)i(send)h(op)q (erations,)f(then)75 1905 y(an)f(arbitrary)f(v)n(alue)h(can)g(b)q(e)g(passed) h(to)f(the)g Fu(status)f Fx(\(resp.)19 b Fu(array)p 1181 1905 V 15 w(of)p 1240 1905 V 15 w(status)p Fx(\))13 b(argumen)o(t.)158 2033 y Fo(Discussion:)158 2079 y Ft(Is)g(the)g(last)h(paragraph)g(agreed?)158 2124 y(W)m(e)h(ma)o(y)g(w)o(an)o(t)g(to)g(allo)o(w)h(n)o(ull)g(p)q(oin)o (ters)h(in)e(an)h(arra)o(y)p 975 2124 12 2 v 14 w(of)p 1020 2124 V 14 w(handles)h(This)e(will)i(allo)o(w)f(to)e(lo)q(op)j(with)e(the)g (same)g(ar-)75 2170 y(ra)o(y)p 130 2170 V 14 w(of)p 175 2170 V 14 w(handles,)e(un)o(til)g(all)g(comm)o(unications)h(ha)o(v)o(e)e (completed.)18 b(On)11 b(the)h(other)g(hand,)g(this)g(will)h(encourage)g(p)q (oten)o(tially)75 2216 y(ine\016cien)o(t)i(co)q(de.)158 2266 y(I)10 b(added)h(the)f(requiremen)o(t)i(that)e(the)h(handel)h(to)e(a)g(comm)o (unication)j(that)d(completes)h(is)g(set)f(to)g(n)o(ull.)18 b(This)11 b(prev)o(en)o(ts)75 2315 y(the)i(o)q(ccurence)h(of)f(dangling)j (references,)d(but)g(add)h(some)f(o)o(v)o(erhead.)75 2535 y Fw(1.9)59 b(Prob)r(e)20 b(and)f(Cancel)75 2626 y Fx(The)h Fu(MPI)p 235 2626 14 2 v 15 w(PROBE)e Fx(and)h Fu(MPI)p 530 2626 V 15 w(IPROBE)f Fx(op)q(erations)i(allo)o(w)d(incoming)g(messages)j(to)f(b)q(e)h (c)o(hec)o(k)o(ed)g(for,)g(without)75 2676 y(actually)15 b(receiving)g(them.) 23 b(The)15 b(user)i(can)f(then)g(decide)g(ho)o(w)g(to)f(receiv)o(e)i(them,)d (based)i(on)g(the)g(information)p eop %%Page: 18 18 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(18)75 45 y(returned)16 b(b)o(y)d(the)i(prob)q(e)g(\(basically) m(,)d(the)i(information)d(returned)16 b(b)o(y)e Fu(status)p Fx(\).)j(In)d(particular,)f(the)i(user)g(ma)o(y)75 95 y(allo)q(cate)e(memory) f(for)h(the)i(receiv)o(e)g(bu\013er,)f(according)g(to)g(the)h(length)e(of)h (the)g(prob)q(ed)h(message.)158 145 y(The)e Fu(MPI)p 311 145 14 2 v 15 w(CANCEL)e Fx(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o (unicatio)o(ns)d(to)i(b)q(e)h(cancelled.)19 b(This)12 b(is)g(required)h(for) 75 195 y(clean)o(up.)25 b(P)o(osting)16 b(a)f(send)i(or)g(a)e(receiv)o(e)j (ties)e(up)h(user)g(resources)h(\(send)f(or)f(receiv)o(e)i(bu\013ers\),)g (and)d(a)h(cancel)75 244 y(ma)o(y)c(b)q(e)i(needed)i(to)e(free)g(these)i (resources)g(gracefully)m(.)158 373 y Fo(Discussion:)158 423 y Ft(Made)j(one)g(c)o(hange)h(from)e(appro)o(v)o(ed)i(draft)f(on)g(prob)q(e:) 29 b(remo)o(v)o(ed)19 b(the)g(datat)o(yp)q(e)h(argumen)o(t)f(from)g(prob)q (e.)34 b(If)75 472 y(generalized)15 b(datat)o(yp)q(e)f(are)e(adopted,)i(then) f(it)g(seems)g(more)f(natural)i(to)f(pro)o(vide)h(the)e(argumen)o(t)i(in)f (the)g(status)g(deco)q(de)75 522 y(function.)75 702 y Fu(MPI)p 144 702 V 15 w(IPROBE\(source,)19 b(tag,)i(comm,)g(flag,)f(status\))117 752 y Ft(in)14 b Fu(source)653 b Ft(source)13 b(rank,)h(or)f Fm(MPI)p 1273 752 12 2 v 13 w(ANY)p 1346 752 V 13 w(SOURCE)d Ft(\(in)o(teger\))117 797 y(in)k Fu(tag)719 b Ft(tag)13 b(v)n(alue)h(or)f Fm(MPI)p 1221 797 V 13 w(ANY)p 1294 797 V 13 w(TAG)f Ft(\(in)o(teger\))117 843 y(in)i Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 889 y(out)h Fu(flag)673 b Ft(\(logical\))117 934 y(out)14 b Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1011 y Fu(MPI)p 227 1011 14 2 v 15 w(IPROBE)h Fx(returns)i Fu(flag)21 b(=)h(true)14 b Fx(if)g(there)i(is)f(a)g(message)f(that)h(can)h(b)q(e)f(receiv)o(ed)i(and)d (that)h(matc)o(hes)75 1060 y(the)d(pattern)g(sp)q(eci\014ed)g(b)o(y)f(the)h (parameters)f Fu(source)p Fx(,)f Fu(tag)p Fx(,)g(and)h Fu(comm)p Fx(.)16 b(It)11 b(returns)i Fu(flag)21 b(=)g(false)p Fx(,)10 b(otherwise.)75 1110 y(If)k Fu(MPI)p 186 1110 V 15 w(IPROBE)f Fx(returns)k Fu(flag)k(=)g(true)p Fx(,)13 b(then)j(the)f(source)h(and)e(tag)g (of)g(the)h(message)f(matc)o(hed)g(are)h(returned)75 1160 y(in)c(the)i (status)g(ob)r(ject.)18 b(These)13 b(are)g(the)f(same)f(v)n(alues)h(that)g(w) o(ould)f(ha)o(v)o(e)h(b)q(een)h(returned)g(b)o(y)f(a)g(call)f(to)h Fu(MPI)p 1776 1160 V 15 w(RECV)75 1210 y Fx(executed)k(at)e(the)g(same)f(p)q (oin)o(t)g(in)h(the)g(program.)158 1260 y(These)19 b(v)n(alues)e(can)h(b)q(e) g(deco)q(ded)h(from)d(the)i(return)h(status)f(ob)r(ject)h(using)e(the)h (status)h(query)f(functions)75 1310 y(describ)q(ed)e(in)d(Section)h(1.3.1.)j (The)d(return)h(status)g(ob)r(ject)f(is)g(unde\014ned)h(if)e Fu(flag=false)p Fx(.)158 1359 y(The)20 b(call)e(to)h Fu(MPI)p 455 1359 V 15 w(PROBE)f Fx(do)q(es)i(not)f(carry)h(a)f(datat)o(yp)q(e)g (argumen)o(t:)28 b(the)19 b(message)g(t)o(yp)q(e)h(ma)o(y)d(not)i(b)q(e)75 1409 y(kno)o(wn)13 b(when)h(the)f(prob)q(e)h(is)g(executed.)19 b(Accordingly)m(,)13 b(the)g(prob)q(e)h(call)f(do)q(es)h(not)f(return)h(in)f (the)h(status)g(ob)r(ject)75 1459 y(the)i(length)f(of)f(the)i(prob)q(ed)g (message,)e(and)h Fu(MPI)p 861 1459 V 15 w(GET)p 942 1459 V 15 w(COUNT)f Fx(cannot)h(b)q(e)h(used)g(after)f(a)g(call)f(to)h Fu(MPI)p 1702 1459 V 15 w(PROBE)f Fx(to)75 1509 y(query)h(on)f(the)i(length)e (of)g(the)h(prob)q(ed)h(message.)k(\(Remem)o(b)q(er)13 b(that)h(the)i(length) e(of)g(a)g(message,)g(in)h(elemen)o(ts,)75 1559 y(dep)q(ends)k(on)d(the)i(t)o (yp)q(es)g(of)e(these)j(elemen)o(ts;)f(some)e(implemen)o(tatio)o(ns)f(ma)o(y) g(not)i(carry)g(in)g(the)g(transmitted)75 1608 y(message)c(t)o(yp)q(e)g (information.)o(\))j(Instead,)d(a)g(di\013eren)o(t)g(status)h(deco)q(ding)f (function)f(that)h(pro)o(vides)g(the)h(datat)o(yp)q(e)75 1658 y(as)g(argumen)o(t)f(can)h(b)q(e)g(used)h(in)f(this)f(case.)75 1755 y Fu(MPI)p 144 1755 V 15 w(PROBE)p 269 1755 V 15 w(COUNT\(status,)19 b(datatype,)h(count\))117 1805 y Ft(in)14 b Fu(status)653 b Ft(status)13 b(ob)r(ject)117 1851 y(in)h Fu(datatype)609 b Ft(message)13 b(datat)o(yp)q(e)117 1896 y(out)h Fu(count)651 b Ft(n)o(um)o(b)q(er)14 b(of)e(en)o(tries)i(in)g(message)g(\(in)o(teger\))158 1973 y Fx(Returns)h(the)f(length)g(of)f(the)h(message)f(that)h(w)o(as)g(last) f(prob)q(ed)i(with)e Fu(status)f Fx(as)i(an)g(argumen)o(t.)j(The)d(call)75 2023 y(is)g(erroneous)h(if)e Fu(datatype)f Fx(do)q(es)j(not)f(matc)o(h)e(the) j(t)o(yp)q(e)f(of)g(the)g(message.)158 2151 y Fo(Discussion:)158 2197 y Ft(Message)19 b(passing)g(in)g(MPI)e(can)i(b)q(e)f(implemen)o(ted)i (without)e(app)q(ending)j(t)o(yp)q(e)d(information)i(to)d(messages.)32 b(A)75 2242 y(message)12 b(is)g(merely)h(a)e(string)i(of)e(b)o(ytes)h(and)g (the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h(a)e(sequence)i (of)e(t)o(yp)q(ed)i(elemen)o(ts)f(is)75 2288 y(done)i(using)i(the)d(datat)o (yp)q(e)i(information)h(pro)o(vided)g(b)o(y)e(the)g(lo)q(cal)h(comm)o (unication)h(call.)21 b(The)13 b(abilit)o(y)j(to)e(use)g(suc)o(h)g(an)75 2333 y(implemen)o(tation)k(strategy)d(is)h(deemed)f(to)g(b)q(e)g(an)g(imp)q (ortan)o(t)h(goal.)23 b(In)15 b(suc)o(h)g(an)g(implemen)o(tation)q(,)i(when)e (a)g(message)75 2379 y(arriv)o(es,)f(it)f(is)g(not)h(kno)o(wn)f(ho)o(w)g(man) o(y)g(elemen)o(ts)h(it)g(con)o(tains,)g(or)f(ev)o(en)g(ho)o(w)g(m)o(uc)o(h)g (storage)h(is)f(needed)h(to)f(receiv)o(e)h(that)75 2425 y(message)g(\(b)q (ecause)g(of)e(p)q(ossible)k(represen)o(tation)f(con)o(v)o(ersion)g(in)f(a)f (heterogeneous)i(en)o(vironmen)o(t\).)k(The)13 b(length)h(of)f(the)75 2470 y(incoming)18 b(message)e(can)g(b)q(e)g(computed)h(b)o(y)f Fm(MPI)p 818 2470 12 2 v 13 w(PROBE)p 931 2470 V 12 w(LEN)e Ft(from)i(the)f(b)o(yte)h(length)h(of)f(the)g(message)g(and)g(from)g(the)75 2516 y(datat)o(yp)q(e)e(parameter.)158 2562 y(The)h(curren)o(t)g(solution)i (sa)o(v)o(es)f(us)f(the)g(need)h(for)e(one)i(additional)i(w)o(ord)d(p)q(er)g (message)h(that)f(w)o(ould)h(otherwise)g(b)q(e)75 2607 y(needed)e(to)f (transfer)g(the)g(message)h(length)g(\(in)g(elemen)o(ts\))g(with)g(the)f (message.)158 2653 y Fo(Need)h(b)q(e)h(discussed)g(b)o(y)g(MPIF)p eop %%Page: 19 19 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(19)158 45 y Ft(W)m(e)15 b(ma)o(y)g(w)o(an)o(t)g(a)f(call)j(to) d Fm(MPI)p 623 45 12 2 v 13 w(PROBE)p 736 45 V 13 w(COUNT)e Ft(to)j(b)q(e)g(signi\014can)o(t)i(not)e(just)g(after)g(prob)q(e,)h(but)f (also)h(after)e(receiv)o(e,)75 91 y(for)f(consistency)i(reasons.)158 136 y(It's)k(tempting)h(to)f(merge)h Fm(MPI)p 637 136 V 13 w(GET)p 710 136 V 13 w(COUNT)d Ft(and)j Fm(MPI)p 981 136 V 13 w(PROBE)p 1094 136 V 12 w(COUNT)d Ft(in)o(to)j(one)g(call.)37 b(E.g.,)20 b(alw)o(a)o(ys)g(pro)o(vide)h(a)75 182 y(datat)o(yp)q(e)15 b(argumen)o(t,)f(ev)o(en)g(though)h(it)f(is)g(redundan)o(t)i(after)d(a)h (receiv)o(e.)19 b(But)14 b(then)g(w)o(e)g(imp)q(ose)h(an)f(additional)j (burden)75 228 y(on)c(regular)i(receiv)o(es.)158 273 y(W)m(e)d(ma)o(y)h(w)o (an)o(t)f(to)g(relax)h(t)o(yp)q(e)g(restrictions)h(so)e(that)h Fm(datatype)j(=)k(MPI)p 1230 273 V 13 w(BYTE)10 b Ft(could)k(b)q(e)e(used)h (as)f(an)h(argumen)o(t)g(to)75 319 y Fm(MPI)p 137 319 V 13 w(PROBE)p 250 319 V 12 w(COUNT)p Ft(,)d(ev)o(en)o(t)k(if)f(the)g(message)g(w) o(as)g(sen)o(t)g(with)g(another)g(datat)o(yp)q(e;)h(this)g(will)g(allo)o(w)g (to)f(coun)o(t)g(the)g(n)o(um)o(b)q(er)75 365 y(of)g(b)o(ytes)g(receiv)o(ed,) h(and)g(use)f(this)h(coun)o(t)g(in)f(a)g(mallo)q(c.)158 497 y Fx(A)h(subsequen)o(t)j(receiv)o(e)e(executed)h(with)e(the)h(same)f(con)o (text,)g(and)h(the)g(source)g(and)f(tag)g(returned)i(b)o(y)e(the)75 547 y(call)f(to)h Fu(MPI)p 271 547 14 2 v 15 w(IPROBE)f Fx(will)f(receiv)o(e) k(the)e(message)g(that)g(w)o(as)g(matc)o(hed)f(b)o(y)h(the)g(prob)q(e,)h(if)e (no)g(other)i(in)o(terv)o(ening)75 597 y(receiv)o(e)c(o)q(ccurred)g(after)f (the)g(prob)q(e.)18 b(If)9 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o (ulti-threaded,)g(it)g(is)h(the)g(user)g(resp)q(onsibilit)o(y)75 647 y(to)k(ensure)h(that)f(the)h(last)e(condition)g(holds.)158 775 y Fo(Discussion:)158 821 y Ft(MPI)d(guaran)o(tees)h(that)e(successiv)o(e) j(messages)e(sen)o(t)g(from)f(a)h(source)g(to)g(a)g(destination)i(within)f (the)f(same)g(con)o(text)g(are)75 866 y(receiv)o(ed)i(in)f(the)g(order)g (they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h (either)f(explicitly)j(or)c(implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75 912 y(to)i(manage)h(messages)g(b)q(et)o(w)o(een)f(eac)o(h)h(pair)g(of)f(pro)q (cesses,)h(for)f(eac)o(h)g(con)o(text.)17 b Fm(MPI)p 1309 912 12 2 v 13 w(PROBE)10 b Ft(returns)j(information)h(on)f(the)75 958 y(\014rst)i(matc)o(hing)g(message)g(in)h(this)f(FIF)o(O;)e(this)j(will)g (also)f(b)q(e)g(the)f(message)h(receiv)o(ed)h(b)o(y)f(the)f(\014rst)h (subsequen)o(t)h(receiv)o(e)75 1003 y(with)d(the)h(same)f(source,)g(tag)g (and)h(con)o(text)f(as)h(the)f(message)g(matc)o(hed)h(b)o(y)f Fm(MPI)p 1261 1003 V 13 w(PROBE)p Ft(.)75 1183 y Fu(MPI)p 144 1183 14 2 v 15 w(PROBE\()21 b(source,)f(tag,)h(comm,)g(datatype,)f(status\)) 117 1233 y Ft(in)14 b Fu(source)653 b Ft(source)13 b(rank,)h(or)f Fm(MPI)p 1273 1233 12 2 v 13 w(ANY)p 1346 1233 V 13 w(SOURCE)d Ft(\(in)o(teger\))117 1279 y(in)k Fu(tag)719 b Ft(tag)13 b(v)n(alue,)h(or)f Fm(MPI)p 1232 1279 V 13 w(ANY)p 1305 1279 V 13 w(TAG)e Ft(\(in)o(teger\))117 1324 y(in)j Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 1370 y(in)h Fu(datatype)609 b Ft(assumed)14 b(t)o(yp)q(e)f(of)g(data)g(in)h (message)g(\(status\))117 1416 y(out)g Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1492 y Fu(MPI)p 227 1492 14 2 v 15 w(PROBE)g Fx(b)q(eha)o(v)o(es)i(lik)o(e)f Fu(MPI)p 663 1492 V 15 w(IPROBE)f Fx(except)i(that)g(it)f(is)g(a)g(blo)q(c)o(king)f(call)g(whic)o(h)i(returns)g (only)f(after)g(a)75 1542 y(matc)o(hing)e(message)i(has)g(b)q(een)h(found.) 158 1670 y Fo(Discussion:)158 1716 y(CANCEL)f(has)h(b)q(een)g(c)o(hanged)f (since)h(last)h(draft)75 1895 y Fu(MPI)p 144 1895 V 15 w(CANCEL\(handle\))117 1945 y Ft(in)e Fu(handle)653 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)158 2021 y Fx(A)g(call)f(to)g Fu(MPI)p 404 2021 V 15 w(CANCEL)g Fx(marks)f(for)h(cancellation)g(a)h(p)q(ending)g(non)o(blo)q(c)o (king)e(comm)o(unicatio)o(n)f(op)q(eration)75 2071 y(\(send)h(or)f(receiv)o (e\).)19 b(The)13 b(cancel)g(call)f(is)h(non-blo)q(c)o(king,)e(and)i(lo)q (cal.)k(It)c(returns)h(immediately)l(,)c(p)q(ossibly)i(b)q(efore)75 2121 y(the)i(comm)o(unication)d(is)j(actually)f(cancelled.)158 2171 y(It)f(is)g(still)f(necessary)j(to)e(complete)g(a)f(comm)o(unication)e (that)j(has)g(b)q(een)h(mark)o(ed)e(for)h(cancellation,)f(using)h(a)75 2221 y(call)f(to)h Fu(MPI)p 267 2221 V 15 w(WAIT)f Fx(or)g Fu(MPI)p 495 2221 V 16 w(TEST)f Fx(\(or)i(an)o(y)g(of)f(the)h(deriv)o(ed)h (op)q(erations\).)k(If)12 b(the)g(op)q(eration)g(has)g(b)q(een)h(cancelled,) 75 2271 y(then)h(information)d(ab)q(out)j(to)g(that)g(e\013ect)h(will)d(b)q (e)j(returned)g(in)f(the)g(status)h(argumen)o(t)e(of)g(the)h(op)q(eration)g (that)75 2320 y(completes)19 b(the)h(comm)o(unicatio)o(n.)31 b(If)19 b(a)g(comm)o(unicatio)o(n)e(is)i(mark)o(ed)f(for)h(cancellation,)g (then)h(a)f Fu(MPI)p 1776 2320 V 15 w(WAIT)75 2370 y Fx(call)d(for)g(that)h (comm)o(unication)d(is)i(guaran)o(teed)i(to)e(return,)i(irresp)q(ectiv)o(e)h (of)d(the)i(status)f(of)f(other)i(pro)q(cesses;)75 2420 y(similarly)13 b(if)h Fu(MPI)p 355 2420 V 15 w(TEST)h Fx(is)h(rep)q(eatedly)h(called)e(in)g (a)g(busy)h(w)o(ait)f(lo)q(op)g(for)g(a)h(cancelled)g(comm)o(unication,)c (then)75 2470 y Fu(MPI)p 144 2470 V 15 w(TEST)h Fx(will)f(ev)o(en)o(tually)i (b)q(e)g(successful.)158 2520 y(Either)21 b(the)g(cancellation)f(succeeds,)k (or)c(the)h(comm)o(unication)c(succeeds,)24 b(but)d(not)f(b)q(oth.)38 b(If)20 b(a)g(send)75 2569 y(is)e(mark)o(ed)f(for)h(cancellation,)g(then)h (it)e(m)o(ust)g(b)q(e)i(the)g(case)g(that)f(either)h(the)g(send)g(completes)f (normally)m(,)d(in)75 2619 y(whic)o(h)e(case)h(the)g(message)e(sen)o(t)i(w)o (as)f(receiv)o(ed)h(at)f(the)h(destination)f(pro)q(cess,)h(or)f(that)g(the)h (send)g(is)f(successfully)75 2669 y(cancelled,)g(in)e(whic)o(h)h(case)h(no)f (part)g(of)f(the)i(message)f(w)o(as)g(receiv)o(ed)h(at)f(the)h(destination.)k (Then,)12 b(an)o(y)g(matc)o(hing)p eop %%Page: 20 20 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(20)75 45 y(receiv)o(e)16 b(has)f(to)f(b)q(e)h(satis\014ed)h(b) o(y)e(another)h(send.)22 b(If)14 b(a)g(receiv)o(e)i(is)f(mark)o(ed)e(for)h (cancellation,)g(then)h(it)g(m)o(ust)e(b)q(e)75 95 y(the)i(case)h(that)f (either)h(the)f(receiv)o(e)i(completes)d(normally)m(,)e(or)j(that)g(the)g (receiv)o(e)h(is)f(successfully)h(cancelled,)f(in)75 145 y(whic)o(h)g(case)g (no)g(part)g(of)f(the)h(receiv)o(e)h(bu\013er)g(is)e(altered.)21 b(Then,)15 b(an)o(y)g(matc)o(hing)d(send)k(has)f(to)f(b)q(e)i(satis\014ed)f (b)o(y)75 195 y(another)f(receiv)o(e.)75 292 y Fu(MPI)p 144 292 14 2 v 15 w(TEST)p 247 292 V 15 w(CANCELLED\(status,)k(flag\))117 341 y Ft(in)c Fu(status)653 b Ft(return)13 b(status)h(ob)r(ject)117 387 y(in)g Fu(flag)697 b Ft(\(logical\))158 463 y Fx(Returns)16 b Fu(flag)21 b(=)g(true)14 b Fx(if)g(the)i(comm)o(unicatio)o(n)c(asso)q (ciated)k(with)e(the)i(return)g(status)f(ob)r(ject)h(w)o(as)f(can-)75 513 y(celled)f(successfully)m(.)19 b(In)14 b(suc)o(h)g(a)g(case,)g(all)f (other)h(\014elds)g(of)f Fu(status)g Fx(\(suc)o(h)i(as)e Fu(length)g Fx(or)h Fu(tag)p Fx(\))f(are)h(unde\014ned.)75 563 y(Returns)j Fu(flag)k(=)h(false)p Fx(,)15 b(otherwise.)27 b(If)16 b(a)g(receiv)o(e)i(op)q (eration)e(migh)o(t)f(b)q(e)i(cancelled)g(then)g(one)g(should)f(call)75 613 y Fu(MPI)p 144 613 V 15 w(TEST)p 247 613 V 15 w(CANCELLED)g Fx(\014rst,)k(to)e(c)o(hec)o(k)h(whether)h(the)e(op)q(eration)g(w)o(as)g (cancelled,)i(b)q(efore)f(c)o(hec)o(king)f(on)g(the)75 663 y(other)d(v)n(alues)e(of)g(the)i(return)g(status.)75 800 y Fw(1.10)59 b(P)n(ersistent)19 b(comm)n(un)o(ication)e(objects)75 969 y Fo(Discussion:)158 1019 y(there)e(are)h(some)e(c)o(hanges)h(here)g(not) g(y)o(et)g(appro)o(v)o(ed)f(b)o(y)h(MPIF)158 1152 y Fx(Often)i(a)f(comm)o (uni)o(cation)d(with)j(the)h(same)e(parameter)h(list)f(is)h(rep)q(eatedly)i (executed)g(within)d(the)i(inner)75 1201 y(lo)q(op)g(of)h(a)g(parallel)f (computation.)29 b(In)18 b(suc)o(h)h(a)e(situation,)i(it)e(ma)o(y)f(b)q(e)j (p)q(ossible)g(to)f(optimize)e(the)j(comm)o(u-)75 1251 y(nication)c(b)o(y)h (binding)g(the)h(list)e(of)h(comm)o(unicatio)o(n)e(parameters)i(to)g(the)h (comm)o(unicatio)o(n)d(ob)r(ject)j(once)g(and,)75 1301 y(then,)f(rep)q (eatedly)g(using)e(the)i(comm)o(unication)c(handle)j(to)f(initiate)h(and)f (complete)h(messages.)21 b(The)16 b(comm)o(u-)75 1351 y(nication)g(handle)g (th)o(us)h(created)h(can)f(b)q(e)g(though)o(t)f(of)g(as)h(a)f(comm)o (unication)d(p)q(ort)k(or)g(a)f(\\half-c)o(hannel")f(.)26 b(It)75 1401 y(do)q(es)18 b(not)f(pro)o(vide)g(the)g(full)f(functionalit)o(y)f(of)i (a)g(con)o(v)o(en)o(tional)f(c)o(hannel,)h(since)h(there)g(is)f(no)g(binding) f(of)g(the)75 1451 y(send)i(p)q(ort)f(to)g(the)g(receiv)o(e)h(p)q(ort:)24 b(this)17 b(construct)i(allo)o(ws)c(reduction)j(of)e(the)i(o)o(v)o(erhead)f (for)f(comm)o(unication)75 1500 y(b)q(et)o(w)o(een)f(pro)q(cessor)g(and)e (comm)o(unication)d(con)o(troller,)k(but)f(not)h(the)g(o)o(v)o(erhead)f(for)h (comm)o(uni)o(cation)d(b)q(et)o(w)o(een)75 1550 y(one)j(comm)o(unication)d (con)o(troller)i(and)h(another.)158 1600 y(A)g(comm)o(unication)c(ob)r(ject) 15 b(is)f(created)h(using)f(one)g(of)f(the)i(four)e(follo)o(wing)e(calls.)18 b(These)d(calls)f(in)o(v)o(olv)o(e)e(no)75 1650 y(comm)o(unication.)158 1700 y(The)k(function)f Fu(MPI)p 478 1700 V 15 w(CREATE)p 625 1700 V 14 w(SEND)g Fx(creates)i(a)e(comm)o(uni)o(cation)d(ob)r(ject)k(for)f (a)g(standard)h(mo)q(de)e(send)i(op-)75 1749 y(eration,)d(and)h(binds)g(to)g (it)f(all)g(the)h(parameters)g(of)g(a)f(send)i(op)q(eration.)75 1846 y Fu(MPI)p 144 1846 V 15 w(CREATE)p 291 1846 V 15 w(SEND\(handle,)k (start,)h(count,)h(datatype,)f(dest,)h(tag,)g(comm\))117 1896 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 1942 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 1988 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er) 14 b(of)e(elemen)o(ts)i(sen)o(t)g(\(in)o(teger\))117 2033 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 2079 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 2125 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2170 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 2247 y Fx(The)g(call)e(allo)q(cates)h(a)g(new)h(comm)o(unicati)o(on)c(ob)r (ject)k(and)f(asso)q(ciates)i(the)e(comm)o(unication)d(handle)j(with)75 2296 y(it.)158 2425 y Fo(Discussion:)158 2470 y Ft(Tw)o(o)e(c)o(hanges:)158 2516 y(1.)29 b Fm(buffer)p 339 2516 12 2 v 12 w(handle)15 b Ft(is)j(replaced)g(with)g Fm(start,)f(count,)h(datatype)p Ft(,)c(to)j(b)q(e)h (in)g(sync)g(with)f(the)h(new)f(datat)o(yp)q(e)75 2562 y(prop)q(osal.)158 2607 y(2.)25 b(T)m(o)q(ok)15 b(out)h Fm(persistenc)o(e)p Ft(.)22 b(Do)q(es)16 b(not)g(seem)f(v)o(ery)h(useful)h(here.)25 b(Instead,)17 b(I)e(in)o(tro)q(duce)i(in)g(the)f(last)g(section,)75 2653 y(a)e(\\univ)o(ersal")j(comm)o(unication)f(function,)f(whic)o(h)g(w)o(ould)g (not)f(usually)i(used)f(directly)g(b)o(y)f(programmers,)h(but)f(can)h(b)q(e) 75 2699 y(used)f(to)f(de\014ne)h(the)f(other)g(comm)o(unication)j(functions.) p eop %%Page: 21 21 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(21)158 45 y(The)14 b(function)f Fu(MPI)p 474 45 14 2 v 16 w(CREATE)p 622 45 V 14 w(RSEND)g Fx(creates)i(a)e(comm)o (unication)e(ob)r(ject)j(for)f(a)h(ready)g(mo)q(de)e(send)j(op)q(era-)75 95 y(tion.)75 192 y Fu(MPI)p 144 192 V 15 w(CREATE)p 291 192 V 15 w(RSEND\(handle,)k(start,)h(count,)h(datatype,)f(dest,)g(tag,)h(comm\)) 117 242 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j (ob)r(ject)117 287 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send) i(bu\013er)117 333 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(sen)o(t)g(\(in)o(teger\))117 378 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 424 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 469 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 515 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 591 y Fx(The)g(function)f Fu(MPI)p 476 591 V 15 w(CREATE)p 623 591 V 14 w(SSEND)f Fx(creates)j(a)e(comm)o(unication)d(ob)r(ject)k(for)f (a)g(sync)o(hronous)h(mo)q(de)e(send)75 641 y(op)q(eration.)75 738 y Fu(MPI)p 144 738 V 15 w(CREATE)p 291 738 V 15 w(SSEND\(handle,)19 b(start,)h(count,)h(datatype,)f(dest,)g(tag,)h(comm\))117 788 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)117 833 y(in)e Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)117 879 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(sen)o(t)g(\(in)o(teger\))117 924 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 970 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 1015 y(in)e Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 1061 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 1137 y Fx(The)f(function)g Fu(MPI)p 475 1137 V 15 w(CREATE)p 622 1137 V 14 w(RECV)f Fx(creates)j(a)e(comm)o(unicatio)o(n)d(ob)r(ject)k (for)e(a)h(receiv)o(e)h(op)q(eration.)75 1234 y Fu(MPI)p 144 1234 V 15 w(CREATE)p 291 1234 V 15 w(RECV\(handle,)k(start,)h(count,)h (datatype,)f(source,)g(tag,)h(comm\))117 1284 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)117 1330 y(out)e Fu(start)651 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i (bu\013er)117 1375 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(receiv)o(ed)h(\(in)o(teger\))117 1421 y(in)f Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(eac)o(h)g(elemen)o(t)117 1466 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(source)h(or)f(MPI)p 1321 1466 12 2 v 14 w(ANY)p 1422 1466 V 13 w(SOUR)o(CE)g(\(in)o(teger\))117 1512 y(in)h Fu(tag)719 b Ft(message)13 b(tag)h(or)f(MPI)p 1285 1512 V 14 w(ANY)p 1386 1512 V 13 w(T)m(A)o(G)f(\(in)o(teger\))117 1557 y(in)i Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)i(ob)r(ject)158 1634 y Fx(A)i(comm)o(uni)o(cation)d(\(send)j(or)f(receiv)o(e\))i(that)f(uses) g(a)g(prede\014ned)h(handle)e(is)g(initiated)g(b)o(y)g(the)h(function)75 1683 y Fu(MPI)p 144 1683 14 2 v 15 w(START)p Fx(.)75 1780 y Fu(MPI)p 144 1780 V 15 w(START\(handle\))117 1830 y Ft(inout)e Fu(handle)597 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r(ject)158 1907 y Fx(The)i(comm)o(unication)c(ob)r(ject)19 b(asso)q(ciated)f(with)g Fu(handle)e Fx(should)h(b)q(e)i(an)e(ob)r(ject)i(that)e(w)o(as)h(created)h(b) o(y)75 1956 y(one)d(of)g(the)h(previous)f(four)g(functions,)g(so)g(that)h (all)d(the)j(comm)o(unication)c(parameters)j(are)h(already)e(de\014ned.)75 2006 y(A)g(send)g(can)g(b)q(e)g(started)h(pro)o(vided)f(that)f(the)h (previous)g(send)h(using)e(the)i(same)d(ob)r(ject)j(has)e(completed,)g(or)h (as)75 2056 y(so)q(on)i(as)f(the)h(ob)r(ject)h(is)e(created,)i(if)e(it)g(has) h(not)f(y)o(et)h(b)q(een)g(used)h(in)e(a)g(comm)o(unication.)23 b(In)16 b(addition,)g(if)f(the)75 2106 y(comm)o(unication)e(mo)q(de)i(is)g Fu(ready)g Fx(then)i(a)f(matc)o(hing)e(receiv)o(e)j(should)f(b)q(e)g(p)q (osted.)26 b(The)16 b(send)h(bu\013er)g(should)75 2156 y(not)d(b)q(e)g(up)q (dated)h(after)f(the)h(send)f(is)g(started,)h(un)o(til)e(the)h(op)q(eration)g (completes.)158 2205 y(A)i(receiv)o(e)i(can)e(b)q(e)h(started)h(pro)o(vided)e (that)g(the)h(preceding)g(receiv)o(e)h(using)e(the)h(same)e(ob)r(ject)i(has)f (com-)75 2255 y(pleted,)i(or)f(as)g(so)q(on)h(as)f(the)h(ob)r(ject)g(is)f (created,)i(if)d(it)h(has)g(not)g(y)o(et)g(b)q(een)i(used)f(in)e(a)h(comm)o (unication.)25 b(The)75 2305 y(receiv)o(e)15 b(bu\013er)g(should)f(not)g(b)q (e)g(accessed)i(after)e(the)h(receiv)o(e)g(is)f(started,)g(un)o(til)f(the)i (op)q(eration)f(completes.)158 2355 y(The)i(call)f(is)h(non)o(blo)q(c)o (king,)e(with)i(similar)d(seman)o(tics)j(as)f(the)i(non)o(blo)q(c)o(king)d (comm)o(unication)f(op)q(erations)75 2405 y(describ)q(ed)j(in)d(Section)h (1.8.)158 2455 y(A)d(comm)o(uni)o(cation)d(started)j(with)f(a)h(call)e(to)i Fu(MPI)p 935 2455 V 15 w(START)e Fx(is)i(completed)e(b)o(y)i(a)f(call)g(to)g Fu(MPI)p 1574 2455 V 15 w(WAIT)p Fx(,)f Fu(MPI)p 1764 2455 V 15 w(TEST)p Fx(,)75 2504 y(or)h(one)g(of)f(the)i(deriv)o(ed)f(functions)g (describ)q(ed)i(in)d(Section)h(1.8.4.)15 b(These)c(comm)o(unication)c (completion)h(functions)75 2554 y(do)13 b(not)g(deallo)q(cate)g(the)g(comm)o (unication)d(ob)r(ject,)j(and)g(this)g(can)g(b)q(e)h(reused)g(anew)f(b)o(y)g (a)g Fu(MPI)p 1578 2554 V 15 w(START)f Fx(call.)17 b(The)75 2604 y(ob)r(ject)f(needs)h(to)e(b)q(e)h(explicitly)e(deallo)q(cated)h(b)o(y)g (a)g(call)g(to)g(the)h(function)f Fu(MPI)p 1358 2604 V 15 w(COMM)p 1461 2604 V 15 w(FREE)p Fx(,)f(b)q(elo)o(w.)22 b(W)m(e)15 b(th)o(us)75 2654 y(ha)o(v)o(e)k(t)o(w)o(o)g(t)o(yp)q(es)h(of)e(comm)o(unication)e(ob)r (jects:)29 b Fv(p)q(ersisten)o(t)16 b Fx(ob)r(jects,)21 b(whic)o(h)e(are)g (allo)q(cated)g(b)o(y)g(a)g(call)f(to)75 2704 y Fu(MPI)p 144 2704 V 15 w(CREATE)p 291 2704 V 15 w(xxx)p Fx(,)h(and)g(are)g(explicitly)f (deallo)q(cated)h(b)o(y)g Fu(MPI)p 1100 2704 V 15 w(COMM)p 1203 2704 V 15 w(FREE)p Fx(,)e(and)i Fv(ephemeral)e Fx(ob)r(jects)j(that)p eop %%Page: 22 22 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(22)75 45 y(p)q(ersist)19 b(for)d(one)i(comm)o(unication)c (only:)24 b(they)18 b(are)f(created)i(b)o(y)e(a)g(non)o(blo)q(c)o(king)f (comm)o(unication)e(initiation)75 95 y(function,)f(and)h(are)g(freed)h(b)o(y) f(the)g(comm)o(unication)c(completion)j(call.)75 192 y Fu(MPI)p 144 192 14 2 v 15 w(COMM)p 247 192 V 15 w(FREE\(handle\))117 242 y Ft(inout)i Fu(handle)597 b Ft(handle)14 b(to)f(comm)o(unication)j(ob)r (ject)158 318 y Fx(Marks)g(the)h(comm)o(uni)o(cation)c(ob)r(ject)k(for)e (deallo)q(cation.)23 b(The)16 b(ob)r(ject)h(will)d(b)q(e)i(deallo)q(cated)g (when)h(there)75 368 y(are)d(no)g(p)q(ending)g(comm)o(unicatio)o(ns)e(in)o(v) o(olving)f(this)j(ob)r(ject,)g(at)g(whic)o(h)g(p)q(oin)o(t)f(the)i(handle)f (b)q(ecomes)g(n)o(ull.)158 418 y(The)21 b(call)e(is)h(non)o(blo)q(c)o(king,)g (and)g(the)h(ob)r(ject)g(ma)o(y)d(not)i(b)q(e)h(deallo)q(cated)f(when)h(the)g (call)e(returns.)39 b(It)75 468 y(is)18 b(p)q(ermissible)g(to)g(call)f Fu(MPI)p 546 468 V 16 w(COMM)p 650 468 V 14 w(FREE\(handle\))f Fx(after)j(a)f(comm)o(unicati)o(on)e(that)i(uses)h Fu(handle)e Fx(has)i(b)q(een)75 517 y(initiated,)g(but)h(b)q(efore)g(it)f(has)g (completed.)34 b(The)20 b(ob)r(ject)g(will)e(b)q(e)i(deallo)q(cated)f(after)h (the)f(comm)o(unication)75 567 y(completes.)f(It)c(is)f(erroneous)j(to)d(try) i(to)e(reuse)j(the)e(handle)g(for)g(a)f(subsequen)o(t)j(comm)o(unicatio)o(n.) 158 617 y(A)g(correct)i(in)o(v)o(o)q(cation)c(of)i(the)g(functions)g(describ) q(ed)i(in)e(this)g(section)g(will)f(o)q(ccur)i(in)e(a)h(sequence)i(of)d(the) 75 667 y(form)179 750 y Fv(Create)g Fx(\()p Fv(Start)g(Complete)p Fx(\))691 733 y Fk(\003)724 750 y Fv(F)l(ree)h Fj(;)75 833 y Fx(where)j Fn(\003)f Fx(indicates)g(zero)g(or)g(more)f(rep)q(etitions.)31 b(If)17 b(the)i(same)e(comm)o(unicati)o(on)e(ob)r(ject)k(is)e(used)i(in)e (sev)o(eral)75 883 y(concurren)o(t)h(threads,)g(it)e(is)g(the)h(user)g(resp)q (onsibilit)o(y)f(to)g(co)q(ordinate)h(calls)f(so)g(that)h(the)g(correct)h (sequence)g(is)75 932 y(ob)q(ey)o(ed.)158 982 y(A)g(send)h(op)q(eration)f (initiated)f(with)h Fu(MPI)p 834 982 V 15 w(START)f Fx(can)h(b)q(e)h(matc)o (hed)e(with)h(an)o(y)f(receiv)o(e)j(op)q(eration)e(and,)75 1032 y(lik)o(ewise,)d(a)g(receiv)o(e)i(op)q(eration)e(initiated)g(with)g Fu(MPI)p 935 1032 V 15 w(START)f Fx(can)i(receiv)o(e)h(messages)e(generated)i (b)o(y)e(an)o(y)g(send)75 1082 y(op)q(eration.)75 1219 y Fw(1.11)59 b(Send-receive)75 1389 y Fo(Discussion:)158 1438 y(This)15 b(section)g(has)g(not)g(y)o(et)g(b)q(een)g(appro)o(v)o(ed)f(b)o(y)h(MPIF.)158 1571 y Fx(The)f Fv(send-receiv)o(e)c Fx(op)q(erations)k(com)o(bines)e(in)h (one)h(call)e(the)i(sending)g(of)e(a)h(message)g(to)h(one)f(destination)75 1621 y(and)h(the)g(receiving)h(of)e(another)h(message,)f(from)g(another)h (destination,)f(p)q(ossibly)h(the)g(same.)k(The)c Fv(exc)o(hange)75 1671 y Fx(op)q(erations)20 b(are)g(the)h(same)e(as)h(send-receiv)o(e,)j (except)e(that)f(the)h(send)f(bu\013er)h(and)f(the)g(receiv)o(e)i(bu\013er)f (are)75 1720 y(iden)o(tical.)j(A)17 b(send-receiv)o(e)h(op)q(eration)e(is)g (v)o(ery)h(useful)f(for)g(executing)h(a)f(shift)g(op)q(eration)g(across)h(a)f (c)o(hain)g(of)75 1770 y(pro)q(cesses.)25 b(If)15 b(blo)q(c)o(king)f(sends)j (and)e(receiv)o(es)i(are)e(used)i(for)e(suc)o(h)h(shift,)f(then)g(one)h (needs)h(to)e(order)h(correctly)75 1820 y(the)d(sends)g(and)f(receiv)o(es)i (\(e.g.)j(ev)o(en)c(sends,)g(next)g(receiv)o(es,)h(o)q(dd)e(receiv)o(es)i (\014rst,)e(next)h(sends\))h(so)e(as)g(to)g(prev)o(en)o(t)75 1870 y(cyclic)j(dep)q(endencies)i(that)e(lead)f(to)h(deadlo)q(c)o(k.)20 b(When)15 b(a)f(send-receiv)o(e)j(or)e(exc)o(hange)g(op)q(eration)g(is)f (used,)h(the)75 1920 y(comm)o(unication)g(subsystem)j(tak)o(es)g(care)h(of)f (these)h(issues.)32 b(Also,)18 b(a)g(send-receiv)o(e)i(op)q(eration)e(is)g (useful)g(for)75 1969 y(implemen)o(ting)10 b(remote)k(pro)q(cedure)i(calls.) 158 2019 y(A)h(message)f(sen)o(t)h(b)o(y)g(a)f(send-receiv)o(e)j(or)d(exc)o (hange)i(op)q(eration)e(can)h(b)q(e)g(receiv)o(ed)h(b)o(y)f(a)f(regular)h (receiv)o(e)75 2069 y(op)q(eration,)c(and)h(vice)g(v)o(ersa.)75 2166 y Fu(MPI)p 144 2166 V 15 w(SENDRECV\(send)p 445 2166 V 13 w(start,)21 b(send)p 699 2166 V 14 w(count,)g(send)p 954 2166 V 15 w(type,)g(dest,)f(recv)p 1318 2166 V 15 w(start,)h(recv)p 1574 2166 V 15 w(count,)f(recv)p 1829 2166 V 15 w(type,)75 2216 y(source,)g(tag,)h(comm,)g(status\))117 2266 y Ft(in)14 b Fu(send)p 253 2266 V 15 w(start)572 b Ft(initial)16 b(address)e(of)e(send)i (bu\013er)g(\(c)o(hoice\))117 2311 y(in)g Fu(send)p 253 2311 V 15 w(count)572 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g (bu\013er)g(\(in)o(teger\))117 2357 y(in)g Fu(send)p 253 2357 V 15 w(type)594 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(send)f(bu\013er) 117 2403 y(in)h Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\)) 117 2448 y(out)e Fu(recv)p 277 2448 V 15 w(start)548 b Ft(initial)16 b(address)e(of)e(receiv)o(e)i(bu\013er)g(\(c)o(hoice\))117 2494 y(in)g Fu(recv)p 253 2494 V 15 w(count)572 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(receiv)o(e)g(bu\013er)g(\(in)o(teger\))117 2540 y(in)g Fu(recv)p 253 2540 V 15 w(type)594 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(receiv)o(e)g(bu\013er)117 2585 y(in)g Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 2631 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 2677 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)p eop %%Page: 23 23 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(23)117 45 y Ft(out)14 b Fu(status)629 b Ft(status)13 b(ob)r(ject)158 121 y Fx(Execute)k(a)d(blo)q(c)o(king)g(send)i(and)f(receiv)o (e)h(op)q(eration.)21 b(Both)16 b(send)g(and)e(receiv)o(e)j(use)e(the)h(same) e(tag)h(v)n(alue)75 171 y(and)d(the)h(same)e(comm)o(unicator.)k(Ho)o(w)o(ev)o (er,)d(the)h(send)g(bu\013er)g(and)f(receiv)o(e)h(bu\013er)g(are)g(disjoin)o (t,)e(and)h(ma)o(y)e(ha)o(v)o(e)75 221 y(di\013eren)o(t)15 b(length)f(and)f(di\013eren)o(t)i(datat)o(yp)q(es.)158 349 y Fo(Discussion:)158 395 y Ft(F)m(or)d(a)f(shift)i(it's)e(more)h(natural)h (to)f(ha)o(v)o(e)g(same)g(t)o(yp)q(e)g(and)g(same)g(length)h(b)q(oth)g(for)e (send)h(and)h(receiv)o(e;)g(for)e(a)h(remote)75 441 y(pro)q(cedure)i(call,)g (the)g(additional)i(freedom)d(mak)o(es)h(sense.)75 620 y Fu(MPI)p 144 620 14 2 v 15 w(EXCHANGE\(start,)19 b(count,)h(datatype,)g(dest,)h (source,)f(tag,)h(comm,)g(status\))117 670 y Ft(in)14 b Fu(start)675 b Ft(initial)16 b(address)e(of)e(send)i(and)g(receiv)o(e)g(bu\013er)g(\(c)o (hoice\))117 713 y(in)g Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(send)g(and)f(receiv)o(e)h(bu\013er)g(\(in)o (teger\))117 755 y(in)g Fu(datatype)609 b Ft(t)o(yp)q(e)13 b(of)g(elemen)o(ts)h(in)g(send)f(and)h(receiv)o(e)g(bu\013er)117 798 y(in)g Fu(dest)697 b Ft(rank)13 b(of)g(destination)j(\(in)o(teger\))117 841 y(in)e Fu(source)653 b Ft(rank)13 b(of)g(source)h(\(in)o(teger\))117 883 y(in)g Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\))117 926 y(in)g Fu(comm)697 b Ft(handle)14 b(to)f(comm)o(unicator)117 968 y(out)h Fu(status)629 b Ft(status)13 b(ob)r(ject)158 1045 y Fx(Execute)20 b(a)e(blo)q(c)o(king)f(send)i(and)f(receiv)o(e;)j(the)e(same) e(bu\013er)i(is)f(used)i(b)q(oth)e(for)g(the)g(send)i(and)e(for)f(the)75 1094 y(receiv)o(e.)158 1223 y Fo(Discussion:)158 1268 y Ft(I)10 b(omitted)h(the)g(ready)g(and)f(sync)o(hronous)j(mo)q(des)e(for)f (send/receiv)o(e)i(and)f(exc)o(hange.)17 b(Also,)11 b(I)f(omitted)h(non)o (blo)q(c)o(kin)q(g)75 1314 y(send/receiv)o(e)k(and)f(exc)o(hange.)k(Do)13 b(w)o(e)g(w)o(an)o(t)g(an)o(y)g(of)g(those?)158 1447 y Fx(The)d(seman)o(tics) f(of)f(a)h(send-receiv)o(e)j(op)q(eration)d(is)g(what)g(w)o(ould)g(obtain)f (if)h(the)h(caller)f(fork)o(ed)g(t)o(w)o(o)g(concurren)o(t)75 1496 y(threads,)18 b(one)g(to)f(execute)h(the)g(send,)g(and)f(one)g(to)g (execute)i(the)f(receiv)o(e,)h(follo)o(w)o(ed)c(b)o(y)i(a)g(join)f(of)g (these)i(t)o(w)o(o)75 1546 y(threads.)25 b(A)16 b(send-receiv)o(e)h(cannot)f (b)q(e)h(implemen)o(ted)c(b)o(y)j(a)f(blo)q(c)o(king)g(send)i(follo)o(w)o(ed) d(b)o(y)h(a)h(blo)q(c)o(king)f(receiv)o(e)75 1596 y(or)e(a)f(blo)q(c)o(king)f (receiv)o(e,)j(follo)o(w)o(ed)d(b)o(y)i(a)f(blo)q(c)o(king)f(send.)19 b(Consider)13 b(the)g(follo)o(wing)d(co)q(de,)j(where)h(t)o(w)o(o)e(pro)q (cesses)75 1646 y(exc)o(hange)j(messages:)75 1722 y Fu(CALL)21 b(MPI_COMM_RANK\(com)o(m,)e(rank,)i(ierr\))75 1772 y(IF)g(\(rank.EQ.0\))119 1821 y(THEN)162 1871 y(CALL)g(MPI_SENDRECV\(send_)o(buff,)d(count,)j (MPI_REAL,)f(0,)h(recv_buff,)f(count,)g(MPI_REAL,)271 1921 y(1,)i(tag,)e(comm,)h(status,)g(ierr\))119 1971 y(ELSE)162 2021 y(CALL)g(MPI_SENDRECV\(send_)o(buff,)d(count,)j(MPI_REAL,)f(1,)h (recv_buff,)f(count,)g(MPI_REAL,)271 2070 y(0,)i(tag,)e(comm,)h(status,)g (ierr\))75 2120 y(END)g(IF)158 2196 y Fx(If)10 b(the)g(send)h(receiv)o(es)h (are)f(replaced)g(either)g(b)o(y)e(blo)q(c)o(king)h(send,)h(follo)o(w)o(ed)d (b)o(y)i(blo)q(c)o(king)f(receiv)o(e,)j(or)e(blo)q(c)o(king)75 2246 y(receiv)o(e,)j(follo)o(w)o(ed)d(b)o(y)h(blo)q(c)o(king)f(send,)i(then)g (the)g(co)q(de)h(ma)o(y)c(deadlo)q(c)o(k.)17 b(On)12 b(the)g(other)g(hand,)f (send-receiv)o(e)j(can)75 2296 y(b)q(e)h(implem)o(en)o(ted)d(using)i(non)o (blo)q(c)o(king)f(sends)i(and)f(receiv)o(es.)20 b(Note)14 b(that)g(some)f (system)h(bu\013ering)g(is)g(required)75 2346 y(for)9 b(a)h(correct)h (implemen)o(tatio)o(n)c(of)i Fu(MPI)p 703 2346 V 15 w(EXCHANGE)p Fx(.)f(\(Consider)i(the)g(last)f(example,)g(with)g(send-receiv)o(e)j (replaced)75 2395 y(b)o(y)i(exc)o(hange.\))75 2530 y Fw(1.12)59 b(Null)21 b(p)n(ro)r(cesses)75 2704 y Fo(Discussion:)35 b Ft(This)14 b(section)g(has)f(not)h(b)q(een)f(review)o(ed)h(b)o(y)g(MPIF.)p eop %%Page: 24 24 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(24)158 45 y(In)16 b(man)o(y)d(instances,)k(it)e(is)h(con)o(v)o (enien)o(t)g(to)f(sp)q(ecify)i(a)e(\\dumm)o(y")d(source)17 b(or)f(destination)f(for)g(comm)o(uni-)75 95 y(cation.)27 b(This)17 b(simpli\014es)e(the)j(co)q(de)f(that)g(is)g(needed)i(for)d(dealing)g(with)h (b)q(oundaries,)h(e.g.,)e(in)g(the)i(case)g(of)e(a)75 145 y(non-circular)e (shift)f(done)i(with)e(calls)h(to)f(send-receiv)o(e.)158 195 y(The)e(sp)q(ecial)g(v)n(alue)f Fu(MPI)p 545 195 14 2 v 15 w(PROCNULL)f Fx(can)i(b)q(e)g(used)h(instead)f(of)f(a)g(rank)h(wherev)o(er)h (a)e(source)i(or)f(a)f(destination)75 244 y(parameter)h(is)f(required)i(in)e (a)h(call.)16 b(A)11 b(comm)o(unication)d(with)i(pro)q(cess)j Fu(MPI)p 1268 244 V 15 w(PROCNULL)c Fx(has)i(no)g(e\013ect:)18 b(a)10 b(send)i(to)75 294 y Fu(MPI)p 144 294 V 15 w(PROCNULL)f Fx(succeeds)16 b(and)d(returns)h(as)f(so)q(on)g(as)g(p)q(ossible.)18 b(A)13 b(receiv)o(e)h(from)d Fu(MPI)p 1443 294 V 15 w(PROCNULL)h Fx(succeeds)j(and)75 344 y(returns)d(as)e(so)q(on)g(as)g(p)q(ossible)g(with)g (no)g(mo)q(di\014cations)e(to)i(the)h(receiv)o(e)g(bu\013er.)18 b(When)10 b(a)g(receiv)o(e)h(with)f Fu(source)20 b(=)75 394 y(MPI)p 144 394 V 15 w(PROCNULL)9 b Fx(is)h(executed)i(then)f(the)g(status)g (ob)r(ject)g(returns)h Fu(source)21 b(=)g(MPI)p 1352 394 V 15 w(PROCNULL)p Fx(,)8 b Fu(tag=MPI)p 1717 394 V 15 w(ANY)p 1798 394 V 15 w(TAG)75 444 y Fx(and)14 b Fu(count=0)p Fx(.)158 572 y Fo(Discussion:)158 618 y Ft(It)f(w)o(ould)h(b)q(e)f(nice)h(to)f(ha)o(v) o(e)g Fm(MPI)p 640 618 12 2 v 13 w(PROCNULL)d Ft(\\equiv)n(alenced")16 b(to)d(-1)g(and)g(to)g(group)p 1393 618 V 15 w(size,)h(so)f(that,)f(when)i(a) f(shift)g(is)75 663 y(executed)h(in)g(a)f(group)h(the)f(\014rst)g(and)h(last) f(pro)q(cessors)i(automatically)h(comm)o(unicate)e(with)g(the)f(n)o(ull)i (pro)q(cess.)158 709 y(A)e(c)o(hoice)i(is)g(to)e(sa)o(y)i(that)e(a)h (source/destinati)q(on)j(with)d(rank)g(-1)g(or)g(group)p 1278 709 V 15 w(size)g(is)g(alw)o(a)o(ys)h(the)f(n)o(ull)i(pro)q(cess.)k(W)m(e)75 755 y(then)13 b(lose)h(some)g(error)f(catc)o(hing)h(capabilit)o(y)m(.)158 800 y(Another)g(c)o(hoice)g(is)f(to)g(sa)o(y)h(that)f(only)h(for)f (send-receiv)o(e)h(and)g(exc)o(hange.)158 846 y(A)d(third)h(c)o(hoice)g(is)g (to)f(liv)o(e)i(without)f(this)g(nicet)o(y)m(.)17 b(It)11 b(means)g(that)h (one)f(still)i(ha)o(v)o(e)f(to)f(execute)h(conditional)i(co)q(de:)j(\(if)75 892 y(m)o(yrank=0)c(then)g(m)o(yneigh)o(b)q(or=mpi)p 640 892 V 17 w(pro)q(cn)o(ull\).)19 b(But)13 b(this)h(co)q(de)f(need)g(b)q(e)f (executed)i(only)g(when)e(the)h(comm)o(unication)75 937 y(pattern)h(is)f (set,)g(and)h(the)f(conditional)j(b)q(oundary)f(co)q(de)e(is)h(not)f(needed)h (in)g(the)f(inner)i(lo)q(op.)75 1157 y Fw(1.13)59 b(Derived)19 b(datat)n(yp)r(es)75 1327 y Fo(Discussion:)158 1372 y(This)c(is)g(a)g(new)g (section,)h(and)e(the)h(material)g(has)h(not)e(y)o(et)i(b)q(een)e(discussed)h (b)o(y)g(MPIF.)158 1505 y Fx(Up)k(to)g(no)o(w,)g(all)f(p)q(oin)o(t)h(to)f(p)q (oin)o(t)h(comm)o(unicatio)o(n)d(in)o(v)o(olv)o(ed)i(only)g(con)o(tiguous)h (bu\013ers)h(con)o(taining)e(a)75 1555 y(sequence)f(of)d(elemen)o(ts)g(of)h (the)g(same)f(t)o(yp)q(e.)21 b(This)14 b(is)h(to)q(o)f(constraining)h(on)f(t) o(w)o(o)g(accoun)o(ts:)21 b(One)15 b(often)g(w)o(an)o(ts)75 1604 y(to)i(pass)g(messages)h(that)f(con)o(tain)f(v)n(alues)h(with)g (di\013eren)o(t)h(datat)o(yp)q(es)f(\(e.g.,)g(an)g(in)o(teger)g(coun)o(t,)h (follo)o(w)o(ed)d(b)o(y)75 1654 y(coun)o(t)d(real)g(n)o(um)o(b)q(ers\);)g (and)f(one)h(often)g(w)o(an)o(ts)g(to)f(send)i(noncon)o(tiguous)f(data)f (\(e.g.)17 b(a)12 b(sub-blo)q(c)o(k)f(of)h(a)f(matrix\).)75 1704 y(One)17 b(solution)e(is)g(to)h(pro)o(vide)f(functions)h(that)g(pac)o(k) g(noncon)o(tiguous)f(data)h(in)o(to)f(a)g(con)o(tiguous)h(bu\013er)h(at)e (the)75 1754 y(sender)f(site)e(and)g(unpac)o(k)g(it)g(bac)o(k)g(at)g(the)g (receiv)o(er)i(site.)k(This)12 b(has)g(the)g(disadv)n(an)o(tage)g(of)f (requiring)h(additional)75 1804 y(memory)f(to)h(memory)f(cop)o(y)i(at)g(b)q (oth)g(sites,)g(ev)o(en)h(when)g(the)f(comm)o(unication)d(subsystem)j(has)g (scatter-gather)75 1853 y(capabilities.)19 b(Instead,)c(MPI)g(pro)o(vides)g (mec)o(hanisms)d(to)j(sp)q(ecify)g(more)e(general,)i(mixed)e(and)h(noncon)o (tiguous)75 1903 y(comm)o(unication)8 b(bu\013ers.)19 b(It)11 b(is)g(up)h(to)f(the)h(implemen)o(tation)c(to)k(decide)g(whether)h(data)e (should)g(b)q(e)h(\014rst)h(pac)o(k)o(ed)75 1953 y(in)i(a)g(con)o(tiguous)g (bu\013er)h(b)q(efore)g(b)q(een)g(transmitted,)f(or)g(whether)i(it)e(can)g(b) q(e)h(collected)g(directly)f(from)f(where)75 2003 y(it)g(resides.)158 2053 y(The)f(general)g(mec)o(hanisms)e(pro)o(vided)i(here)h(allo)o(ws)e(one)h (to)f(transfer)i(directly)m(,)e(without)h(cop)o(ying,)f(ob)r(jects)75 2103 y(of)20 b(v)n(arious)g(shap)q(e)h(and)f(size.)39 b(It)20 b(is)h(not)f(assumed)g(that)h(the)g(MPI)g(library)e(is)i(cognizan)o(t)f(of)g (the)h(ob)r(jects)75 2152 y(declared)15 b(in)f(the)g(host)h(language;)e(th)o (us,)h(if)f(one)i(w)o(an)o(ts)f(to)g(transfer)h(a)f(structure,)h(or)f(an)g (arra)o(y)g(section,)h(it)e(will)75 2202 y(b)q(e)k(necessary)h(to)e(pro)o (vide)g(in)g(MPI)h(a)e(de\014nition)h(of)g(a)g(comm)o(unicatio)o(n)e (bu\013er)j(that)f(mimi)o(cs)e(the)j(de\014nition)75 2252 y(of)f(the)i (structure)h(or)e(arra)o(y)g(section)h(in)f(question.)27 b(These)19 b(facilities)d(can)h(b)q(e)h(used)g(b)o(y)e(library)h(designers)h(to)75 2302 y(de\014ne)13 b(comm)o(unicatio)o(n)c(functions)j(that)g(can)g(transfer) h(ob)r(jects)g(de\014ned)g(in)e(the)i(host)f(language)f({)g(b)o(y)h(deco)q (ding)75 2352 y(their)k(de\014nitions)f(as)h(a)o(v)n(ailable)d(in)i(a)g(sym)o (b)q(ol)f(table)h(or)h(a)f(dop)q(e)h(v)o(ector.)23 b(Suc)o(h)16 b(higher-lev)o(el)f(comm)o(unication)75 2401 y(functions)f(are)g(not)g(part)g (of)f(MPI.)158 2451 y(More)k(general)g(comm)o(unicatio)o(n)d(bu\013ers)k(are) f(sp)q(eci\014ed)h(b)o(y)e(replacing)h(the)g(basic)f(datat)o(yp)q(es)i(that)e (ha)o(v)o(e)75 2501 y(used)e(so)e(far)g(with)h(deriv)o(ed)g(datat)o(yp)q(es)g (that)g(are)g(constructed)i(from)10 b(basic)j(datat)o(yp)q(es)g(using)g(the)g (constructors)75 2551 y(describ)q(ed)e(in)e(this)h(section.)17 b(These)10 b(metho)q(ds)f(of)g(constructing)i(deriv)o(ed)e(datat)o(yp)q(es)h (can)g(b)q(e)g(applied)f(recursiv)o(ely)m(.)158 2601 y(A)14 b Fv(general)g(datat)o(yp)q(e)f Fx(is)g(an)h(opaque)g(ob)r(ject)h(that)f(sp)q (eci\014es)h(t)o(w)o(o)f(things:)137 2684 y Fn(\017)21 b Fx(A)14 b(sequence)i(of)d(basic)h(t)o(yp)q(es)p eop %%Page: 25 25 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(25)137 45 y Fn(\017)21 b Fx(A)14 b(sequence)i(of)d(in)o(teger) h(\(b)o(yte\))h(displacemen)o(ts)158 128 y(The)e(displacemen)o(ts)f(are)h (not)f(required)i(to)e(b)q(e)h(p)q(ositiv)o(e,)f(distinct,)h(or)f(in)g (increasing)h(order;)g(therefore)h(the)75 178 y(order)g(of)e(items)g(need)j (not)e(coincide)g(with)g(their)g(order)h(in)f(store,)g(and)g(an)g(item)f(ma)o (y)f(app)q(ear)j(more)e(than)h(once.)158 228 y(Let)179 311 y Fj(D)q(ataty)q(pe)g Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 317 y Fi(0)538 311 y Fj(;)7 b(disp)633 317 y Fi(0)651 311 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 317 y Fh(n)p Fk(\000)p Fi(1)897 311 y Fj(;)g(disp)992 317 y Fh(n)p Fk(\000)p Fi(1)1057 311 y Fx(\))p Fn(g)p Fj(;)75 394 y Fx(b)q(e)15 b(suc)o(h)g(a)f(general)h (datat)o(yp)q(e,)f(where)i Fj(ty)q(pe)788 400 y Fh(i)817 394 y Fx(are)f(basic)f(t)o(yp)q(es,)h(and)f Fj(disp)1270 400 y Fh(i)1299 394 y Fx(are)g(displacemen)o(ts.)19 b(This)c(general)75 444 y(datat)o(yp)q(e,)d(together)h(with)f(a)g(base)g(address)i Fu(start)p Fx(,)c(sp)q(eci\014es)k(a)e(comm)o(unication)d(bu\013er:)18 b(the)12 b(comm)o(unication)75 493 y(bu\013er)i(that)e(consists)i(of)e Fj(n)g Fx(en)o(tries,)h(where)h(the)f Fj(i)p Fx(-th)g(en)o(try)g(is)g(at)f (address)i Fu(start)5 b Fx(+)i Fj(disp)1502 499 y Fh(i)1529 493 y Fx(and)12 b(has)h(t)o(yp)q(e)g Fj(ty)q(pe)1849 499 y Fh(i)1863 493 y Fx(.)75 543 y(A)18 b(message)f(assem)o(bled)g(from)f(this)i (comm)o(unicatio)o(n)d(bu\013er)k(will)d(con)o(tain)h(a)h(sequence)h(of)e Fj(n)h Fx(v)n(alues,)g(where)75 593 y(v)n(alue)d Fj(i)h Fx(has)g(t)o(yp)q(e)g Fj(ty)q(pe)461 599 y Fh(i)476 593 y Fx(.)23 b(The)16 b(general)g(datat)o(yp)q (e)g(determines)g(the)g Fv(t)o(yp)q(e)h(signature)c Fx(of)i(the)h(message)g (\(i.e.)75 643 y(the)e(n)o(um)o(b)q(er)g(of)f(v)n(alues)h(carried)g(b)o(y)g (the)g(message)g(and)g(the)g(basic)g(t)o(yp)q(e)g(of)g(eac)o(h)g(v)n(alue\).) 158 693 y(W)m(e)h(can)i(use)f(a)g(handle)f(to)h(a)g(general)g(datat)o(yp)q(e) g(as)g(an)f(argumen)o(t)g(in)g(a)h(send)h(or)f(receiv)o(e)h(op)q(eration,)e (in)75 742 y(replacemen)o(t)j(of)g(a)g(basic)g(datat)o(yp)q(e)h(argumen)o(t.) 30 b(The)19 b(op)q(eration)f Fu(MPI)p 1253 742 14 2 v 15 w(SEND\(start,)h(1,) j(datatype,...\))75 792 y Fx(will)11 b(use)j(the)f(send)h(bu\013er)f (de\014ned)h(b)o(y)f(the)g(base)g(address)h Fu(start)e Fx(and)h(the)g (general)g(datat)o(yp)q(e)g(asso)q(ciated)g(with)75 842 y Fu(datatype)p Fx(;)d(it)h(will)e(generate)k(a)e(message)f(with)h(the)h(t)o(yp)q(e)g (signature)f(determined)g(b)o(y)g(the)h Fu(datatype)e Fx(argumen)o(t.)75 892 y Fu(MPI)p 144 892 V 15 w(RECV\(start,)20 b(1,)h(datatype,...\))15 b Fx(will)d(use)i(the)g(receiv)o(e)g(bu\013er)g(de\014ned)g(b)o(y)f(the)h (base)g(address)g Fu(start)75 942 y Fx(and)g(the)g(general)g(datat)o(yp)q(e)g (asso)q(ciated)h(with)f Fu(datatype)p Fx(.)158 992 y(General)k(datat)o(yp)q (es)h(can)f(b)q(e)g(used)h(in)f(all)e(send)j(and)f(receiv)o(e)h(op)q (erations.)31 b(W)m(e)17 b(address)j(later)e(in)f(Sec-)75 1041 y(tion)c(1.13.3)f(the)j(case)g(where)g(the)f(second)h(argumen)o(t)e Fu(count)g Fx(has)h(v)n(alue)f Fj(>)f Fx(1.)158 1091 y(The)21 b(prede\014ned)h(basic)e(datat)o(yp)q(es)h(presen)o(ted)h(in)e(Section)g (1.2.1)e(are)j(particular)f(cases)h(of)f(a)g(general)75 1141 y(datat)o(yp)q(e.)h(Th)o(us,)15 b Fu(MPI)p 455 1141 V 15 w(INT)f Fx(is)h(a)f(prede\014ned)j(handle)d(to)h(the)g(datat)o(yp)q(e)h Fn(f)p Fx(\()p Fu(int)o Fj(;)7 b Fx(0\))p Fn(g)p Fx(,)13 b(with)i(one)g(en)o (try)g(of)f(t)o(yp)q(e)75 1191 y Fu(int)f Fx(and)h(displacemen)o(t)f(zero.)19 b(And)14 b(similarly)d(for)i(all)g(other)i(basic)f(datat)o(yp)q(es.)158 1241 y(The)f Fv(exten)o(t)d Fx(of)i(a)g(datat)o(yp)q(e)h(is)f(de\014ned)i(to) e(b)q(e)h(the)g(span)f(from)f(the)i(\014rst)g(b)o(yte)g(to)f(the)h(last)f(b)o (yte)h(o)q(ccupied)75 1290 y(b)o(y)h(en)o(tries)h(in)e(this)h(datat)o(yp)q (e,)g(rounded)g(up)g(to)g(satisfy)f(alignmen)o(t)f(requiremen)o(ts.)18 b(I.e.,)13 b(if)179 1373 y Fj(D)q(ataty)q(pe)g Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 1379 y Fi(0)538 1373 y Fj(;)7 b(disp)633 1379 y Fi(0)651 1373 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 1379 y Fh(n)p Fk(\000)p Fi(1)897 1373 y Fj(;)g(disp)992 1379 y Fh(n)p Fk(\000)p Fi(1)1057 1373 y Fx(\))p Fn(g)p Fj(;)75 1456 y(disp)151 1462 y Fh(r)181 1456 y Fx(=)12 b(min)294 1462 y Fh(j)319 1456 y Fj(disp)395 1462 y Fh(j)426 1456 y Fx(and)i Fj(disp)583 1462 y Fh(s)612 1456 y Fx(=)e(max)733 1462 y Fh(j)758 1456 y Fj(disp)834 1462 y Fh(j)865 1456 y Fx(then)179 1548 y Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))h(=)f Fj(disp)631 1554 y Fh(s)658 1548 y Fx(+)e Fj(siz)r(eof)t Fx(\()p Fj(ty)q(pe)909 1554 y Fh(s)930 1548 y Fx(\))f Fn(\000)h Fj(disp)1073 1554 y Fh(r)1091 1548 y Fx(;)-1028 b(\(1.1\))75 1639 y(if)8 b(furthermore,)i Fj(ty)q(pe)422 1645 y Fh(i)446 1639 y Fx(requires)h(alignmen)o(t)c(to)i(a)g (b)o(yte)g(address)i(that)e(is)g(is)h(a)f(m)o(ultiple)d(of)j Fj(k)1505 1645 y Fh(i)1519 1639 y Fx(,)g(then)h Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))75 1689 y(is)k(rounded)g(up)g(to)g(the)g (next)h(m)o(ultiple)c(of)j(max)841 1695 y Fh(i)862 1689 y Fj(k)884 1695 y Fh(i)897 1689 y Fx(.)158 1739 y(Example:)j(Assume)c(that)179 1822 y Fj(T)6 b(y)q(pe)p Fx(1)12 b(=)g Fn(f)p Fx(\()p Fu(double)o Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(8\))p Fn(g)75 1905 y Fx(\(a)15 b Fu(double)g Fx(at)g(displacemen)o(t)g(zero,)h (follo)o(w)o(ed)e(b)o(y)i(a)f Fu(char)g Fx(at)g(displacemen)o(t)g(eigh)o (t\).)23 b(Assume,)15 b(furthermore,)75 1955 y(that)f(doubles)g(ha)o(v)o(e)g (to)f(b)q(e)i(strictly)f(aligned)f(at)g(addresses)j(that)e(are)g(m)o(ultiple) e(of)h(eigh)o(t.)18 b(Then,)13 b(the)i(exten)o(t)g(of)75 2004 y(this)h(datat)o(yp)q(e)f(is)h(16)f(\(9)g(rounded)h(to)f(the)h(next)g(m)o (ultiple)e(of)h(8\).)22 b(A)16 b(datat)o(yp)q(e)f(that)h(consists)h(of)d(a)i (c)o(haracter)75 2054 y(immediately)10 b(follo)o(w)o(ed)j(b)o(y)g(a)h(double) g(will)e(also)h(ha)o(v)o(e)h(an)g(exten)o(t)h(of)e(16.)75 2170 y Fq(1.13.1)49 b(Datat)o(yp)q(e)17 b(constructo)o(rs)75 2247 y Fx(The)e(simplest)f(datat)o(yp)q(e)i(constructor)g(is)f Fu(MPI)p 832 2247 V 15 w(TYPE)p 935 2247 V 15 w(CONTIGUOUS)e Fx(whic)o(h)i(allo)o(ws)e (replication)i(of)f(a)h(datat)o(yp)q(e)75 2297 y(in)o(to)e(con)o(tiguous)h (lo)q(cations.)75 2394 y Fu(MPI)p 144 2394 V 15 w(TYPE)p 247 2394 V 15 w(CONTIGUOUS\(count,)k(oldtype,)i(newtype\))117 2444 y Ft(in)14 b Fu(count)675 b Ft(replication)16 b(coun)o(t)d(\(in)o(teger\))117 2489 y(in)h Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob) r(ject)117 2535 y(out)g Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o (yp)q(e)g(ob)r(ject)p eop %%Page: 26 26 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(26)158 45 y Fu(newtype)12 b Fx(is)i(the)g(datat)o(yp)q(e)g (obtained)g(b)o(y)g(concatenating)g Fu(count)e Fx(copies)j(of)e Fu(oldtype)p Fx(.)j(P)o(adding)d(ma)o(y)f(b)q(e)75 95 y(added)i(to)g(satisfy) g(alignmen)o(t)d(requiremen)o(ts)j(of)g(the)g(underlying)g(arc)o(hitecture.) 158 145 y(Example:)j(let)d Fu(oldtype)e Fx(b)q(e)i(a)g(handle)g(to)f(the)i (datat)o(yp)q(e)179 228 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 311 y Fx(with)14 b(exten)o(t)g(16,)f(and)h(let)g Fu(count)c Fx(=)i(3.)18 b(The)c(resulting)g(datat)o(yp)q(e)g(returned)i(b)o(y)e Fu(newtype)e Fx(is)179 394 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(32\))p Fj(;)g Fx(\()p Fu(char)m Fj(;)g Fx(40\))p Fn(g)p Fx(;)75 477 y(i.e.,)12 b(alternating)i Fu(double)e Fx(and)i Fu(char)f Fx(elemen)o(ts,)g (with)h(displacemen)o(ts)f(0)p Fj(;)7 b Fx(8)p Fj(;)g Fx(16)p Fj(;)g Fx(24)p Fj(;)g Fx(3)o(2)p Fj(;)g Fx(4)o(0.)158 527 y(In)14 b(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle)f(to)h(the)h (datat)o(yp)q(e)179 610 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 616 y Fi(0)311 610 y Fj(;)7 b(disp)406 616 y Fi(0)425 610 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 616 y Fh(n)p Fk(\000)p Fi(1)671 610 y Fj(;)g(disp)766 616 y Fh(n)p Fk(\000)p Fi(1)830 610 y Fx(\))p Fn(g)p Fj(;)75 693 y Fx(with)14 b(exten)o(t)g Fj(extent)p Fx(.)19 b(Then)14 b Fu(newtype)e Fx(is)i(a)g(handle)g(to)f(the)i (datat)o(yp)q(e)f(with)f Fu(count)c Fn(\001)f Fu(n)14 b Fx(en)o(tries)h (de\014ned)g(b)o(y:)179 784 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 790 y Fi(0)311 784 y Fj(;)7 b(disp)406 790 y Fi(0)425 784 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 790 y Fh(n)p Fk(\000)p Fi(1)671 784 y Fj(;)g(disp)766 790 y Fh(n)p Fk(\000)p Fi(1)830 784 y Fx(\))p Fj(;)g Fx(\()p Fj(ty)q(pe)957 790 y Fi(0)977 784 y Fj(;)g(disp)1072 790 y Fi(0)1099 784 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)1440 790 y Fh(n)p Fk(\000)p Fi(1)1504 784 y Fj(;)g(disp)1599 790 y Fh(n)p Fk(\000)p Fi(1)1673 784 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)179 875 y Fx(\()p Fj(ty)q(pe)271 881 y Fi(0)291 875 y Fj(;)g(disp)386 881 y Fi(0)413 875 y Fx(+)j Fj(extent)f Fn(\001)g Fx(\()p Fu(count)f Fn(\000)i Fx(1\)\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)997 881 y Fh(n)p Fk(\000)p Fi(1)1061 875 y Fj(;)g(disp)1156 881 y Fh(n)p Fk(\000)p Fi(1)1230 875 y Fx(+)i Fj(extent)h Fn(\001)e Fx(\()p Fu(count)h Fn(\000)g Fx(1\)\))p Fn(g)p Fj(:)158 967 y Fu(MPI)p 227 967 14 2 v 15 w(TYPE)p 330 967 V 15 w(VECTOR)k Fx(is)g(a)h(more)f (general)h(constructor)h(that)f(allo)o(ws)f(replication)g(of)h(a)f(datat)o (yp)q(e)h(in)o(to)g(lo)q(ca-)75 1016 y(tions)f(that)h(consist)g(of)f(equally) f(spaced)j(con)o(tiguous)e(blo)q(c)o(ks)h(of)e(equal)h(size)i({)e(blo)q(c)o (k)g(sizes)i(and)e(blo)q(c)o(k)g(displace-)75 1066 y(men)o(ts)g(are)i(all)d (m)o(ultiples)g(of)h(the)i(old)e(t)o(yp)q(e)i(exten)o(t.)75 1163 y Fu(MPI)p 144 1163 V 15 w(TYPE)p 247 1163 V 15 w(VECTOR\()20 b(count,)h(oldtype,)f(stride,)g(blocklen,)g(newtype\))117 1213 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j(\(in) o(teger\))117 1259 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat) o(yp)q(e)f(ob)r(ject)117 1304 y(in)g Fu(stride)653 b Ft(n)o(um)o(b)q(er)10 b(of)f(elemen)o(ts)i(b)q(et)o(w)o(een)f(start)f(of)g(eac)o(h)h(blo)q(c)o(k)h (\(in)o(teger\))117 1350 y(in)j Fu(blocklen)609 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g(\(p)q(ositiv)o(e)h(in)o (teger\))117 1396 y(out)f Fu(newtype)607 b Ft(handle)14 b(to)f(output)h (datat)o(yp)q(e)g(ob)r(ject)158 1472 y Fx(Example:)j(Assume,)c(again,)f(that) i Fu(oldtype)e Fx(p)q(oin)o(ts)i(to)g(the)g(t)o(yp)q(e)179 1555 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 1638 y Fx(with)j(exten)o(t)h (16.)16 b(A)10 b(call)f(to)h Fu(MPI)p 588 1638 V 15 w(TYPE)p 691 1638 V 15 w(VECTOR\()20 b(2,)i(oldtype,)e(4,)h(3,)h(newtype\))8 b Fx(will)h(create)i(the)g(datat)o(yp)q(e)179 1721 y Fn(f)p Fx(\()p Fu(double)n Fj(;)c Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(32\))p Fj(;)g Fx(\()p Fu(char)m Fj(;)g Fx(40\))p Fj(;)179 1812 y Fx(\()p Fu(double)o Fj(;)g Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(72\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(80\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(88\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(96\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(104\))p Fn(g)j Fx(:)75 1904 y(t)o(w)o(o)j(blo)q(c)o(ks)h (with)g(three)h(copies)g(eac)o(h)f(of)f(the)i(old)e(t)o(yp)q(e,)h(starting)g (4*16)f(apart.)158 1954 y(A)h(call)f(to)h Fu(MPI)p 399 1954 V 15 w(TYPE)p 502 1954 V 15 w(VECTOR\(oldtype,)k(3,)k(-2,)f(1,)g(newtype\))13 b Fx(will)f(create)j(the)g(datat)o(yp)q(e)179 2037 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fn(\000)p Fx(32\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fn(\000)p Fx(24\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fn(\000)p Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fn(\000)p Fx(56\))p Fn(g)p Fj(:)158 2120 y Fx(In)14 b(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle) f(to)h(the)h(datat)o(yp)q(e)179 2203 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 2209 y Fi(0)311 2203 y Fj(;)7 b(disp)406 2209 y Fi(0)425 2203 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 2209 y Fh(n)p Fk(\000)p Fi(1)671 2203 y Fj(;)g(disp)766 2209 y Fh(n)p Fk(\000)p Fi(1)830 2203 y Fx(\))p Fn(g)p Fj(;)75 2286 y Fx(with)15 b(exten)o(t)h Fj(extent)p Fx(.)22 b(The)16 b(newly)f(created)h(datat)o(yp)q(e)g(is)f(a)g (sequence)i(of)d(length)i Fu(count)9 b Fn(\001)g Fu(blocklen)g Fn(\001)g Fj(n)15 b Fx(with)75 2336 y(en)o(tries:)179 2419 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 2425 y Fi(0)311 2419 y Fj(;)7 b(disp)406 2425 y Fi(0)425 2419 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 2425 y Fh(n)p Fk(\000)p Fi(1)671 2419 y Fj(;)g(disp)766 2425 y Fh(n)p Fk(\000)p Fi(1)830 2419 y Fx(\))p Fj(;)179 2510 y Fx(\()p Fj(ty)q(pe)271 2516 y Fi(0)291 2510 y Fj(;)g(disp)386 2516 y Fi(0)413 2510 y Fx(+)j Fj(extent)p Fx(\))p Fj(;)d(:::;)g Fx(\()p Fj(ty)q(pe)754 2516 y Fh(n)p Fk(\000)p Fi(1)818 2510 y Fj(;)g(disp)913 2516 y Fh(n)p Fk(\000)p Fi(1)987 2510 y Fx(+)i Fj(extent)p Fx(\))p Fj(;)e(:::;)179 2601 y Fx(\()p Fj(ty)q(pe)271 2607 y Fi(0)291 2601 y Fj(;)g(disp)386 2607 y Fi(0)413 2601 y Fx(+)j(\()p Fu(blocklen)d Fn(\000)j Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1062 2607 y Fh(n)p Fk(\000)p Fi(1)1126 2601 y Fj(;)g(disp)1221 2607 y Fh(n)p Fk(\000)p Fi(1)1295 2601 y Fx(+)i(\()p Fu(blocklen)f Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)179 2693 y Fx(\()p Fj(ty)q(pe)271 2699 y Fi(0)291 2693 y Fj(;)e(disp)386 2699 y Fi(0)413 2693 y Fx(+)j Fu(stride)e Fn(\001)g Fj(extent)p Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)914 2699 y Fh(n)p Fk(\000)p Fi(1)979 2693 y Fj(;)g(disp)1074 2699 y Fh(n)p Fk(\000)p Fi(1)1148 2693 y Fx(+)i Fu(stride)f Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)p eop %%Page: 27 27 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(27)179 45 y(\()p Fj(ty)q(pe)271 51 y Fi(0)291 45 y Fj(;)7 b(disp)386 51 y Fi(0)413 45 y Fx(+)j(\()p Fu(stride)e Fx(+)h Fu(blocklen)f Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 135 y Fx(\()p Fj(ty)q(pe)271 141 y Fh(n)p Fk(\000)p Fi(1)337 135 y Fj(;)g(disp)432 141 y Fh(n)p Fk(\000)p Fi(1)506 135 y Fx(+)j(\()p Fu(stride)e Fx(+)h Fu(blocklen)f Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(::::;)179 225 y Fx(\()p Fj(ty)q(pe)271 231 y Fi(0)291 225 y Fj(;)g(disp)386 231 y Fi(0)413 225 y Fx(+)j Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 315 y Fx(\()p Fj(ty)q(pe)271 321 y Fh(n)p Fk(\000)p Fi(1)337 315 y Fj(;)g(disp)432 321 y Fh(n)p Fk(\000)p Fi(1)506 315 y Fx(+)j Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 404 y Fx(\()p Fj(ty)q(pe)271 410 y Fi(0)291 404 y Fj(;)g(disp)386 410 y Fi(0)413 404 y Fx(+)j(\()p Fu(stride)e Fn(\001)h Fx(\()p Fu(count)f Fn(\000)h Fx(1\))g(+)h Fu(blocklen)e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 494 y Fx(\()p Fj(ty)q(pe)271 500 y Fh(n)p Fk(\000)p Fi(1)337 494 y Fj(;)g(disp)432 500 y Fh(n)p Fk(\000)p Fi(1)506 494 y Fx(+)j(\()p Fu(stride)e Fn(\001)g Fx(\()p Fu(count)h Fn(\000)g Fx(1\))g(+)h Fu(blocklen)e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fn(g)158 663 y Fo(Discussion:)158 708 y Ft(I)17 b(c)o(hanged)h Fm(count)d Ft(from)i(b)q(eing)h(the)f(n)o(um)o (b)q(er)h(of)f(en)o(tries)h(\(or)f(replicates\),)i(as)f(in)f(the)h(old)g (draft,)f(to)g(b)q(eing)i(the)75 754 y(n)o(um)o(b)q(er)13 b(of)f(blo)q(c)o (ks.)18 b(The)12 b(reason)h(is)g(that)f(in)h(the)g(next)f(calls)i(I)e(allo)o (w)h(blo)q(c)o(k)h(lengths)g(to)e(v)n(ary;)g(it)h(is)g(easy)g(to)f(sa)o(y)g (what)g(it)75 800 y(means)i(to)f(ha)o(v)o(e)h Fg(n)g Ft(blo)q(c)o(ks)h(of)e (sizes)h Fg(b)631 804 y Ff(0)649 800 y Fg(;)6 b(:::;)f(b)731 804 y Fe(n)p Fd(\000)p Ff(1)792 800 y Ft(;)13 b(it)h(is)g(harder)g(to)g (explain)i(what)d(it)h(means)g(to)f(ha)o(v)o(e)h Fg(n)g Ft(elemen)o(ts)g (\014lling)75 845 y(up)f(successiv)o(e)i(p)q(ositions)h(in)d(blo)q(c)o(ks)i (of)e(sizes)h Fg(b)779 849 y Ff(0)796 845 y Fg(;)6 b(b)829 849 y Ff(1)847 845 y Fg(;)g(::::)p Ft(.)158 891 y(W)m(e)16 b(lo)q(ose)h(the)g(abilit)o(y)h(to)e(use)h Fm(MPI)p 699 891 12 2 v 13 w(TYPE)p 792 891 V 12 w(VECTOR)d Ft(to)i(create)g(a)g(partially)j (\014lled)f(last)f(blo)q(c)o(k)g(\(the)f(more)g(general)75 937 y Fm(MPI)p 137 937 V 13 w(TYPE)p 230 937 V 13 w(INDEXED)d Ft(function)18 b(is)f(needed\);)h(w)o(e)e(gain)h(consistency)i(with)d(the)h (next)g(calls,)h(and)f(con)o(v)o(enience)h(in)f(these)75 982 y(calls.)158 1028 y(Opinions?)158 1160 y Fx(A)d(call)f(to)h Fu(MPI)p 399 1160 14 2 v 15 w(TYPE)p 502 1160 V 15 w(CONTIGUOUS\(oldty)o(pe,) 19 b(count,)h(newtype\))12 b Fx(is)i(equiv)n(alen)o(t)f(to)h(a)g(call)f(to)75 1210 y Fu(MPI)p 144 1210 V 15 w(TYPE)p 247 1210 V 15 w(VECTOR\(oldtype,)19 b(count,)h(1,)i(1,)f(newtype\))p Fx(.)158 1260 y(The)16 b(function)f Fu(MPI)p 478 1260 V 15 w(TYPE)p 581 1260 V 15 w(HVECTOR)f Fx(is)h(iden)o (tical)g(to)h Fu(MPI)p 1095 1260 V 15 w(TYPE)p 1198 1260 V 15 w(VECTOR)p Fx(,)d(except)k(that)f Fu(stride)e Fx(is)h(giv)o(en)75 1310 y(in)i(b)o(ytes,)i(rather)f(then)g(in)f(elemen)o(ts.)29 b(The)18 b(use)h(for)e(b)q(oth)g(t)o(yp)q(es)i(of)e(v)o(ector)h(constructors) i(is)d(illustrated)g(in)75 1360 y(Section)d(1.13.4.)i(\()p Fu(H)e Fx(is)g(used)h(for)e(\\heterogeneous"\).)75 1457 y Fu(MPI)p 144 1457 V 15 w(TYPE)p 247 1457 V 15 w(HVECTOR\()20 b(count,)h(oldtype,)f (stride,)g(blocklen,)g(newtype\))117 1507 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j(\(in)o(teger\))117 1552 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob) r(ject)117 1597 y(in)g Fu(stride)653 b Ft(n)o(um)o(b)q(er)14 b(of)e(b)o(ytes)i(b)q(et)o(w)o(een)f(start)g(of)g(eac)o(h)g(blo)q(c)o(k)i (\(in)o(teger\))117 1643 y(in)f Fu(blocklen)609 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g(\(p)q(ositiv)o(e)h(in)o (teger\))117 1688 y(out)f Fu(newtype)607 b Ft(handle)14 b(to)f(output)h (datat)o(yp)q(e)g(ob)r(ject)158 1764 y Fu(MPI)p 227 1764 V 15 w(TYPE)p 330 1764 V 15 w(INDEXED)f Fx(is)h(a)h(more)e(general)i(function)f (that)h(allo)o(ws)e(replication)h(of)g(an)g(old)g(datat)o(yp)q(e)g(in)o(to)g (a)75 1814 y(sequence)i(of)e(con)o(tiguous)f(blo)q(c)o(ks,)h(where)h(blo)q(c) o(k)f(lengths)g(and)g(blo)q(c)o(k)g(displacemen)o(ts)f(can)h(all)f(di\013er)i ({)e(but)i(are)75 1864 y(all)e(m)o(ultiples)f(of)h(the)h(old)g(t)o(yp)q(e)g (exten)o(t.)75 1961 y Fu(MPI)p 144 1961 V 15 w(TYPE)p 247 1961 V 15 w(INDEXED\()20 b(count,)h(oldtype,)f(array)p 917 1961 V 14 w(of)p 975 1961 V 16 w(indices,)g(array)p 1297 1961 V 14 w(of)p 1355 1961 V 16 w(blocklen,)g(newtype\))117 2011 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j(\(in)o (teger\))117 2056 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o (yp)q(e)f(ob)r(ject)117 2102 y(in)g Fu(array)p 275 2102 V 15 w(of)p 334 2102 V 15 w(indices)447 b Ft(displacemen)o(t)14 b(for)d(eac)o(h)h(blo)q(c)o(k,)g(in)g(m)o(ultiples)i(of)d Fm(oldtype)e Ft(ex-)905 2151 y(ten)o(t)k(\(arra)o(y)h(of)e(in)o(teger\))117 2197 y(in)i Fu(array)p 275 2197 V 15 w(of)p 334 2197 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(p)q(er)g(blo)q(c)o(k)g(\(arra)o (y)f(of)g(in)o(teger\))117 2242 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g(ob)r(ject)158 2318 y Fx(Example:)j(let)d Fu(oldtype)e Fx(b)q(e)i(a)g(handle)g(to)f(the)i(t)o(yp)q(e)179 2400 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 2482 y Fx(with)15 b(exten)o(t)h(16.)22 b(Let)16 b Fu(I)21 b(=)h(\(64,)f(0\))15 b Fx(and)g(let)g Fu(B)22 b(=)f(\(3,)g(1\))p Fx(.)h(A)15 b(call)g(to)g Fu(MPI)p 1346 2482 V 15 w(TYPE)p 1449 2482 V 15 w(INDEXED\(2,)20 b(oldtype,)75 2532 y(I,)h(B,)h(newtype\))12 b Fx(returns)j(in)f Fu(newtype)e Fx(a)i(handle)g(to)f(the)i(t)o(yp)q(e)179 2614 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(64\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(72\))p Fj(;)g Fx(\()p Fu(double)n Fj(;)g Fx(80\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(88\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(96\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(104\))179 2704 y(\()p Fu(double)o Fj(;)g Fx(0\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(8\))p Fn(g)j Fx(:)p eop %%Page: 28 28 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(28)75 45 y(three)15 b(copies)g(of)e(the)h(old)g(t)o(yp)q(e)g (starting)g(at)g(displacemen)o(t)f(64,)g(and)g(one)h(cop)o(y)g(starting)g(at) g(displacemen)o(t)f(0.)158 95 y(In)h(general,)g(assume)f(that)h Fu(oldtype)e Fx(is)i(a)g(handle)f(to)h(the)h(datat)o(yp)q(e)179 178 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 184 y Fi(0)311 178 y Fj(;)7 b(disp)406 184 y Fi(0)425 178 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)607 184 y Fh(n)p Fk(\000)p Fi(1)671 178 y Fj(;)g(disp)766 184 y Fh(n)p Fk(\000)p Fi(1)830 178 y Fx(\))p Fn(g)p Fj(;)75 261 y Fx(with)17 b(exten)o(t)h Fp(extent)p Fx(.)29 b(Let)17 b Fu(B)g Fx(b)q(e)h(the)g Fu(array)p 816 261 14 2 v 15 w(of)p 875 261 V 15 w(blocklen)e Fx(parameter)h(and)g Fu(I)g Fx(b)q(e)h(the)g Fu(array)p 1652 261 V 14 w(of)p 1710 261 V 15 w(indices)75 318 y Fx(parameter.)f(The)12 b(newly)f(created)i(t)o(yp)q(e)f(returned)h(b)o (y)e Fu(newtype)f Fx(is)h(the)h(sequence)h(of)e Fj(n)t Fn(\001)1474 287 y Fc(P)1517 297 y Fu(count)o Fk(\000)p Fi(1)1517 331 y Fh(i)p Fi(=0)1678 318 y Fu(B)p Fx([)p Fu(i)p Fx(])f(en)o(tries)179 401 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 407 y Fi(0)311 401 y Fj(;)d(disp)406 407 y Fi(0)434 401 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])f Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)871 407 y Fh(n)p Fk(\000)p Fi(1)935 401 y Fj(;)g(disp)1030 407 y Fh(n)p Fk(\000)p Fi(1)1104 401 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])e Fn(\001)g Fj(extent)p Fx(\))p Fj(;)f(:::;)179 493 y Fx(\()p Fj(ty)q(pe)271 499 y Fi(0)291 493 y Fj(;)g(disp)386 499 y Fi(0)413 493 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(0)p Fx(])e(+)h Fu(B)p Fx([)p Fu(0)p Fx(])f Fn(\000)i Fx(1\))f Fn(\001)f Fj(extent)p Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)1071 499 y Fh(n)p Fk(\000)p Fi(1)1136 493 y Fj(;)g(disp)1231 499 y Fh(n)p Fk(\000)p Fi(1)1305 493 y Fx(+)i(\()p Fu(I)p Fx([)p Fu(0)p Fx(])f(+)i Fu(B)p Fx([)p Fu(0)p Fx(])e Fn(\000)h Fx(1\))g Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 584 y Fx(\()p Fj(ty)q(pe)271 590 y Fi(0)291 584 y Fj(;)g(disp)386 590 y Fi(0)413 584 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1010 590 y Fh(n)p Fk(\000)p Fi(1)1074 584 y Fj(;)g(disp)1169 590 y Fh(n)p Fk(\000)p Fi(1)1243 584 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\001)h Fj(extent)p Fx(\))p Fj(;)e(:::;)179 675 y Fx(\()p Fj(ty)q(pe)271 681 y Fi(0)291 675 y Fj(;)g(disp)386 681 y Fi(0)413 675 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fj(;)e(:::;)179 767 y Fx(\()p Fj(ty)q(pe)271 773 y Fh(n)p Fk(\000)p Fi(1)337 767 y Fj(;)g(disp)432 773 y Fh(n)p Fk(\000)p Fi(1)506 767 y Fx(+)j(\()p Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)p Fx(\))p Fn(g)p Fj(:)158 858 y Fx(A)15 b(call)e(to)i Fu(MPI)p 401 858 V 15 w(TYPE)p 504 858 V 15 w(VECTOR\(count,)k(oldtype,)h(stride,)g(blocklen,)g(newtype\))13 b Fx(is)h(equiv)n(alen)o(t)g(to)75 908 y(a)g(call)f(to)75 957 y Fu(MPI)p 144 957 V 15 w(TYPE)p 247 957 V 15 w(INDEX\(count,)19 b(oldtype,)h(I,)i(B,)f(newtype\))12 b Fx(where)179 1040 y Fu(I)p Fx([)p Fu(j)p Fx(])e(=)i Fj(j)f Fn(\001)e Fu(stride)k Fj(;)7 b(j)13 b Fx(=)f(0)p Fj(;)7 b(:::;)g Fu(count)f Fn(\000)j Fx(1)p Fj(;)75 1123 y Fx(and)179 1207 y Fu(B)p Fx([)p Fu(j)p Fx(])h(=)i Fu(blocklen)g Fj(;)7 b(j)13 b Fx(=)f(0)p Fj(;)7 b(:::;)g Fu(count)f Fn(\000)j Fx(1)p Fj(:)158 1290 y Fx(The)k(function)g Fu(MPI)p 473 1290 V 15 w(TYPE)p 576 1290 V 15 w(HINDEXED)e Fx(is)h(iden)o(tical)g(to)h Fu(MPI)p 1100 1290 V 15 w(TYPE)p 1203 1290 V 15 w(INDEXED)p Fx(,)e(except)j(that)f(blo)q(c)o(k)f(displace-)75 1339 y(men)o(ts)h(in)h Fu(array)p 358 1339 V 14 w(of)p 416 1339 V 16 w(indices)e Fx(are)i(sp)q (eci\014ed)i(in)d(b)o(ytes,)h(rather)h(than)f(in)f(m)o(ultiples)f(of)i(the)g (old)f(t)o(yp)q(e)i(exten)o(t.)75 1436 y Fu(MPI)p 144 1436 V 15 w(TYPE)p 247 1436 V 15 w(HINDEXED\()20 b(count,)g(oldtype,)h(array)p 939 1436 V 14 w(of)p 997 1436 V 15 w(indices,)g(array)p 1319 1436 V 14 w(of)p 1377 1436 V 16 w(blocklen,)e(newtype\))117 1486 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j (\(in)o(teger\))117 1532 y(in)f Fu(oldtype)631 b Ft(handle)14 b(to)f(input)i(datat)o(yp)q(e)f(ob)r(ject)117 1578 y(in)g Fu(array)p 275 1578 V 15 w(of)p 334 1578 V 15 w(indices)447 b Ft(b)o(yte)13 b(displacemen)o(t)j(of)c(eac)o(h)i(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o (teger\))117 1623 y(in)h Fu(array)p 275 1623 V 15 w(of)p 334 1623 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g (eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o(teger\))117 1669 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g (ob)r(ject)158 1745 y Fu(MPI)p 227 1745 V 15 w(TYPE)p 330 1745 V 15 w(STRUCT)j Fx(is)h(the)g(most)f(general)i(constructor:)28 b(It)18 b(further)h(generalizes)g(the)g(previous)f(one)g(in)75 1795 y(that)c(it)f(allo)o(ws)g(eac)o(h)h(blo)q(c)o(k)g(to)g(consist)g(of)g (replications)f(of)h(a)f(di\013eren)o(t)i(datat)o(yp)q(e.)75 1892 y Fu(MPI)p 144 1892 V 15 w(TYPE)p 247 1892 V 15 w(STRUCT\(count,)k (array)p 677 1892 V 15 w(of)p 736 1892 V 15 w(types,)i(array)p 1014 1892 V 14 w(of)p 1072 1892 V 15 w(indices,)g(array)p 1394 1892 V 14 w(of)p 1452 1892 V 16 w(blocklen,)e(newtype\))117 1942 y Ft(in)14 b Fu(count)675 b Ft(n)o(um)o(b)q(er)14 b(of)e(blo)q(c)o(ks)j (\(in)o(teger\))117 1988 y(in)f Fu(array)p 275 1988 V 15 w(of)p 334 1988 V 15 w(types)491 b Ft(t)o(yp)q(e)17 b(of)g(elemen)o(ts)h(in)f(eac)o (h)h(blo)q(c)o(k)g(\(arra)o(y)f(of)g(datat)o(yp)q(e)h(ob-)905 2037 y(jects\))117 2083 y(in)c Fu(array)p 275 2083 V 15 w(of)p 334 2083 V 15 w(indices)447 b Ft(b)o(yte)13 b(displacemen)o(t)j(of)c(eac)o(h) i(blo)q(c)o(k)g(\(arra)o(y)f(of)g(in)o(teger\))117 2129 y(in)h Fu(array)p 275 2129 V 15 w(of)p 334 2129 V 15 w(blocklen)425 b Ft(n)o(um)o(b)q(er)14 b(of)e(elemen)o(ts)i(in)g(eac)o(h)g(blo)q(c)o(k)g (\(arra)o(y)f(of)g(in)o(teger\))117 2174 y(out)h Fu(newtype)607 b Ft(handle)14 b(to)f(output)h(datat)o(yp)q(e)g(ob)r(ject)158 2251 y Fx(Example:)j(Let)d Fu(type1)f Fx(b)q(e)h(a)g(handle)g(to)f(the)i(t)o (yp)q(e)179 2334 y Fn(f)p Fx(\()p Fu(double)n Fj(;)7 b Fx(0\))p Fj(;)g Fx(\()p Fu(char)o Fj(;)g Fx(8\))p Fn(g)p Fj(;)75 2417 y Fx(with)14 b(exten)o(t)i(16.)j(Let)c Fu(I)22 b(=)f(\(0,)g(16,)h(26\))p Fx(,)13 b Fu(B)22 b(=)f(\(2,)g(1,)h(3\))14 b Fx(and)g Fu(T)22 b(=)f(\(MPI)p 1356 2417 V 15 w(FLOAT,)g(type1,)f(MPI)p 1742 2417 V 16 w(BYTE\))p Fx(.)75 2466 y(Then)11 b(a)f(call)f(to)h Fu(MPI)p 399 2466 V 15 w(TYPE)p 502 2466 V 15 w(STRUC\(3,)20 b(T,)i(I,)f(B,)h(newtype\))8 b Fx(returns)k(in)e Fu(newtype)f Fx(a)h(handle)g(to)g(the)h(datat)o(yp)q(e)179 2549 y Fn(f)p Fx(\()p Fu(float)o Fj(;)c Fx(0\))p Fj(;)g Fx(\()p Fu(float)n Fj(;)g Fx(4\))p Fj(;)g Fx(\()p Fu(double)m Fj(;)g Fx(16\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(24\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(26\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(27\))p Fj(;)g Fx(\()p Fu(char)n Fj(;)g Fx(28\))p Fn(g)j Fx(:)75 2633 y(t)o(w)o(o)17 b(copies)h(of)e Fu(MPI)p 402 2633 V 15 w(FLOAT)g Fx(starting)i(at)f(0,)g(follo)o(w)o(ed)f(b)o(y)h(one)g(cop)o(y)g(of)g Fu(type1)f Fx(starting)h(at)g(16,)h(follo)o(w)o(ed)d(b)o(y)75 2682 y(three)g(copies)g(of)e Fu(MPI)p 420 2682 V 15 w(CHAR)p Fx(,)g(starting)g(at)h(26.)k(\(W)m(e)13 b(assume)h(that)g(a)f Fu(float)g Fx(o)q(ccupies)i(four)f(b)o(ytes.\))p eop %%Page: 29 29 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(29)158 45 y(In)14 b(general,)g(let)g Fu(T)f Fx(b)q(e)i(the)f Fu(array)p 701 45 14 2 v 15 w(of)p 760 45 V 15 w(types)f Fx(parameter,)g(where)i Fu(T[i])e Fx(is)h(a)f(handle)h(to)179 124 y Fj(ty)q(pe)255 130 y Fh(i)281 124 y Fx(=)e Fn(f)p Fx(\()p Fj(ty)q(pe)438 107 y Fh(i)438 134 y Fi(0)458 124 y Fj(;)7 b(disp)553 107 y Fh(i)553 134 y Fi(0)571 124 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)753 107 y Fh(i)753 134 y(n)774 138 y Fb(i)787 124 y Fj(;)g(disp)882 107 y Fh(i)882 134 y(n)903 138 y Fb(i)918 124 y Fx(\))p Fn(g)p Fj(;)75 203 y Fx(with)15 b(exten)o(t)i Fj(extent)417 209 y Fh(i)431 203 y Fx(.)24 b(Let)16 b Fu(B)g Fx(b)q(e)g(the)g Fu(array)p 825 203 V 15 w(of)p 884 203 V 15 w(blocklen)e Fx(parameter)i(and)f Fu(I)h Fx(b)q(e)g(the)h Fu(array)p 1652 203 V 14 w(of)p 1710 203 V 15 w(indices)75 260 y Fx(parameter.)g(Then)c (the)h(newly)e(created)i(datat)o(yp)q(e)f(is)g(the)g(sequence)i(of)d(length) 1355 229 y Fc(P)1398 239 y Fu(count)o Fk(\000)p Fi(1)1398 273 y Fh(i)p Fi(=0)1559 260 y Fu(B)p Fx([)p Fu(i)p Fx(])5 b Fn(\001)i Fj(extent)1768 266 y Fh(i)1794 260 y Fx(with)75 310 y(en)o(tries)179 389 y Fn(f)p Fx(\()p Fj(ty)q(pe)292 372 y Fi(0)292 399 y(0)311 389 y Fj(;)g(disp)406 372 y Fi(0)406 399 y(0)434 389 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])o(\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)724 372 y Fi(0)724 399 y Fh(n)745 403 y Fs(0)762 389 y Fj(;)g(disp)857 372 y Fi(0)857 399 y Fh(n)878 403 y Fs(0)904 389 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])o(\))p Fj(;)d(:::;)179 480 y Fx(\()p Fj(ty)q(pe)271 463 y Fi(0)271 490 y(0)291 480 y Fj(;)g(disp)386 463 y Fi(0)386 490 y(0)413 480 y Fx(+)j Fu(I)p Fx([)p Fu(0)p Fx(])e(+)h(\()p Fu(B)p Fx([)p Fu(0)p Fx(])f Fn(\000)i Fx(1\))f Fn(\001)f Fj(extent)889 486 y Fi(0)908 480 y Fx(\))p Fj(;)f(:::;)g Fx(\()p Fj(ty)q(pe)1090 463 y Fi(0)1090 490 y Fh(n)1111 494 y Fs(0)1128 480 y Fj(;)g(disp)1223 463 y Fi(0)1223 490 y Fh(n)1244 494 y Fs(0)1271 480 y Fx(+)i Fu(I)p Fx([)p Fu(0)p Fx(])f(+)i(\()p Fu(B)p Fx([)p Fu(0)p Fx(])e Fn(\000)h Fu(1)p Fx(\))g Fn(\001)g Fj(extent)1748 486 y Fi(0)1767 480 y Fx(\))p Fj(;)e(:::;)179 576 y Fx(\()p Fj(ty)q(pe)271 558 y Fu(count)p Fk(\000)p Fi(1)271 587 y(0)425 576 y Fj(;)g(disp)520 558 y Fu(count)o Fk(\000)p Fi(1)520 587 y(0)683 576 y Fx(+)i Fu(I)p Fx([)p Fu(count)f Fn(\000)h Fu(1)p Fx(]\))p Fj(;)e(:::;)g Fx(\()p Fj(ty)q(pe)1133 558 y Fu(count)n Fk(\000)p Fi(1)1133 586 y Fh(n)1154 599 y Fu(count)m Fa(\000)p Fs(1)1302 576 y Fj(;)g(disp)1397 558 y Fu(count)o Fk(\000)p Fi(1)1397 586 y Fh(n)1418 599 y Fu(count)n Fa(\000)p Fs(1)1576 576 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])o(\))p Fj(;)d(:::;)179 684 y Fx(\()p Fj(ty)q(pe)271 666 y Fu(count)p Fk(\000)p Fi(1)271 695 y(0)425 684 y Fj(;)g(disp)520 666 y Fu(count)o Fk(\000)p Fi(1)520 695 y(0)683 684 y Fx(+)i Fu(I)p Fx([)p Fu(count)f Fn(\000)h Fu(1)p Fx(])g(+)h(\()p Fu(B)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e Fn(\000)i Fx(1\))f Fn(\001)g Fj(extent)1479 692 y Fu(count)o Fk(\000)p Fi(1)1632 684 y Fx(\))p Fj(;)e(:::;)179 776 y Fx(\()p Fj(ty)q(pe)271 759 y Fu(count)p Fk(\000)p Fi(1)271 787 y Fh(n)292 800 y Fu(count)o Fa(\000)p Fs(1)442 776 y Fj(;)g(disp)537 759 y Fu(count)o Fk(\000)p Fi(1)537 787 y Fh(n)558 800 y Fu(count)n Fa(\000)p Fs(1)716 776 y Fx(+)j Fu(I)p Fx([)p Fu(count)d Fn(\000)j Fu(1)p Fx(])e(+)i(\()p Fu(B)p Fx([)p Fu(count)e Fn(\000)h Fu(1)p Fx(])g Fn(\000)g Fu(1)p Fx(\))g Fn(\001)g Fj(extent)1513 785 y Fu(count)o Fk(\000)p Fi(1)1667 776 y Fx(\))p Fn(g)158 873 y Fx(A)14 b(call)f(to)h Fu(MPI)p 399 873 V 15 w(TYPE)p 502 873 V 15 w(HINDEXED\()20 b(count,)g(oldtype,)g(I,)i(B,)f(newtype\))12 b Fx(is)i(equiv)n(alen)o(t)f(to)h(a)g(call)f(to)75 922 y Fu(MPI)p 144 922 V 15 w(TYPE)p 247 922 V 15 w(STRUC\()20 b(count,)h(T,)g(I,)h(B,)f (newtype\))p Fx(,)12 b(where)j(eac)o(h)f(en)o(try)h(of)e Fu(T)h Fx(is)f(equal)h(to)g Fu(oldtype)p Fx(.)158 1051 y Fo(Discussion:)158 1096 y Ft(I)f(am)g(op)q(en)g(to)g(suggestions)j(for)c(b)q(etter)i(names)f(to) g(these)g(functions.)75 1294 y Fq(1.13.2)49 b(Additional)19 b(functions)75 1371 y Fx(The)c(displacemen)o(ts)g(in)g(a)f(general)h(datat)o (yp)q(e)h(are)f(relativ)o(e)g(to)g(some)f(start)h(address.)23 b Fv(Absolute)15 b(addresses)75 1421 y Fx(can)10 b(b)q(e)g(substituted)h(for) e(these)i(displacemen)o(ts:)16 b(w)o(e)9 b(treat)i(them)d(as)i(displacemen)o (ts)f(relativ)o(e)h(to)f(\\address)i(zero",)75 1471 y(the)17 b(start)g(of)f(the)h(address)h(space.)26 b(This)16 b(initial)f(address)j (zero)f(is)f(indicated)h(b)o(y)f(the)h(constan)o(t)g Fu(MPI)p 1721 1471 V 15 w(BOTTOM)p Fx(.)75 1520 y(Th)o(us,)h(a)f(datat)o(yp)q(e)g(can) h(sp)q(ecify)f(the)h(absolute)f(address)i(of)e(the)g(en)o(tries)i(in)d(the)i (comm)o(unication)c(bu\013er,)k(in)75 1570 y(whic)o(h)c(case)h(the)f Fu(start)f Fx(parameter)g(is)h(passed)h(the)g(v)n(alue)e Fu(MPI)p 1096 1570 V 15 w(BOTTOM)p Fx(.)158 1620 y(The)h(address)i(of)d(a)g(lo)q (cation)g(in)h(memory)d(can)j(b)q(e)h(found)e(b)o(y)h(in)o(v)o(oking)e(the)i (function)g Fu(MPI)p 1616 1620 V 15 w(ADDRESS)p Fx(.)75 1717 y Fu(MPI)p 144 1717 V 15 w(ADDRESS\(location,)k(address\))117 1767 y Ft(in)c Fu(location)609 b Ft(lo)q(cation)15 b(in)f(caller)g(memory)g (\(c)o(hoice\))117 1812 y(out)g Fu(address)607 b Ft(address)14 b(of)f(lo)q(cation)i(\(in)o(teger\))158 1888 y Fx(Returns)g(the)f(\(b)o (yte\))h(address)g(of)e Fu(location)p Fx(.)158 1938 y(Another)h(useful)g (auxiliary)d(function)i(is)g Fu(MPI)p 883 1938 V 15 w(EXTENT)p Fx(,)f(that)h(returns)i(the)f(exten)o(t)g(of)f(a)g(datat)o(yp)q(e)h({)f (where)75 1987 y(exten)o(t)i(is)f(as)g(de\014ned)h(in)e(Eq.)g(1.1.)75 2085 y Fu(MPI)p 144 2085 V 15 w(TYPE)p 247 2085 V 15 w(EXTENT\(datatype,)18 b(extent\))117 2134 y Ft(in)c Fu(datatype)609 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r(ject)117 2179 y(out)g Fu(extent)629 b Ft(datat)o(yp)q(e)14 b(exten)o(t)f(\(in)o(teger\))158 2255 y Fx(As)i(de\014ned)h(no)o(w,)f(the)g(exten)o(t)h(of)e(a)h(datat)o(yp)q(e)g (go)q(es)h(from)d(its)i(\014rst)g(o)q(ccupied)h(b)o(yte)g(to)e(its)h(last)g (o)q(ccupied)75 2305 y(b)o(yte.)35 b(It)20 b(is)f(often)h(con)o(v)o(enien)o (t)f(to)h(de\014ne)g(a)f(datat)o(yp)q(e)h(that)g(has)f(a)g(\\hole")g(at)h (its)f(b)q(eginning)g(and/or)g(its)75 2355 y(end;)d(or)f(a)g(datat)o(yp)q(e)h (with)f(en)o(tries)h(that)f(extend)i(b)q(ey)o(ond)e(the)h(datat)o(yp)q(e)f (\\exten)o(t".)23 b(This)15 b(allo)o(ws)f(to)h(use)h(the)75 2405 y(functions)f Fu(MPI)p 324 2405 V 15 w(TYPE)p 427 2405 V 15 w(CONTIGUOUS)p Fx(,)d Fu(MPI)p 752 2405 V 15 w(TYPE)p 855 2405 V 15 w(VECTOR)p Fx(,)h(or)i Fu(MPI)p 1145 2405 V 15 w(TYPE)p 1248 2405 V 15 w(INDEX)f Fx(to)h(replicate)g(a)g(more)f(general)75 2455 y(pattern;)22 b(examples)c(of)h(suc)o(h)h(a)f(usage)g(are)h(pro)o(vided) f(in)g(Section)g(1.13.4.)32 b(T)m(o)19 b(ac)o(hiev)o(e)g(this,)h(w)o(e)f(add) g(t)o(w)o(o)75 2504 y(additional)10 b(\\pseudo-datat)o(yp)q(es")j Fu(MPI)p 707 2504 V 15 w(LB)p Fx(,)e(and)h Fu(MPI)p 934 2504 V 15 w(UB)g Fx(that)g(can)g(b)q(e)h(used,)f(resp)q(ectiv)o(ely)m(,)h(to)f (mark)f(the)h(lo)o(w)o(er)75 2554 y(b)q(ound)g(or)g(the)g(upp)q(er)h(b)q (ound)f(of)f(a)h(datat)o(yp)q(e.)17 b(these)d(pseudo-datat)o(yp)q(es)f(o)q (ccup)o(y)f(no)g(space)h(\()p Fj(extent)p Fx(\()p Fu(MP)p 1759 2554 V 15 w(LB)p Fx(\))f(=)75 2604 y Fj(extent)p Fx(\()p Fu(MP)p 255 2604 V 16 w(UB)o Fx(\))20 b(=)g(0\).)32 b(They)19 b(do)g(not)g(a\013ect)g (the)h(signature)f(of)f(a)g(message)h(created)h(with)e(this)h(datat)o(yp)q (e.)75 2654 y(Ho)o(w)o(ev)o(er,)13 b(they)h(do)f(a\013ect)h(the)f (de\014nition)g(of)f(the)i(exten)o(t)g(of)e(a)h(datat)o(yp)q(e)g(and,)g (therefore,)h(a\013ect)g(the)g(outcome)75 2704 y(of)f(a)h(replication)f(of)h (this)f(datat)o(yp)q(e)i(b)o(y)e(a)h(datat)o(yp)q(e)g(constructor.)p eop %%Page: 30 30 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(30)158 45 y(Example:)16 b(Let)e Fu(I)21 b(=)h(\(-3,)f(0,)g (6\))p Fx(;)12 b Fu(T)22 b(=)g(\(MPI)p 928 45 14 2 v 14 w(LB,)g(MPI)p 1096 45 V 15 w(INT,)f(MPI)p 1286 45 V 15 w(UB\))p Fx(,)12 b(and)g Fu(B)22 b(=)g(\(1,)f(1,)g(1\))p Fx(.)c(Then)75 95 y(a)22 b(call)g(to)g Fu(MPI)p 331 95 V 15 w(TYPE)p 434 95 V 15 w(STRUC\(3,)e(T,)i(I,)f(B,)g (type1\))h Fx(creates)i(a)e(new)h(datat)o(yp)q(e)g(that)f(has)h(an)f(exten)o (t)i(of)75 145 y(9)c(\(from)g(-3)g(to)h(6\),)g(and)g(con)o(tains)g(an)f(in)o (teger)i(at)e(displacemen)o(t)g(0.)39 b(This)20 b(is)h(the)g(datat)o(yp)q(e)g (de\014ned)h(b)o(y)75 195 y(the)h(sequence)h Fn(f)p Fu(\(lb,)c(-3\),)h (\(int,)g(0\),)g(\(ub,)g(6\))p Fn(g)g Fx(.)43 b(If)21 b(this)h(t)o(yp)q(e)h (is)f(replicated)g(t)o(wice)h(b)o(y)e(a)h(call)f(to)75 244 y Fu(MPI)p 144 244 V 15 w(TYPE)p 247 244 V 15 w(CONTIGUOUS\(2,)e(type1,)i (type2\))d Fx(then)i(the)h(newly)e(created)j(t)o(yp)q(e)e(can)g(b)q(e)g (describ)q(ed)i(b)o(y)d(the)75 294 y(sequence)d Fn(f)p Fu(\(lb,)21 b(-3\),)g(\(int,)g(0\),)g(\(int,9\),)f(\(ub,)h(15\))p Fn(g)g Fx(.)d(\(En)o(tries)d(of)e(t)o(yp)q(e)i Fu(lb)f Fx(or)f Fu(ub)h Fx(can)g(b)q(e)h(deleted)75 344 y(if)e(they)h(are)h(not)f(at)f(the)i(endp)q (oin)o(ts)f(of)f(the)i(datat)o(yp)q(e.\))158 394 y(In)f(general,)g(if)179 475 y Fj(D)q(ataty)q(pe)f Fx(=)f Fn(f)p Fx(\()p Fj(ty)q(pe)519 481 y Fi(0)538 475 y Fj(;)7 b(disp)633 481 y Fi(0)651 475 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)833 481 y Fh(n)p Fk(\000)p Fi(1)897 475 y Fj(;)g(disp)992 481 y Fh(n)p Fk(\000)p Fi(1)1057 475 y Fx(\))p Fn(g)p Fj(;)75 556 y Fx(then)15 b(the)f Fv(lo)o(w)o(er)g(b)q (ound)f Fx(of)g Fj(D)q(ataty)q(pe)i Fx(is)f(de\014ned)h(to)f(b)q(e)179 660 y Fj(l)q(b)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f(=)469 602 y Fc(\032)521 635 y Fx(min)590 641 y Fh(j)614 635 y Fj(disp)690 641 y Fh(j)1036 635 y Fx(if)g(no)h(en)o(try)g(has)h(basic)f(t)o(yp)q(e)g Fj(ty)q(pe)1588 641 y Fh(j)1618 635 y Fx(=)e Fu(lb)521 685 y Fx(min)n Fn(f)p Fj(disp)687 691 y Fh(j)730 685 y Fx(:)25 b Fj(ty)q(pe)843 691 y Fh(j)873 685 y Fx(=)12 b Fu(lb)o Fn(g)41 b Fx(otherwise)158 764 y(Similarly)l(,)11 b(the)j Fv(upp)q(er)g(b)q(ound)f Fx(of)g Fj(D)q(ataty)q(pe)i Fx(is)f(de\014ned)h(to)f(b)q(e)179 868 y Fj(ub)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f(=)479 810 y Fc(\032)531 843 y Fx(max)609 849 y Fh(j)633 843 y Fj(disp)709 849 y Fh(j)736 843 y Fx(+)c Fj(siz)r(eof)t Fx(\()p Fj(ty)r(pe)987 849 y Fh(j)1007 843 y Fx(\))56 b(if)13 b(no)g(en)o(try)i(has)f(basic)g(t)o (yp)q(e)g Fj(ty)q(pe)1630 849 y Fh(j)1660 843 y Fx(=)e Fu(ub)531 893 y Fx(max)o Fn(f)p Fj(disp)706 899 y Fh(j)748 893 y Fx(:)25 b Fj(ty)q(pe)861 899 y Fh(j)891 893 y Fx(=)12 b Fu(ub)p Fn(g)65 b Fx(otherwise)158 972 y(Then)179 1054 y Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))13 b(=)f Fj(ub)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))f Fn(\000)e Fj(l)q(b)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\);)158 1135 y(F)m(urthermore,)h(if)e Fj(ty)q(pe)516 1141 y Fh(i)540 1135 y Fx(requires)j(alignemen)o(t)c(to)i(an)g(address)i(that)e(is)g(a)g(m)o (ultiple)e(of)i Fj(k)1516 1141 y Fh(i)1529 1135 y Fx(,)h(then)g Fj(extent)p Fx(\()p Fj(D)q(ataty)q(pe)p Fx(\))75 1185 y(is)i(further)h (rounded)f(up)g(to)g(the)g(nearest)i(m)o(ultiple)9 b(of)j(max)1013 1191 y Fh(i)1034 1185 y Fj(k)1056 1191 y Fh(i)1069 1185 y Fx(.)17 b(The)c(formal)c(de\014nitions)j(giv)o(en)g(for)f(the)i(v)n(arious)75 1235 y(datat)o(yp)q(e)h(constructors)i(apply)d(no)o(w,)g(with)h(the)g (amended)g(de\014nition)f(of)g Fv(exten)o(t)p Fx(.)158 1284 y(A)j(datat)o(yp)q(e)g(ob)r(ject)h(has)f(to)g(b)q(e)h Fv(committed)c Fx(b)q(efore)k(it)f(can)g(b)q(e)h(used)g(in)e(a)h(comm)o(unication.)21 b(A)16 b(com-)75 1334 y(mitted)h(datat)o(yp)q(e)i(ob)r(ject)h(cannot)e(b)q(e) i(used)f(to)f(build)g(new)h(deriv)o(ed)g(datat)o(yp)q(es.)33 b(The)19 b(system)f(ma)o(y)f(use)i(a)75 1384 y(di\013eren)o(t)e(in)o(ternal)g (represen)o(tation)h(for)e(committed)e(datat)o(yp)q(e)j(ob)r(ject)g(so)g(as)f (to)g(facilitate)g(comm)o(unicatio)o(n,)75 1434 y(e.g.)h(c)o(hange)c(from)e (a)i(compacted)f(represen)o(tation)j(to)d(a)h(\015at)f(represen)o(tation)i (of)f(the)g(datat)o(yp)q(e,)g(and)f(select)i(the)75 1484 y(most)f(con)o(v)o (enien)o(t)h(transfer)h(mec)o(hanism.)75 1581 y Fu(MPI)p 144 1581 V 15 w(TYPE)p 247 1581 V 15 w(COMMIT\(intype,)k(outtype\))117 1631 y Ft(in)14 b Fu(intype)653 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r (ject)g(to)f(b)q(e)g(committed)117 1676 y(out)h Fu(outtype)607 b Ft(handle)14 b(to)f(committed)h(datat)o(yp)q(e)g(ob)r(ject)158 1752 y Fx(There)f(is)f(no)g(need)h(to)e(commit)e(basic)j(datat)o(yp)q(es;)h (they)f(are)h(\\pre-committed".)i(They)d(are)h(the)f(exception)75 1802 y(to)i(the)g(rule)g(that)g(committed)e(datat)o(yp)q(es)i(cannot)g(b)q(e) h(used)g(to)e(form)g(deriv)o(ed)h(datat)o(yp)q(es.)158 1930 y Fo(Discussion:)158 1976 y Ft(1.)j(Do)c(w)o(e)g(really)i(need)e(a)g(commit)h (here?)158 2022 y(2.)j(Do)c(w)o(e)g(w)o(an)o(t)g(one)g(inout)h(argumen)o(t,)g (or)f(t)o(w)o(o)f(distinct)j(argumen)o(ts?)158 2067 y(3.)24 b(Are)15 b(committed)h(and)g(uncommitted)i(ob)r(jects)d(of)g(the)h(same)f(t)o (yp)q(e?)25 b(\(The)16 b(answ)o(er)f(m)o(ust)h(b)q(e)f(y)o(es,)h(if)g(it)f (is)h(an)75 2113 y(inout)e(argumen)o(t.\))158 2159 y(4.)k(If)13 b(w)o(e)f(ha)o(v)o(e)i(only)h(one)e(inout)i(argumen)o(t,)f(do)q(es)g(a)f (commit)h(c)o(hange)g(the)g(datat)o(yp)q(e)g(ob)r(ject,)f(or)h(do)q(es)g(it)f (create)h(a)75 2204 y(new)d(ob)r(ject)g(and)h(return)g(a)f(p)q(oin)o(ter)h (to)f(it)h(in)g Fm(datatype)p Ft(?)h(I.e.,)d(if)i(w)o(e)e(ha)o(v)o(e)i (another)g(p)q(oin)o(ter)h(to)e(the)g(same)g(uncommitted)75 2250 y(ob)r(ject,)i(can)g(w)o(e)g(con)o(tin)o(ue)h(to)f(use)h(it?)158 2296 y(W)m(e)j(prohibit)j(the)d(use)g(of)g(committed)h(datat)o(yp)q(e)h(in)e (the)h(construction)h(of)e(new)g(datat)o(yp)q(es)h(in)g(order)g(to)f(allo)o (w)75 2341 y(the)c(commit)g(function)g(to)g(c)o(hange)g(a)g(datat)o(yp)q(e)g (represen)o(tation)i(in)o(to)e(one)g(suitable)h(for)e(comm)o(unication,)j (and)e(p)q(erhaps)75 2387 y(unsuitable)i(for)e(recursiv)o(e)h(datat)o(yp)q(e) g(construction.)k(This)c(restriction)g(could)g(b)q(e)f(relaxed)h(b)o(y)f (assuming)i(that)e(the)f(\\un-)75 2433 y(committed")i(datat)o(yp)q(e)h (represen)o(tation)g(is)f(k)o(ept)g(alongside)i(the)d(\\committed")i(one.)j (With)c(suc)o(h)g(an)g(implemen)o(tation)75 2478 y Fm(MPI)p 137 2478 12 2 v 13 w(TYPE)p 230 2478 V 13 w(COMMIT)h Ft(could)j(ha)o(v)o(e)g (only)h(one)f(argumen)o(t,)h(and)f(only)h(one)f(t)o(yp)q(e)g(of)f(datat)o(yp) q(e)i(ob)r(ject)e(is)i(needed.)31 b(Suc)o(h)75 2524 y(represen)o(tation)14 b(could)f(also)g(simplify)h(the)e(implemen)o(tation)j(of)d(a)g(datat)o(yp)q (e)h(\\\015atten")g(function)g(\(e.g.,)e(for)h(debugging\).)75 2704 y Fu(MPI)p 144 2704 14 2 v 15 w(TYPE)p 247 2704 V 15 w(FREE\(datatype\)) p eop %%Page: 31 31 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(31)117 45 y Ft(inout)15 b Fu(datatype)553 b Ft(handle)14 b(to)f(datat)o(yp)q(e)h(ob)r(ject)158 121 y Fx(Marks)g(the)g (datat)o(yp)q(e)f(ob)r(ject)h(asso)q(ciated)g(with)f Fu(datatype)f Fx(for)h(deallo)q(cation.)k(The)c(ob)r(ject)i(will)c(b)q(e)j(deal-)75 171 y(lo)q(cated)f(after)h(an)o(y)f(p)q(ending)g(comm)o(unication)d(that)j (uses)i(this)e(ob)r(ject)h(completes,)f(at)g(whic)o(h)g(p)q(oin)o(t)g Fu(datatype)75 221 y Fx(b)q(ecomes)h(n)o(ull.)158 349 y Fo(Discussion:)158 395 y Ft(If)d(w)o(e)g(ha)o(v)o(e)i(di\013eren)o(t)g(datat)o(yp)q(es)g(for)e (committed)i(and)f(uncommitted,)i(then)e(w)o(e)f(ha)o(v)o(e)h(2)g(free)f (functions)j(\(or)d(nev)o(er)75 441 y(free)i(uncommitted)h(datat)o(yp)q(e\)) 75 637 y Fq(1.13.3)49 b(Use)16 b(of)h(general)f(datat)o(yp)q(es)g(in)h(comm)n (unication)75 714 y Fx(Handles)c(to)g(deriv)o(ed)g(datat)o(yp)q(es)g(can)g(b) q(e)g(passed)h(to)f(a)f(comm)o(unication)d(call)j(wherev)o(er)j(a)d(datat)o (yp)q(e)h(argumen)o(t)75 764 y(is)i(required.)23 b(A)16 b(call)e(of)h(the)h (form)d Fu(MPI)p 715 764 14 2 v 15 w(SEND\(start,)20 b(count,)g(datatype,)g (...\))p Fx(,)14 b(where)j Fu(count)c Fx(=)h Fu(1)h Fx(uses)75 814 y(the)e(send)g(bu\013er)h(de\014ned)g(b)o(y)e Fu(start)f Fx(and)i Fu(datatype)p Fx(;)d(i.e.)17 b(if)12 b Fu(datatype)f Fx(is)h(a)g(handle)h(to)f(the)h(datat)o(yp)q(e)g(de\014ned)75 864 y(b)o(y)h(the)i(sequence)h Fn(f)p Fx(\()p Fj(ty)q(pe)492 870 y Fi(0)511 864 y Fj(;)7 b(disp)606 870 y Fi(0)625 864 y Fx(\))p Fj(;)g(:::;)g Fx(\()p Fj(ty)q(pe)807 870 y Fh(n)p Fk(\000)p Fi(1)870 864 y Fj(;)g(disp)965 870 y Fh(n)p Fk(\000)p Fi(1)1030 864 y Fx(\))p Fn(g)p Fx(,)15 b(\(with)f(empt)o(y)g(en)o(tries)h(deleted\))h (then)g(the)f(send)75 913 y(bu\013er)f(consists)f(of)f(the)i(en)o(tries)f(of) f(t)o(yp)q(es)i Fj(ty)q(pe)823 919 y Fh(i)838 913 y Fx(,)e(at)g(addresses)j Fu(start)6 b Fx(+)h Fj(disp)1324 919 y Fh(i)1338 913 y Fx(,)12 b Fj(i)g Fx(=)g(0)p Fj(;)7 b(:::;)g(n)g Fn(\000)g Fx(1.)14 b(The)f(message)75 963 y(generated)h(consists)f(of)f(a)g(sequence)i(of)e Fj(n)g Fx(en)o(tries,)h(of)f(t)o(yp)q(es)h Fj(ty)q(pe)1119 969 y Fi(0)1139 963 y Fj(;)7 b(:::;)g(ty)q(pe)1289 969 y Fh(n)p Fk(\000)p Fi(1)1353 963 y Fx(.)17 b(The)c(receiv)o(e)g(bu\013er)h(for)e(a)g (call)75 1013 y Fu(MPI)p 144 1013 V 15 w(RECV\()21 b(start,)f(count,)h (datatype)f(,)i(...\))p Fx(,)10 b(is)h(similarly)e(de\014ned,)j(for)f Fu(count)21 b(=1)p Fx(.)c(The)12 b(same)e(applies)75 1063 y(to)k(all)e(other) j(send)g(and)e(receiv)o(e)j(op)q(erations)e(that)g(ha)o(v)o(e)f(a)h Fu(datatype)e Fx(parameter.)158 1113 y(A)f(datat)o(yp)q(e)h(used)g(in)e(a)h (receiv)o(e)i(op)q(eration)e(ma)o(y)e(not)i(ha)o(v)o(e)g(o)o(v)o(erlapping)f (en)o(tries;)j(otherwise)f(the)g(outcome)75 1162 y(of)h(the)i(receiv)o(e)g (op)q(eration)f(is)f(unde\014ned.)158 1212 y(If)i(a)g(send)i(op)q(eration)e (using)g(a)g(general)h(datat)o(yp)q(e)f Fu(type1)g Fx(is)g(matc)o(hed)g(b)o (y)g(a)g(receiv)o(e)i(op)q(eration)e(using)g(a)75 1262 y(general)d(datat)o (yp)q(e)f Fu(type2)f Fx(then)i(the)g(sequence)i(of)c(\(nonempt)o(y\))h(basic) g(t)o(yp)q(e)h(en)o(tries)g(de\014ned)h(b)o(y)e(eac)o(h)h(datat)o(yp)q(e)75 1312 y(should)i(b)q(e)h(iden)o(tical.)j(F)m(urthermore,)13 b(eac)o(h)i(en)o(try)f(in)g(the)h(send)g(bu\013er)g(should)f(b)q(e)g(of)g(t)o (yp)q(e)g(that)h(matc)o(hes)e(the)75 1362 y(t)o(yp)q(e)h(of)g(the)g(corresp)q (onding)h(en)o(try)g(in)e Fu(type1)g Fx(and)h(eac)o(h)g(en)o(try)h(in)e(the)i (receiv)o(e)g(bu\013er)g(should)f(of)f(a)g(t)o(yp)q(e)i(that)75 1411 y(matc)o(hes)c(the)g(t)o(yp)q(e)h(of)f(the)g(corresp)q(onding)h(en)o (try)g(in)f Fu(type2)f Fx(\(with)h(t)o(yp)q(e)g(matc)o(hing)f(de\014ned)i(as) f(in)g(Section)g(1.5\).)75 1461 y(Th)o(us,)17 b(t)o(yp)q(e)f(matc)o(hing)f (is)h(de\014ned)h(b)o(y)f(the)h(\\\015at")e(structure)k(of)c(a)h(datat)o(yp)q (e,)h(and)f(do)q(es)g(not)h(dep)q(end)g(on)f(its)75 1511 y(de\014ning)e (expression.)158 1561 y(Example:)75 1634 y Fu(...)75 1684 y (MPI_TYPE_CONTIGUO)o(US\()19 b(MPI_REAL,)h(2,)h(type2\))75 1734 y(MPI_TYPE_CONTIGUO)o(US\()e(MPI_REAL,)h(4,)h(type4\))75 1784 y(MPI_TYPE_CONTIGUO)o(US\()e(type2,)h(2,)i(type22\))158 1857 y Fx(A)14 b(message)g(sen)o(t)g(with)g(datat)o(yp)q(e)g Fu(type4)f Fx(can)h(b)q(e)g(receiv)o(ed)i(with)d(datat)o(yp)q(e)h Fu(type22)p Fx(.)158 1907 y(T)o(yp)q(e)19 b(matc)o(hing)d(requiremen)o(ts)i (imply)e(that)j(\(nonempt)o(y\))e(en)o(tries)i(of)f(di\013eren)o(t)h(t)o(yp)q (es)g(in)f(a)g(datat)o(yp)q(e)75 1957 y(cannot)d(o)o(v)o(erlap,)e(unless)j (they)f(coincide)g(and)f(ha)o(v)o(e)h(the)g(same)f(basic)g(t)o(yp)q(e,)h(or)g (one)g(of)e(them)h(ha)o(v)o(e)h(basic)f(t)o(yp)q(e)75 2007 y Fu(byte)p Fx(.)158 2056 y(The)19 b(length)f(of)g(a)g(message)h(receiv)o(ed) g(\(i.e.,)g(the)g(v)n(alue)f(returned)i(when)f(the)g(receiv)o(e)h(status)f (ob)r(ject)g(is)75 2106 y(deco)q(ded)f(b)o(y)f(the)g(function)g Fu(MPI)p 608 2106 V 15 w(GET)p 689 2106 V 15 w(LEN)p Fx(\))f(is)h(equal)f(to) h(the)g(n)o(um)o(b)q(er)f(of)h(basic)g(en)o(tries)g(receiv)o(ed.)29 b(Consider)75 2156 y(again)13 b(the)i(datat)o(yp)q(es)h(de\014ned)f(in)f(the) h(last)g(example.)j(If)d(a)f(message)g(is)g(sen)o(t)i(using)e(either)h(datat) o(yp)q(e)g Fu(type22)75 2206 y Fx(or)d Fu(type4)f Fx(and)h(is)f(receiv)o(ed)j (using)d(either)i(of)f(these)h(t)o(w)o(o)e(datat)o(yp)q(es,)i(then)g(the)f (length)g(of)f(the)i(message)f(receiv)o(ed)75 2256 y(will)g(b)q(e)j(four.)j (In)c(the)g(general)g(case)h(the)f(basic)g(en)o(tries)h(need)g(not)f(b)q(e)h (all)d(of)i(the)g(same)f(t)o(yp)q(e.)158 2384 y Fo(Discussion:)158 2430 y Ft(An)h(alternativ)o(e)i(de\014nition)h(that)d(\014nds)h(fa)o(v)o(or)g (with)f(man)o(y)h(mem)o(b)q(ers)f(of)g(the)g(sub)q(committee)i(is)f(that)f (the)g(length)75 2475 y(of)g(the)g(message)h(receiv)o(ed)h(is)f(measured)g (in)g(terms)f(of)g(subunits)i(of)e(the)g(receiving)j(datat)o(yp)q(e.)k(Th)o (us,)15 b(if)f(four)h(reals)g(are)75 2521 y(receiv)o(ed)d(using)h Fm(type4)p Ft(,)c(the)i(length)i(of)e(the)g(receiv)o(ed)h(message)g(is)g (four;)f(if)h(they)f(are)g(receiv)o(ed)i(using)g Fm(type22)p Ft(,)8 b(the)j(length)75 2567 y(of)g(the)g(receiv)o(ed)h(message)g(is)f(t)o (w)o(o.)16 b(With)c(suc)o(h)g(a)f(de\014nition,)i(it)f(is)f(required)i(that)e (the)g(arriving)i(message)f(\014lls)g(an)g(in)o(teger)75 2612 y(n)o(um)o(b)q(er)i(of)f(subunits.)158 2658 y(The)f(rationale)i(is)e(that)g (the)g(these)g(are)g(the)g(units)h(that)f(are)f(meaningful)k(to)c(the)h (programmer.)18 b(Consider)13 b(the)f(case)75 2704 y(of)h(an)h(arra)o(y)f(of) g(structures)h(that)g(is)f(passed)i(in)f(a)f(message)h(\(example)g(4.1)g(b)q (elo)o(w\).)k(Then)13 b(one)h(migh)o(t)g(w)o(an)o(t)f(to)g(enforce)p eop %%Page: 32 32 bop 75 -100 a Fr(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(32)75 45 y Ft(the)15 b(restriction)h(that)f(only)h(complete)g (structures)f(are)g(passed;)h(and)f(is)g(natural)h(to)f(return)g(in)g(the)g (receiv)o(e)h(status)f(the)75 91 y(n)o(um)o(b)q(er)c(of)f(structures)h (receiv)o(ed,)h(rather)f(than)g(this)g(n)o(um)o(b)q(er)g(m)o(ultipli)q(ed)i (b)o(y)e(14,)g(the)f(n)o(um)o(b)q(er)h(of)f(en)o(tries)i(p)q(er)e(structure.) 158 136 y(Consider,)19 b(on)f(the)f(other)g(hand,)i(the)e(case)h(of)f(a)g (sequence)h(of)f(reals)h(that)f(is)h(receiv)o(ed)h(in)e(a)h(3D)f(arra)o(y)h (section)75 182 y(\(de\014ned)d(as)f(in)h(the)f(\014rst)g(example)i(b)q(elo)o (w\).)k(W)m(e)14 b(ma)o(y)g(w)o(an)o(t)g(to)g(receiv)o(e)h(an)f(arbitrary)i (n)o(um)o(b)q(er)e(of)g(reals,)h(and)f(the)g(1D)75 228 y(and)g(2D)f (subsections)i(ma)o(y)e(not)h(carry)f(an)o(y)h(signi\014cance)h(for)e(the)g (purp)q(ose)i(of)d(this)i(comm)o(unication.)158 273 y(Opinions?)158 406 y Fx(A)19 b(call)e(of)h(the)i(form)c Fu(MPI)p 588 406 14 2 v 16 w(SEND\(start,)j(count,)i(datatype)f(,)h(...\))p Fx(,)e(where)g Fu(count)g Fj(>)g Fu(1)p Fx(,)g(is)g(in)o(ter-)75 456 y(preted)i(as)g(if)e (the)i(call)e(w)o(as)h(passed)h(a)f(new)g(datat)o(yp)q(e)g(whic)o(h)g(is)g (the)h(concatenation)f(of)g Fu(count)f Fx(copies)i(of)75 506 y Fu(datatype)p Fx(,)12 b(suc)o(h)j(as)f(w)o(ould)f(b)q(e)h(pro)q(duced)h(b)o (y)f(a)g(call)f(to)h Fu(MPI)p 1067 506 V 15 w(CONTIGUOUS\(count,)k(datatype,) i(newtype\))p Fx(,)75 555 y(and)12 b(a)f Fu(count)g Fx(argumen)o(t)f(of)h (one.)18 b(The)12 b(same)f(holds)h(true)g(for)g(the)g(other)g(comm)o (unication)d(functions)i(that)h(ha)o(v)o(e)75 605 y(a)i(datat)o(yp)q(e)g (argumen)o(t.)75 721 y Fq(1.13.4)49 b(Examples)75 798 y Fx(The)14 b(follo)o(wing)e(examples)h(illustrate)g(the)i(use)f(of)g(deriv)o(ed)g(datat) o(yp)q(es.)158 898 y(First)g(example:)j(extract)e(a)f(section)g(of)f(a)h(3D)f (matrix.)206 981 y Fu(REAL)21 b(a\(100,100,100\),)d(e\(9,9,9\))206 1030 y(INTEGER)i(oneslice,)g(twoslice,)g(threeslice,)f(finaltype,)h (sizeofreal)75 1130 y(C)131 b(extract)20 b(the)h(section)f(a\(1:17:2,)g (3:11,)h(2:10\))75 1180 y(C)131 b(and)21 b(store)f(it)i(in)f(e\(*,*,*\).)206 1280 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 1379 y(C)109 b(create)20 b(datatype)g(for)i(a)f(1D)h(section)206 1429 y(CALL)f(MPI_TYPE_VECTOR\()d(9,)k(MPI_REAL,)41 b(2,)22 b(1,)f(oneslice,)f(ierr\))75 1529 y(C)109 b(create)20 b(datatype)g(for)i(a)f (2D)h(section)206 1578 y(CALL)f(MPI_TYPE_HVECTOR)o(\()e(9,)i(oneslice,)42 b(100*sizeofreal,)19 b(1,)i(twoslice,)f(ierr\))75 1678 y(C)109 b(create)20 b(datatype)g(for)i(the)f(entire)f(section)206 1728 y(CALL)h(MPI_TYPE_HVECTOR)o(\()e(9,)i(twoslice,)42 b(100*100*sizeofreal)o(,) 19 b(1,)467 1778 y(threeslice,)h(ierr\))206 1877 y(CALL)h(MPI_TYPE_COMMIT\()d (threeslice,)i(finaltype,)f(ierr\))206 1927 y(CALL)i(MPI_SENDRECV\(a\(1)o (,3,2\))o(,)e(1,)i(finaltype,)f(MPI_ME,)g(e,)i(9*9*9,)467 1977 y(MPI_REAL,)e(MPI_ME,)h(0,)g(MPI_ALL,)f(status,)g(ierr\))158 2060 y Fx(Second)15 b(example:)h(Cop)o(y)e(the)g(\(strictly\))h(lo)o(w)o(er)e (triangular)g(part)h(of)g(a)f(matrix.)206 2143 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 2193 y(INTEGER)42 b(index\(100\),)19 b(blocklen\(100\),)g(ltype,)i(finaltype)75 2292 y(C)109 b(copy)21 b(lower)f(triangular)g(part)h(of)h(array)e(a)75 2342 y(C)109 b(onto)21 b(lower)f(triangular)g(part)h(of)h(array)e(b)75 2442 y(C)109 b(compute)20 b(start)h(and)g(size)g(of)g(each)g(column)206 2492 y(DO)g(i=1,)g(100)249 2541 y(index\(i\))f(=)i(100*\(i-1\))e(+)i(i)249 2591 y(block\(i\))e(=)i(100-i)206 2641 y(END)f(DO)p eop %%Page: 33 33 bop 75 -100 a Fr(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(33)75 45 y Fu(C)109 b(create)20 b(datatype)g(for)i(lower)e (triangular)g(part)206 95 y(CALL)h(MPI_TYPE_INDEX\()d(MPI_REAL,)i(100,)h (index,)g(block,)f(ltype,)h(ierr\))206 195 y(CALL)g(MPI_TYPE_COMMIT\()o (ltype)o(,)e(finaltype,)h(ierr\))206 244 y(CALL)h(MPI_SENDRECV\()e(a,)i(1,)h (finaltype,)d(MPI_ME,)i(b,)g(1,)424 294 y(finaltype,)e(MPI_ME,)i(0,)g (MPI_ALL,)f(status,)h(ierr\))158 473 y Fx(Third)14 b(example:)i(T)m(ransp)q (ose)f(a)f(matrix)206 552 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 602 y(INTEGER)h(row,)h(xpose,)g(finaltype,)e(sizeofreal)75 702 y(C)109 b(transpose)20 b(matrix)g(a)i(onto)f(b)206 851 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 951 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row)206 1001 y(CALL)g(MPI_TYPE_VECTOR\()d(100,)j(MPI_REAL,)f(100,)h(1,)g(row,)g(ierr\))75 1100 y(C)109 b(create)20 b(datatype)g(for)i(matrix)e(in)i(row-major)d(order) 206 1150 y(CALL)i(MPI_TYPE_HVECTOR)o(\()e(100,)i(row,)g(sizeofreal,)e(1,)j (xpose,)e(ierr\))206 1250 y(CALL)h(MPI_TYPE_COMMIT\()d(xpose,)j(finaltype,)e (ierr\))75 1349 y(C)109 b(send)21 b(matrix)f(in)i(row-major)e(order)g(and)i (receive)e(in)h(column)g(major)g(order)206 1399 y(CALL)g(MPI_SENDRECV\()e(a,) i(1,)h(finaltype,)d(MPI_ME,)i(b,)g(100*100,)424 1449 y(MPI_REAL,)f(MPI_ME,)g (0,)h(MPI_ALL,)f(status,)h(ierr\))158 1628 y Fx(Another)15 b(approac)o(h)f(to)f(the)i(transp)q(ose)g(problem:)206 1707 y Fu(REAL)21 b(a\(100,100\),)e(b\(100,100\))206 1757 y(INTEGER)42 b(index\(2\),)20 b(blen\(2\),)g(type\(2\),)g(row,)h(row1,)g(finaltype,)f (sizeofreal)75 1907 y(C)109 b(transpose)20 b(matrix)g(a)i(onto)f(b)206 2056 y(CALL)g(MPI_TYPE_EXTENT\()d(MPI_REAL,)i(sizeofreal,)g(ierr\))75 2156 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row)206 2205 y(CALL)g(MPI_TYPE_VECTOR\()d(MPI_REAL,)i(100,)h(100,)g(1,)g(row,)g(ierr\))75 2305 y(C)109 b(create)20 b(datatype)g(for)i(one)f(row,)g(with)g(the)g(extent) f(of)i(one)f(real)g(number)206 2355 y(index\(1\))f(=)h(0)206 2405 y(index\(2\))f(=)h(sizeofreal)206 2455 y(type\(1\))42 b(=)21 b(row)206 2504 y(type\(2\))42 b(=)21 b(MPI_UB)206 2554 y(blen\(1\))42 b(=)21 b(1)206 2604 y(blen\(2\))42 b(=)21 b(1)206 2654 y(CALL)g(MPI_TYPE_STRUC\()d(2,)k(type,)f(index,)f(blen,)h(row1\))p eop %%Page: 34 34 bop 75 -100 a Fr(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(34)206 45 y Fu(CALL)21 b(MPI_TYPE_COMMIT\()d(row1,)j (finaltype,)f(ierr\))75 145 y(C)109 b(send)21 b(100)g(rows)g(and)g(receive)f (in)i(column)e(major)h(order)206 195 y(CALL)g(MPI_SENDRECV\()e(a,)i(100,)g (finaltype,)f(MPI_ME,)g(b,)i(100*100,)424 244 y(MPI_REAL,)e(MPI_ME,)g(0,)h (MPI_ALL,)f(status,)h(ierr\))158 427 y Fx(F)m(ourth)14 b(example:)j (manipulate)11 b(an)j(arra)o(y)g(of)f(structures.)75 510 y Fu(struct)21 b(Partstruct)140 560 y({)140 610 y(int)h(index;)85 b(/*)22 b(particle)e(type)h(*/)140 659 y(double)g(d[6])43 b(/*)21 b(particle)f(coordinates)g(*/)140 709 y(char)h(b[7])87 b(/*)21 b(some)g(additional)f(information)f(*/)140 759 y(};)75 859 y(struct)i(Partstruct)e(particle[1000];)315 958 y(/*)i(build)g(datatype)f (describing)g(first)g(array)h(entry)g(*/)75 1058 y(MPI_datatype)e (Particletype;)75 1108 y(MPI_datatype)g(stype[3])h(=)i({MPI_int,)e (MPI_double,)f(MPI_char};)75 1158 y(int)217 b(sblock[3])20 b(=)i({1,)f(6,)g(7};)75 1207 y(int)217 b(sindex[3];)75 1307 y(MPI_address\()19 b(\(void)i(*\)particle,)f(sindex\);)75 1357 y(MPI_address\()f(\(void)i(*\)particle[0].d,)e(sindex+1\);)75 1407 y(MPI_address\()g(\(void)i(*\)particle[0].b,)e(sindex+2\);)75 1456 y(MPI_type_struct\()f(3,)k(stype,)e(sindex,)h(sblock,)f(Particletype\);) 75 1506 y(/*)h(Particletype)f(describes)g(first)g(array)h(entry)g(--)g(using) g(absolute)140 1556 y(addresses)f(*/)249 1706 y(/*)i(4.1:)f(send)g(the)g (entire)f(array)h(*/)75 1805 y(MPI_datatype)e(Type1;)75 1855 y(MPI_commit\()h(Particletype,)f(Type1\);)75 1905 y(MPI_send\()h(MPI_bottom,) f(1000,)i(Type1,)g(dest,)g(tag,)f(comm\);)249 2054 y(/*)i(4.2:)f(send)g(the)g (entries)f(with)h(index)g(value)g(zero,)293 2104 y(preceded)f(by)h(the)h (number)e(of)i(such)f(entries)f(*/)75 2204 y(MPI_datatype)f(Zparticles;)63 b(/*)22 b(datatype)e(describing)g(all)h(particles)860 2253 y(with)g(index)f(zero)h(\(needs)g(to)g(be)h(recomputed)860 2303 y(if)f(indices)f(change\))h(*/)75 2353 y(MPI_datatype)e(Ztype,)i(Type2;) 75 2453 y(int)g(zindex[1000],)e(zblock[1000],)g(i,)j(j,)f(k;)75 2503 y(int)g(zzblock[2])f(=)h({1,1};)75 2552 y(int)g(zztype[2],)f (zzindex[2];)75 2702 y(/*)h(compute)g(indices)f(of)h(particles)f(with)h (index)g(zero)g(*/)p eop %%Page: 35 35 bop 75 -100 a Fr(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(35)75 45 y Fu(j)22 b(=)f(0;)75 95 y(for\(i=0;)f(i)i(<)f(1000;) g(i++\))119 145 y(if)g(\(particle[i].inde)o(x==0\))d(then)184 195 y({)184 244 y(zindex[j])i(=)h(i;)184 294 y(zblock[i])f(=)h(1;)184 344 y(j++;)184 394 y(})75 493 y(/*)g(create)g(datatype)f(for)h(particles)f (with)h(index)g(zero)g(*/)75 543 y(MPI_type_index\()e(j,)i(Particletype,)41 b(zindex,)20 b(zblock,)h(Zparticles\);)75 643 y(/*)g(prepend)g(particle)f (count)h(*/)75 693 y(MPI_address\(\(void)d(*\)&j,)j(zzindex\);)75 742 y(zzindex[1])f(=)h(MPI_bottom;)75 792 y(zztype[0])f(=)i(MPI_int;)75 842 y(zztype[1])e(=)i(Zparticles;)75 892 y(MPI_type_struc\(2,)c(zztype,)j (zzindex,)f(zzblock,)g(Ztype\);)75 992 y(MPI_type_commit\()e(Ztype,)j (Type2\);)75 1041 y(MPI_send\()f(MPI_bottom,)f(1,)j(Type2,)e(dest,)h(tag,)g (comm\);)249 1191 y(/*)h(A)f(possibly)f(more)h(efficient)f(way)i(of)f (defining)f(Zparticles)g(*/)75 1290 y(/*)h(consecutive)f(particles)g(with)h (index)g(zero)g(are)g(handled)f(as)h(one)h(block)e(*/)75 1340 y(j=0;)75 1390 y(for)h(\(i=0;)g(i)h(<)f(1000;)g(i++\))119 1440 y(if)g(\(particle[i].inde)o(x==0\))d(then)162 1490 y({)162 1539 y(for)j(\(k=i+1;)g(\(k)g(<)h(1000\)&&\(particle)o([k].i)o(ndex)c(==)k (0\))f(;)h(k++\);)162 1589 y(zindex[j])e(=)i(i;)162 1639 y(zblock[j])e(=)i (k-i;)162 1689 y(j++;)162 1739 y(i)g(=)f(k;)162 1789 y(})75 1838 y(MPI_type_index\()e(j,)i(Particletype,)e(zindex,)i(zblock,)f (Zparticles\);)249 1988 y(/*)i(4.3:)f(send)g(the)g(first)g(two)g(coordinates) e(of)j(all)f(entries)f(*/)75 2087 y(MPI_datatype)f(Allpairs;)129 b(/*)22 b(datatype)e(for)h(all)g(pairs)g(of)g(coordinates)f(*/)75 2137 y(MPI_datatype)f(Type3;)75 2237 y(int)217 b(sizeofentry;)75 2337 y(MPI_extent\()20 b(Particletype,)f(sizeofentry\);)184 2436 y(/*)i(sizeofentry)f(can)h(also)g(be)g(computed)g(by)g(subtracting)e (the)j(address)249 2486 y(of)g(particle[0])d(from)i(the)g(address)g(of)g (particle[1])f(*/)75 2586 y(MPI_type_hvector\()e(1000,)j(MPI_real,)f (sizeofentry,)f(2,)j(Allpairs\);)75 2635 y(MPI_type_commit\()c(Allpairs,)i (Type3\);)75 2685 y(MPI_send\()g(particle.d,)f(1,)j(Type3,)e(dest,)h(tag,)g (comm\);)p eop %%Page: 36 36 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(36)206 95 y Fu(/*)21 b(an)h(alternative)d(solution)h(to)i(4.3) f(*/)75 195 y(MPI_datatype)e(Onepair;)64 b(/*)21 b(datatype)g(for)g(one)g (pair)g(of)g(coordinates,)f(with)642 244 y(the)h(extent)f(of)i(one)f (particle)f(entry)h(*/)75 294 y(MPI_datatype)e(Type4;)75 394 y(int)i(indexp[3];)75 444 y(int)g(typep[3])f(=)i({MPI_lb,)e(MPI_double,)g (MPI_ub};)75 493 y(int)h(blockp[3])f(=)i({1,)f(2,)g(1};)75 593 y(MPI_address\()e(\(void)i(*\)particle,)f(indexp\);)75 643 y(MPI_address\()f(\(void)i(*\)particle[0].d,)e(indexp+1\);)75 693 y(MPI_address\()g(\(void)i(*\)\(particle+1\),)e(indexp+2\);)75 792 y(MPI_type_struc\()g(3,)i(typep,)g(indexp,)f(blockp,)g(Onepair\);)75 842 y(MPI_commit\()g(Onepair,)g(Type4\);)75 892 y(MPI_send\()g(MPI_bottom,)f (1000,)i(Type4,)g(dest,)g(tag,)f(comm\);)158 1099 y Fo(Discussion:)158 1145 y Ft(The)13 b(examples)i(carry)e(t)o(w)o(o)f(implicit)k(assumptions)f (on)f(the)f(w)o(a)o(y)g(padding)i(is)f(done)f(in)h(an)f(arra)o(y)h(of)f (structures.)158 1191 y(First,)h(that)f(padding)j(is)e(done)g(in)h(the)e (same)h(w)o(a)o(y)f(in)h(the)g(successiv)o(e)h(arra)o(y)f(en)o(tries,)g(so)g (that)f(all)i(en)o(tries)f(ha)o(v)o(e)g(an)75 1236 y(iden)o(tical)i(memory)e (la)o(y)o(out.)20 b(E.g.,)13 b(w)o(e)g(assume)h(that)g(displacemen)o(t)i (from)d Fm(particle[i)o(].b)d Ft(to)j Fm(particle[j)o(].b)c Ft(is)15 b(the)75 1282 y(same)e(as)g(the)h(displacemen)o(t)h(from)e Fm(particle[)o(i].)o(d)d Ft(to)j Fm(particle)o([j])o(.d)p Ft(.)158 1328 y(Second,)g(that)e(the)h(amoun)o(t)g(of)f(padding)j(added)e(is,)g(in)h (some)e(sense,)h(minimal.)19 b(I.e.,)10 b(if)i(the)g(most)f(stringen)o(t)i (aligne-)75 1373 y(men)o(t)h(requiremen)o(t)h(for)e(a)h(basic)h(comp)q(onen)o (t)g(of)e(a)h(structure)g(is)h(alignmen)o(t)h(to)d(m)o(ultiples)j(of)e(2)1511 1357 y Fe(k)1530 1373 y Ft(,)f(then)i(no)f(con)o(tiguous)75 1419 y(padding)h(space)f(has)f(size)h(greater)f(or)g(equal)i(to)e(2)811 1403 y Fe(k)830 1419 y Ft(.)k(This)c(assumption)i(is)f(implicit)i(in)e(the)f (de\014nition)i(of)e Fo(exten)o(t)p Ft(.)158 1465 y(I)i(don't)g(think)h (either)g(assumptions)h(are)f(mandated)g(b)o(y)f(C)g(of)f(F)m(ortran)i (standards,)g(although)h(I)e(w)o(ould)h(b)q(e)f(v)o(ery)75 1510 y(surprised)d(to)d(see)h(an)g(implemen)o(tation)j(that)d(do)q(es)h(not)f (ful\014l)h(these)f(conditions.)19 b(If)9 b(I)h(am)f(wrong)h(\(e.g.,)g(if)g (some)g(compilers)75 1556 y(alw)o(a)o(ys)j(align)h(structures)e(to)g(double)i (w)o(ord)e(b)q(oundaries,)i(ev)o(en)f(if)f(they)g(ha)o(v)o(e)h(no)f(double)i (w)o(ord)e(comp)q(onen)o(ts\),)h(then)g(the)75 1602 y(de\014nition)j(of)c Fm(extent)f Ft(should)k(b)q(e)e(revisited.)75 1800 y Fq(1.13.5)49 b(Co)o(rrect)15 b(use)h(of)g(addresses)75 1876 y Fx(Successiv)o(ely)i (declared)f(v)n(ariables)f(in)g(C)g(or)h(F)m(ortran)f(are)h(not)f (necessarily)i(stored)f(at)g(con)o(tiguous)f(lo)q(cations.)75 1926 y(Th)o(us,)g(care)g(m)o(ust)f(b)q(e)h(exercised)i(that)e(displacemen)o (ts)f(do)g(not)h(cross)h(from)d(one)i(v)n(ariable)e(to)i(another.)24 b(Also,)75 1976 y(in)18 b(mac)o(hines)f(with)g(segmen)o(ted)h(address)i (space,)f(addresses)h(are)f(not)f(unique)g(and)g(address)h(arithmetic)e(has) 75 2026 y(some)e(p)q(eculiar)g(prop)q(erties.)25 b(Th)o(us,)16 b(use)g(of)f Fv(addresses)p Fx(,)f(i.e.)22 b(displacemen)o(ts)15 b(relativ)o(e)h(to)f(the)h(start)h(address)75 2075 y Fu(MPI)p 144 2075 14 2 v 15 w(BOTTOM)p Fx(,)12 b(has)i(to)g(b)q(e)g(restricted.)158 2125 y(V)m(ariables)d(b)q(elong)h(to)g(the)h(same)e Fv(sequen)o(tial)g (storage)g Fx(if)g(they)h(b)q(elong)g(to)g(the)h(same)e(arra)o(y)m(,)g(to)h (the)h(same)75 2175 y Fu(COMMON)h Fx(blo)q(c)o(k)i(in)f(F)m(ortran,)g(or)h (to)f(the)i(same)d(structure)k(in)d(C.)h(V)m(alid)e(addresses)j(are)g (de\014ned)f(recursiv)o(ely)h(as)75 2225 y(follo)o(ws:)126 2311 y(1.)j(The)15 b(function)f Fu(MPI)p 497 2311 V 15 w(ADDRESS)f Fx(returns)j(a)e(v)n(alid)f(address,)j(when)f(passed)g(as)g(argumen)o(t)f(a)g (host)h(program)179 2361 y(v)n(ariable.)126 2442 y(2.)20 b(The)14 b Fu(start)g Fx(parameter)g(of)f(a)h(comm)o(unication)d(function)j(ev)n (aluates)g(to)g(a)g(v)n(alid)f(address,)i(when)g(passed)179 2492 y(as)f(argumen)o(t)f(a)g(host)h(program)f(v)n(ariable.)126 2573 y(3.)20 b(If)13 b Fu(v)g Fx(is)g(a)g(v)n(alid)e(address,)j(and)f Fu(i)g Fx(is)g(an)g(in)o(teger,)h(then)g Fu(v+i)e Fx(is)h(a)g(v)n(alid)f (address,)i(pro)o(vided)f Fu(v)g Fx(and)g Fu(v+i)g Fx(are)179 2623 y(in)g(the)i(same)e(sequen)o(tial)h(storage.)126 2704 y(4.)20 b(If)13 b Fu(v)h Fx(is)g(a)f(v)n(alid)g(address)i(then)f Fu(MPI)p 744 2704 V 15 w(BOTTOM)21 b(+)h(v)13 b Fx(is)h(a)f(v)n(alid)g (address.)p eop %%Page: 37 37 bop 75 -100 a Fr(SECTION)15 b(1.)26 b(POINT)15 b(TO)f(POINT)h(COMMUNICA)m (TION)736 b Fx(37)158 45 y(A)11 b(correct)h(program)d(uses)i(only)f(v)n(alid) f(addresses)k(to)d(iden)o(tify)g(the)h(lo)q(cations)f(of)g(en)o(tries)h(in)f (comm)o(unication)75 95 y(bu\013ers.)20 b(F)m(urthermore,)13 b(if)g Fu(u)h Fx(and)f Fu(v)h Fx(are)g(t)o(w)o(o)g(v)n(alid)e(addresses,)k (then)e(the)h(\(in)o(teger\))g(di\013erence)g Fu(u)22 b(-)g(v)13 b Fx(can)h(b)q(e)75 145 y(computed)h(only)g(if)g(b)q(oth)h Fu(u)g Fx(and)g Fu(v)f Fx(are)h(in)g(the)g(same)f(sequen)o(tial)h(storage;)h (no)f(other)g(arithmetic)f(op)q(erations)75 195 y(can)f(b)q(e)h(meaningfully) 10 b(executed)16 b(on)e(addresses.)158 244 y(W)m(e)c(sa)o(y)h(that)f(a)h (datat)o(yp)q(e)f(is)h Fv(absolute)d Fx(if)i(all)f(displacemen)o(ts)h(within) g(this)h(datat)o(yp)q(e)g(are)g(v)n(alid)e(\(absolute\))75 294 y(addresses;)16 b(it)d(is)h Fv(relativ)o(e)d Fx(otherwise.)158 344 y(A)j(correct)i(program)c(ob)q(eys)i(the)h(follo)o(wing)c(constrain)o(ts) k(in)e(the)i(use)f(of)g(datat)o(yp)q(es:)137 435 y Fn(\017)21 b Fx(If)9 b(the)h Fu(oldtype)d Fx(argumen)o(t)h(used)j(in)d Fu(MPI)p 828 435 14 2 v 16 w(TYPE)p 932 435 V 14 w(CONTIGUOUS,)20 b(MPI)p 1274 435 V 15 w(TYPE)p 1377 435 V 15 w(VECTOR,)g(MPI)p 1632 435 V 15 w(TYPE)p 1735 435 V 15 w(HVECTOR,)179 485 y(MPI)p 248 485 V 15 w(TYPE)p 351 485 V 15 w(INDEX)12 b Fx(or)i Fu(MPI)p 605 485 V 15 w(TYPE)p 708 485 V 15 w(HINDEX)e Fx(is)h(absolute,)g(then)h(all) f(addresses)i(within)e(it)g(m)o(ust)g(b)q(e)h(to)f(v)n(ari-)179 535 y(ables)i(con)o(tained)h(within)e(the)i(same)f(arra)o(y)g(or)g (structure;)j(the)e(result)g(datat)o(yp)q(e)g(is)f(also)g(absolute,)g(and)179 585 y(all)d(computed)i(addresses)i(m)o(ust)d(also)g(fall)f(within)h(the)i (same)e(arra)o(y)h(or)f(structure.)137 668 y Fn(\017)21 b Fx(If)12 b(all)e(en)o(tries)k(of)d(the)i Fu(array)p 634 668 V 14 w(of)p 692 668 V 16 w(types)e Fx(argumen)o(t)g(of)g Fu(MPI)p 1124 668 V 15 w(TYPE)p 1227 668 V 15 w(STRUC)g Fx(are)i(absolute)f(addresses)i (\(com-)179 718 y(puted)j(b)o(y)f Fu(MPI)p 428 718 V 15 w(ADDR)p Fx(\),)f(then)i(the)g(result)g(datat)o(yp)q(e)f(is)h(also)e(absolute.)26 b(Eac)o(h)16 b(new)h(address)g(computed)179 767 y(from)12 b(an)i(old)g (address)h(m)o(ust)f(fall)e(within)i(the)h(same)e(arra)o(y)h(or)h(record)g (as)f(the)h(old)f(address.)20 b(\(Addresses)179 817 y(in)13 b(the)i(old)e(t)o(yp)q(e)h(ma)o(y)e(not)i(b)q(e)h(all)d(within)h(the)i(same)e (arra)o(y)g(or)h(record.\))137 900 y Fn(\017)21 b Fx(If)12 b(a)g(comm)o(unication)e(call)h(is)i(in)o(v)o(ok)o(ed)f(with)g(parameters)h Fu(start)e Fx(and)i Fu(datatype)p Fx(,)e(then)i(either)g Fu(start)21 b(=)179 950 y(MPI)p 248 950 V 15 w(BOTTOM)12 b Fx(and)i Fu(datatype)e Fx(is)h(a)h(handle)f(to)h(an)f(absolute)h(datat)o(yp)q(e,)f(or)h Fu(start)f Fx(is)g(set)i(to)e(the)i(address)179 1000 y(of)e(a)h(program)e(v)n (ariable,)g(and)i(all)f(displacemen)o(ts)g(in)h Fu(datatype)e Fx(relativ)o(e)h(to)h(this)g(base)h(yield)e(addresses)179 1050 y(that)h(are)g(within)f(the)i(same)e(arra)o(y)g(or)h(record)h(as)f(this)g(v)n (ariable.)158 1141 y(In)9 b(summary)m(,)e(the)j(t)o(yp)q(e)g(constructors)h Fu(MPI)p 847 1141 V 15 w(TYPE)p 950 1141 V 15 w(CONTIGUOUS,)19 b(MPI)p 1292 1141 V 16 w(TYPE)p 1396 1141 V 14 w(VECTOR,)i(MPI)p 1651 1141 V 15 w(TYPE)p 1754 1141 V 15 w(HVECTOR,)75 1191 y(MPI)p 144 1191 V 15 w(TYPE)p 247 1191 V 15 w(INDEX)12 b Fx(and)i Fu(MPI)p 531 1191 V 15 w(TYPE)p 634 1191 V 15 w(HINDEX)e Fx(can)i(b)q(e)g (recursiv)o(ely)g(applied)f(to)h(build)f(datat)o(yp)q(es)h(that)f(will)f (com-)75 1241 y(bine)i(v)n(ariables)f(that)h(b)q(elong)g(to)g(the)h(same)e (sequen)o(tial)h(storage;)g(v)n(ariables)f(that)h(do)g(not)g(b)q(elong)f(to)h (the)h(same)75 1290 y(sequen)o(tial)f(storage)g(can)g(b)q(e)h(com)o(bined)e (together)h(using)g(one)g(application)f(of)g Fu(MPI)p 1410 1290 V 15 w(TYPE)p 1513 1290 V 15 w(STRUC)p Fx(.)158 1419 y Fo(Discussion:)158 1464 y Ft(It)e(is)i(not)f(exp)q(ected)h(that)f(MPI)g (implemen)o(tations)j(will)e(b)q(e)g(able)g(to)e(detect)h(erroneous,)h(\\out) g(of)e(b)q(ound")j(displace-)75 1510 y(men)o(ts)h({)g(unless)i(those)f(o)o(v) o(er\015o)o(w)f(the)h(user)f(address)h(space)g({)f(since)h(the)g(MPI)f(call)h (ma)o(y)g(not)f(no)o(w)g(the)g(exten)o(t)h(of)f(the)75 1556 y(arra)o(ys)f(and)f(records)h(in)g(the)f(host)g(program.)75 1776 y Fw(1.14)59 b(Universal)20 b(comm)m(unica)o(tion)d(functions)75 1867 y Fx(Although)11 b(the)i(n)o(um)o(b)q(er)e(of)g(comm)o(unicati)o(on)e (functions)j(de\014ned)h(in)e(this)h(c)o(hapter)g(is)g(fairly)e(large,)i(it)f (is)h(p)q(ossible)75 1916 y(to)k(construct)j(all)c(of)h(them)g(from)f(a)h(m)o (uc)o(h)g(smaller)f(n)o(um)o(b)q(er)h(of)g(primitiv)o(e)e(functions.)26 b(The)17 b(construction)h(is)75 1966 y(outlined)c(in)g(this)h(section.)20 b(The)15 b(construction)h(can)f(b)q(e)g(used)g(to)g(build)e(a)i(p)q(ortable)f (implemen)o(tation)d(of)j(MPI,)75 2016 y(although)j(it)g(is)g(exp)q(ected)i (that)f(more)e(e\016cien)o(t)i(implemen)o(tation)c(will)i(implemen)o(t)f (directly)j(most)e(or)h(all)g(of)75 2066 y(the)e(functions)f(in)g(this)h (section.)20 b(Also,)14 b(this)g(construction)i(can)e(b)q(e)h(used)g(in)f (order)h(to)g(deriv)o(e)f(the)h(b)q(eha)o(vior)g(of)75 2116 y(MPI)i(p)q(oin)o(t)f(to)h(p)q(oin)o(t)f(comm)o(unicatio)o(n)e(from)h(the)i (b)q(eha)o(vior)g(of)f(a)g(small)e(subset)19 b(of)d(functions.)26 b(W)m(e)16 b(assume)75 2166 y(the)e(a)o(v)n(ailabili)o(t)o(y)c(of)j(the)h (primitiv)o(e)d(functions)i(listed)g(at)g(the)h(end)g(of)f(the)g(section,)h (and)f(the)h(abilit)o(y)d(to)i(allo)q(cate)75 2215 y(memory)e(for)j(new)g(v)n (ariables.)158 2265 y(W)m(e)d(assume)f(that)h(the)h(function)f Fu(MPI)p 752 2265 V 15 w(TYPE)p 855 2265 V 15 w(COMMIT)f Fx(is)g(not)h(c)o (hanging)g(the)g(represen)o(tation)i(of)d(a)h(datat)o(yp)q(e,)75 2315 y(but)j(merely)f(creates)j(a)d(new)i(cop)o(y)f(of)f(it.)158 2365 y(Comm)o(uni)o(cation)i(with)j(a)f Fu(count)g Fx(argumen)o(t)g(that)h (is)g Fj(>)h Fx(1)e(can)i(b)q(e)f(reduced)i(to)e(comm)o(unicatio)o(n)d(with) 75 2415 y Fu(count)21 b(=1)p Fx(,)d(b)o(y)g(replicating)g(the)h Fu(datatype)e Fx(argumen)o(t)g Fu(count)g Fx(times.)31 b(Comm)o(uni)o(cation) 15 b(with)j(a)g Fu(a)k(start)75 2464 y Fx(argumen)o(t)14 b(that)i(is)f(not)g Fu(MPI)p 541 2464 V 15 w(BOTTOM)f Fx(can)i(b)q(e)g(reduced)h(to)e(comm)o (unicatio)o(n)e(with)i Fu(start)20 b(=)i(MPI)p 1673 2464 V 15 w(BOTTOM)14 b Fx(b)o(y)75 2514 y(creating)g(a)g(suitable)g(displaced)g (new)g(datat)o(yp)q(e.)k(Th)o(us:)158 2564 y Fu(MPI)p 227 2564 V 15 w(SEND\()j(start,)f(count,)h(datatype,)f(dest,)h(tag,)g(comm\))12 b Fx(is)75 2647 y Fu(Type[1])20 b(=)i(datatype)75 2697 y(MPI_ADDRESS\()d (start,)i(Index[1]\);)p eop %%Page: 38 38 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(38)75 45 y Fu(Block[1])20 b(=)i(count;)75 95 y(MPI_TYPE_STRUCT\()c(1,)k(Type,)f(Index,)f(Block,)h(Newtype\))75 145 y(MPI_TYPE_COMMIT\()d(Newtype,)j(datatype1\))75 195 y(MPI_SEND\()f (MPI_BOTTOM,)f(1,)j(datatype1,)e(dest,)g(tag,)h(comm\))158 264 y Fx(The)13 b(same)e(construction)i(applies)f(to)g(all)f(other)i(comm)o (unicatio)o(n)d(functions)i(with)g(argumen)o(ts)f Fu(count)g Fx(and)75 314 y Fu(datatype)p Fx(.)19 b(W)m(e)14 b(shall)g(henceforth)i (restrict)g(ourselv)o(es)g(to)e(comm)o(unication)d(that)k(in)o(v)o(olv)o(es)f (only)g(one)g(elemen)o(t)75 364 y(with)g(an)f(absolute)h(address)h(\()p Fu(count)21 b(=1)p Fx(,)13 b(and)h Fu(start)21 b(=)g(MPI)p 1079 364 14 2 v 15 w(BOTTOM)p Fx(\).)158 413 y(\(If)15 b(the)h(op)q(eration)f Fu(MPI)p 546 413 V 15 w(TYPE)p 649 413 V 15 w(COMMIT\))f Fx(is)h(not)h (assumed)f(to)g(b)q(e)h(trivial,)e(then)i(the)g(ab)q(o)o(v)o(e)f (construction)75 463 y(cannot)h(b)q(e)h(used;)h(this)e(adds)h(one)f(more)f (primitiv)o(e)f(function,)i(namely)f Fu(MPI)p 1322 463 V 15 w(TYPE)p 1425 463 V 15 w(COMMIT)p Fx(,)f(to)i(our)g(list,)g(and)75 513 y(in)o(tro)q(duces)f(small)d(c)o(hanges)i(in)g(the)g(construction.\))158 563 y(It)j(is)h(con)o(v)o(enien)o(t)f(to)g(to)h(mak)o(e)d(explicit)i(the)h Fv(p)q(ersistence)d Fx(attribute)j(of)e(comm)o(unication)e(ob)r(jects.)30 b(A)75 613 y Fu(persistent)16 b Fx(comm)o(unicatio)o(n)f(ob)r(ject)k(need)g (b)q(e)g(explicitly)e(deallo)q(cated)h(b)o(y)g(a)f Fu(MPI)p 1471 613 V 16 w(COMM)p 1575 613 V 14 w(FREE)h Fx(op)q(eration.)75 662 y(On)d(the)h(other)f(hand,)g(an)f Fu(ephemeral)f Fx(comm)o(unication)f (ob)r(ject)k(is)e(go)q(o)q(d)h(for)f(a)h(single)g(comm)o(uni)o(cation.)j(It)d (is)75 712 y(deallo)q(cated)f(b)o(y)g(the)g(system)g(when)g(the)h(\014rst)f (comm)o(unication)d(it)i(is)h(used)h(for)e(completes.)158 762 y(The)h(function)g Fu(MPI)p 475 762 V 15 w(COMM)p 578 762 V 15 w(INIT)f Fx(is)g(a)h(univ)o(ersal)f(function)h(for)f(the)i(creation)f(of)f (comm)o(unicatio)o(n)e(ob)r(jects:)75 859 y Fu(MPI)p 144 859 V 15 w(COMM)p 247 859 V 15 w(INIT\(handle,)19 b(datatype,)h(source-dest,)f (tag,)i(comm,)g(op-mode,)f(persistence\))117 909 y Ft(out)14 b Fu(handle)629 b Ft(handle)14 b(to)f(newly)h(created)g(comm)o(unication)h (ob)r(ject)117 951 y(in)f Fu(datatype)609 b Ft(datat)o(yp)q(e)14 b(of)f(elemen)o(t)g(sen)o(t)h(or)f(receiv)o(ed)117 993 y(in)h Fu(source-dest)543 b Ft(rank)13 b(of)g(destination)j(or)d(source)g(\(in)o (teger\))117 1036 y(in)h Fu(tag)719 b Ft(message)13 b(tag)h(\(in)o(teger\)) 117 1078 y(in)g Fu(comm)697 b Ft(comm)o(unication)15 b(ob)r(ject)117 1120 y(in)f Fu(op-mode)631 b Ft(one)19 b(of)g Fm(MPI)p 1135 1120 12 2 v 13 w(STANDARD)p Ft(,)d Fm(MPI)p 1395 1120 V 13 w(READY)p Ft(,)g Fm(MPI)p 1595 1120 V 13 w(SYNCHRONOU)o(S)g Ft(or)905 1170 y Fm(MPI)p 967 1170 V 13 w(RECV)117 1212 y Ft(in)e Fu(persistence)543 b Ft(one)13 b(of)g Fm(MPI)p 1123 1212 V 13 w(PERSISTEN)o(T)d Ft(or)j Fm(MPI)p 1452 1212 V 13 w(EPHEMERAL)158 1289 y Fx(The)h(comm)o(unication)d(in)o(v)o(olv)o(es)i(only)g(one)h(elemen)o (t,)f(with)g(an)h(absolute)g(address.)75 1402 y Fq(1.14.1)49 b(P)o(ersistent)14 b(comm)o(unication)g(objects)75 1479 y Fu(MPI)p 144 1479 14 2 v 15 w(CREATE)p 291 1479 V 15 w(SEND\(handle,)19 b(MPI)p 655 1479 V 15 w(BOTTOM,)h(1,)i(datatype,)e(dest,)g(tag,)h(comm\))13 b Fx(is)75 1553 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,) g(comm,)g(MPI_STANDARD,)e(MPI_PERSISTENT\))158 1627 y Fx(The)14 b(functions)g Fu(MPI)p 491 1627 V 15 w(CREATE)p 638 1627 V 15 w(RSEND)p Fx(,)e Fu(MPI)p 853 1627 V 15 w(CREATE)p 1000 1627 V 14 w(RSEND)h Fx(and)h Fu(MPI)p 1284 1627 V 15 w(CREATE)p 1431 1627 V 14 w(RECV)f Fx(are)h(dealt)g(in)f(a)h(sim-)75 1677 y(ilar)f(manner.)158 1727 y(The)h(functions)g Fu(MPI)p 491 1727 V 15 w(START)f Fx(and)h Fu(MPI)p 776 1727 V 15 w(COMM)p 879 1727 V 15 w(FREE)f Fx(are)h(primitiv)o(e.)75 1840 y Fq(1.14.2)49 b(Nonblo)q(cking)19 b(comm)n(unication)14 b(initiation)75 1917 y Fu(MPI)p 144 1917 V 15 w(ISEND\(handle,)19 b(MPI)p 530 1917 V 15 w(BOTTOM,)i(1,)g(datatype,)f(dest,)h(tag,)g(comm\))12 b Fx(is)75 1991 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,) g(comm,)g(MPI_STANDARD,)e(MPI_EPHEMERAL\))75 2041 y(MPI_START\(handle\))158 2115 y Fx(The)14 b(functions)g Fu(MPI)p 491 2115 V 15 w(IRSEND)p Fx(,)f Fu(MPI)p 729 2115 V 15 w(ISSEND)f Fx(and)i Fu(MPI)p 1035 2115 V 15 w(IRECV)f Fx(are)h(handled)g(in)f(a)h(similar)d(manner.)75 2228 y Fq(1.14.3)49 b(Comm)o(unication)14 b(completion)75 2305 y Fx(The)g(t)o(w)o(o)g(primitiv)o(e)d(completion)i(op)q(erations)h(are)g Fu(MPI)p 969 2305 V 15 w(WAITANY)e Fx(and)i Fu(MPI)p 1297 2305 V 15 w(TESTALL)p Fx(.)158 2355 y(The)c(function)f Fu(MPI)p 466 2355 V 15 w(WAIT)f Fx(can)i(b)q(e)g(implemen)o(ted)d(b)o(y)i(a)g(call)g (to)g Fu(MPI)p 1209 2355 V 15 w(WAITANY)f Fx(with)h(an)g Fu(array-of-handles) 75 2405 y Fx(argumen)o(t)k(of)g(length)h(one.)158 2455 y(The)j(function)e Fu(MPI)p 479 2455 V 15 w(WAITALL)g Fx(can)h(b)q(e)h(implemen)o(ted)d(as)i(a)g (lo)q(op)f(where)j(an)d Fu(MPI)p 1479 2455 V 15 w(WAIT)h Fx(call)f(is)h (executed)75 2504 y(for)e(eac)o(h)g(successiv)o(e)i(handle)e(in)f(the)i Fu(array-of-handles)o Fx(.)158 2554 y(The)d(function)e Fu(MPI)p 469 2554 V 15 w(TEST)h Fx(can)g(b)q(e)g(implemen)o(ted)e(b)o(y)i(a)g(call)f (to)h Fu(MPI)p 1226 2554 V 15 w(TESTALL)f Fx(with)g(an)h Fu(array-ofhandles) 75 2604 y Fx(of)i(length)h(one.)158 2654 y(The)j(function)e Fu(MPI)p 479 2654 V 15 w(TESTALL)g Fx(can)h(b)q(e)h(implemen)o(ted)d(as)i(a)g (lo)q(op)f(where)j(an)d Fu(MPI)p 1479 2654 V 15 w(TEST)h Fx(call)f(is)h (executed)75 2704 y(for)e(eac)o(h)g(successiv)o(e)i(handle)e(in)f(the)i Fu(array-of-handle)p Fx(.)p eop %%Page: 39 39 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(39)75 45 y Fq(1.14.4)49 b(Blo)q(cking)18 b(comm)o(unication)75 122 y Fu(MPI)p 144 122 14 2 v 15 w(SEND\(MPI)p 335 122 V 14 w(BOTTOM,)i(1,)i(datatype,)e(dest,)h(tag,)f(comm\))13 b Fx(is)75 210 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,)g(comm,)g (MPI_STANDARD,)e(MPI_EPHEMERAL\))75 260 y(MPI_START\(handle\))75 309 y(MPI_WAIT\(handle,)f(dontcarestatus\))158 398 y Fx(The)c(functions)g Fu(MPI)p 491 398 V 15 w(RSEND)f Fx(and)h Fu(MPI)p 776 398 V 15 w(SSEND)f Fx(are)h(handled)g(in)f(a)h(similar)d(manner.)158 447 y Fu(MPI)p 227 447 V 15 w(RECV\(MPI)p 418 447 V 14 w(BOTTOM,)20 b(1,)i(datatype,)e(source,)g(tag,)h(comm\))13 b Fx(is)75 535 y Fu(MPI_COMM_INIT\(han)o(dle,)18 b(datatype,)i(dest,)h(tag,)g(comm,)g (MPI_RECV,)f(MPI_EPHEMERAL\))75 585 y(MPI_START\(handle\))75 635 y(MPI_WAIT\(handle,)e(status\))75 751 y Fq(1.14.5)49 b(Prob)q(e)17 b(and)f(cancel)75 827 y Fx(The)e(functions)g Fu(MPI)p 408 827 V 15 w(PROBE)f Fx(and)h Fu(MPI)p 693 827 V 15 w(CANCEL)f Fx(are)h(primitiv)o (e.)75 943 y Fq(1.14.6)49 b(Return)16 b(status)75 1020 y Fx(The)e(functions)g Fu(MPI)p 408 1020 V 15 w(GET)p 489 1020 V 15 w(SOURCE)p Fx(,)e Fu(MPI)p 726 1020 V 15 w(GET)p 807 1020 V 15 w(TAG)p Fx(,)g Fu(MPI)p 978 1020 V 15 w(GET)p 1059 1020 V 15 w(LEN)p Fx(,)h Fu(MPI)p 1231 1020 V 15 w(PROBE)p 1356 1020 V 15 w(LEN)p Fx(,)f(and)i Fu(MPI)p 1608 1020 V 15 w(IS)p 1667 1020 V 15 w(CANCELLED)75 1069 y Fx(are)g(primitiv)o(e.)158 1119 y(These)h(functions)f(are)g(simple)f (macros)g(that)h(access)h(records)h(in)d(a)h(structure.)75 1235 y Fq(1.14.7)49 b(send-receive)15 b(and)i(exchange)75 1311 y Fu(MPI)p 144 1311 V 15 w(SENDRECV\()j(send)p 465 1311 V 15 w(start,)g(send)p 720 1311 V 15 w(count,)h(send)p 976 1311 V 15 w(type,)f(dest,)h(recv)p 1340 1311 V 15 w(start,)g(recv)p 1596 1311 V 14 w(count,)g(recv)p 1851 1311 V 15 w(type,)75 1361 y(source,)f(tag,)h(comm,)g(status\))12 b Fx(is)75 1449 y Fu(MPI_ISEND\(handle[)o(0],)19 b(send_start,)g(send_count,)h(send_type,)f (dest,)i(tag,)g(comm\))75 1499 y(MPI_IRECV\(handle[)o(1],)e(recv_start,)g (recv_count,)h(recv_type,)f(source,)i(tag,)g(comm\))75 1549 y(MPI_WAITALL\(2,)e(handle,)h(status_array\))75 1599 y(status)h(=)g (status_array[1])158 1687 y Fx(The)14 b(non)o(blo)q(c)o(king)f(sends)i(and)f (receiv)o(es)h(can)f(b)q(e)h(replaced)g(b)o(y)f(their)g(primitiv)o(e)d (implemen)o(tation.)158 1737 y(The)17 b(function)f Fu(MPI)p 480 1737 V 15 w(EXCHANGE)e Fx(can)i(b)q(e)h(handled)f(in)g(a)g(similar)e (manner;)i(a)g(temp)q(orary)f(bu\013er)i(need)g(b)q(e)75 1787 y(allo)q(cated)d(to)g(replicate)h(the)g(send)g(and)f(receiv)o(e)i(bu\013er.) 21 b(A)14 b(size)h(for)f(this)g(bu\013er)i(can)e(b)q(e)h(computed)f(using)g (the)75 1836 y(function)g Fu(MPI)p 307 1836 V 15 w(TYPE)p 410 1836 V 15 w(EXTENT)p Fx(.)75 1952 y Fq(1.14.8)49 b(Derived)16 b(datat)o(yp)q(es)75 2029 y Fx(W)m(e)22 b(ha)o(v)o(e)g(outlined)f(in)h (Section)g(1.13.1)f(ho)o(w)g(eac)o(h)i(datat)o(yp)q(e)f(constructor)i(can)e (b)q(e)h(expressed)i(in)c(terms)75 2078 y(of)f(the)h(next)h(one.)38 b(Th)o(us)21 b(all)f(datat)o(yp)q(e)g(constructors)j(can)e(b)q(e)g(expressed) i(in)e(terms)f(of)g(the)h(constructor)75 2128 y Fu(MPI)p 144 2128 V 15 w(TYPE)p 247 2128 V 15 w(STRUC)p Fx(;)12 b(the)j(functions)f Fu(MPI)p 713 2128 V 15 w(TYPE)p 816 2128 V 15 w(FREE)p Fx(,)e Fu(MPI)p 1009 2128 V 15 w(TYPE)p 1112 2128 V 15 w(EXTENT)h Fx(and)g Fu(MPI)p 1418 2128 V 16 w(ADDRESS)f Fx(are)i(primitiv)o(e.)75 2244 y Fq(1.14.9)49 b(Summ)o(a)o(r)o(y)75 2321 y Fx(In)13 b(order)h(to)f (implemen)o(t)e(the)j(MPI)f(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication)e (one)k(needs)g(16)f(functions;)g(ab)q(out)g(half)g(of)75 2370 y(those)i(are)f(trivial)e(macros:)126 2458 y(1.)20 b Fu(MPI)p 248 2458 V 15 w(COMM)p 351 2458 V 15 w(INIT)126 2540 y Fx(2.)g Fu(MPI)p 248 2540 V 15 w(START)126 2622 y Fx(3.)g Fu(MPI)p 248 2622 V 15 w(WAITALL)126 2704 y Fx(4.)g Fu(MPI)p 248 2704 V 15 w(TESTANY)p eop %%Page: 40 40 bop 75 -100 a Fr(SECTION)15 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m (TION)732 b Fx(40)126 45 y(5.)20 b Fu(MPI)p 248 45 14 2 v 15 w(PROBE)126 128 y Fx(6.)g Fu(MPI)p 248 128 V 15 w(CANCEL)126 211 y Fx(7.)g Fu(MPI)p 248 211 V 15 w(GET)p 329 211 V 15 w(SOURCE)126 294 y Fx(8.)g Fu(MPI)p 248 294 V 15 w(GET)p 329 294 V 15 w(TAG)126 377 y Fx(9.)g Fu(MPI)p 248 377 V 15 w(GET)p 329 377 V 15 w(LEN)105 460 y Fx(10.)g Fu(MPI)p 248 460 V 15 w(PROBE)p 373 460 V 15 w(LEN)105 543 y Fx(11.)g Fu(MPI)p 248 543 V 15 w(IS)p 307 543 V 15 w(CANCELLED)105 626 y Fx(12.)g Fu(MPI)p 248 626 V 15 w(TYPE)p 351 626 V 15 w(STRUC)105 709 y Fx(13.)g Fu(MPI)p 248 709 V 15 w(TYPE)p 351 709 V 15 w(EXTENT)105 792 y Fx(14.)g Fu(MPI)p 248 792 V 15 w(ADDRESS)105 875 y Fx(15.)g Fu(MPI)p 248 875 V 15 w(TYPE)p 351 875 V 15 w(FREE)105 958 y Fx(16.)g Fu(MPI)p 248 958 V 15 w(COMM)p 351 958 V 15 w(FREE)158 1050 y Fx(On)13 b(the)h(other)g(hand,)e(w)o(e)i(exp)q(ect)g(most)e(implemen)o(tations)e(to)j (use)h(more)e(primitiv)o(es,)f(in)i(order)g(to)g(reduce)75 1099 y(comm)o(unication)d(o)o(v)o(erheads,)15 b(reduce)g(bu\013ering,)f(and)g (impro)o(v)o(e)e(error)j(rep)q(orting.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Mon Aug 2 15:39:52 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA08277; Mon, 2 Aug 93 15:39:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06314; Mon, 2 Aug 93 15:38:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 2 Aug 1993 15:38:32 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06304; Mon, 2 Aug 93 15:38:31 -0400 Message-Id: <9308021938.AA06304@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8293; Mon, 02 Aug 93 15:38:33 EDT Date: Mon, 2 Aug 93 15:38:33 EDT From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: new point to point draft Reply-To: SNIR@watson.ibm.com Minor corrections with respect to version broadcast a week ago: mostly spelling and grammar. Did clarify how padding for alignment is handled, and how current dataypes can be used to send in one message variables that do not belong to the same sequential storage. This version is likely to stay unchanged until our coming meeting. At that meeting we shall need to vote on the new datatype proposal (1st vote) and on changes elsewhere (2nd vote, I assume). From owner-mpi-pt2pt@CS.UTK.EDU Thu Aug 19 07:07:06 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA04820; Thu, 19 Aug 93 07:07:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28482; Thu, 19 Aug 93 07:06:18 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 19 Aug 1993 07:06:17 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28474; Thu, 19 Aug 93 07:06:11 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA05474 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Thu, 19 Aug 1993 12:06:05 +0100 Date: Thu, 19 Aug 1993 12:06:05 +0100 From: James Cownie Message-Id: <199308191106.AA05474@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA01732; Thu, 19 Aug 93 12:04:57 BST To: mpi-pt2pt@cs.utk.edu Subject: Error handling (really Environmental ?) Content-Length: 4250 Is error handling still in point to point ? or should this be goint to Enviromental ? Anyhow I can't remember the alias for environmental, so I'll send it here... Error handling (and modes) ========================== Marc used to have a section in the point to point chapter on how MPI handles user errors. (But it seems to have gone away...) This suggested (in effect) that we "raise a signal" on an MPI error, and have a default handler which executes to core dump, or whatever. It would be illegal to attempt to communicate after an error had been raised. This raises a few questions :- 1) We can't really rely on raising a "real" signal, since MPI need not run on Unix, therefore we probably have to have our own MPI routines for registering error handlers, analogous to the signal style ones. 2) Do we have a single MPI error routine, or do we want to distinguish different MPI errors. (e.g. Receive overrun vs Insufficient system buffering vs ...) ? If we want to distinguish, then do we have to enumerate all of the MPI standard error conditions ? Or can we allow these to be implementation defined (and if so what's the use of the standard ?) ? 3) What arguments get passed to the user's error handler ? e.g. If the error is associated with a communicator does it get that ? 4) When is the error handler called ? Possibilities include :- Asynchronously as soon as the library knows there's a problem. When the next MPI routine is called by the user. When the next "heavyweight" MPI routine is called. ... What do we promise ? 5) As detailed above this error handler represents global state. It is not easy for a library to insert its own handler(s). One possible solution here would be to associate the handler(s) with a communicator (and its descendents). This seems elegant, and solves the library problem. (It's also relatively cheap unless many different handlers are used) However there's still the problem of errors which represent a more global failure (e.g. no buffering available for a received message), which handler do we now invoke ? (Note that there may not even be a communicator for the message at the time it arrives) My opinion (for what it's worth) ================================ I'd have a single MPI error handler associated with a communicator, so I'd provide a routine for associating an error handler with a communicator. e.g. MPI_ERRHANDLER mpi_set_handler(MPI_COMM comm, (void (*handler)(MPI_COMM,int *,...)); Which sets the handler and returns the previous one. This is a LOCAL routine, the communicator is NOT used for communication ! I'd inherit this error handler in all communicators derived from this one. I'd accept that the errors will be machine dependent, but provide a defined way of mapping an error code into a string, so you can at least write a handler which says something beyond "Error 92". I'd give the error handler a prototype like this void MPI_ERROR_HANDLER( MPI_COMM communicator, int * errcode, ...) { } (errcode is only int * so that you can write this routine in Fortran if you must !) The ... arguments would be implementation defined. The communicator could be MPI_COMM_NULL if the error could not be related to a particular communicator. I'd give errors which couldn't be associated with a specific communicator to the error handler for MPI_COMM_ALL, for want of anywhere better. I'd make no promises about when the handler gets called, beyond that it's before the user completion routine (WAIT, TEST, blocking op) for the failing operation completes. (I think this is the least you can say. It certainly seems unreasonable to return a success status to the user, and then raise an error exception for the same transaction !) I'd state that the handler should call MPI_ABORT(), or return in which case MPI_ABORT(errcode) will be called anyway. Thoughts, other options ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Aug 19 19:44:48 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA11735; Thu, 19 Aug 93 19:44:48 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15634; Thu, 19 Aug 93 19:43:57 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 19 Aug 1993 19:43:54 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15625; Thu, 19 Aug 93 19:43:51 -0400 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA18701; Thu, 19 Aug 93 23:43:33 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA22792; Thu, 19 Aug 93 17:47:31 MDT Date: Thu, 19 Aug 93 17:47:31 MDT From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9308192347.AA22792@nipmuc.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu, jim@meiko.co.uk Subject: Re: Error handling (really Environmental ?) Jim, I like your comments regarding error handlers (I voted in favor of retaining communicators because of the reasons you gave). I would like to see that error statuses are returned if MPI_NULL_HANDLER (or some such name) is given as the handler. With the inheritance you suggest, this would allow an easy way for the user to always get statuses by assigning the null handler to MPI_COMM_ALL. I realize that there are some errors that cannot be associated with a particular call or are so severe that the only thing to do is abort. This is a fine behaviour for those (implemnentation dependent) cases. Regards, Leslie P.S. I forgot how the vote went on always having a status returned. P.P.S. I would like to see indicating the null handler actually turn off signalling for that communicator rather than going through a benign handler. IEEE floating point underflows kill us in some configurations of our application (so, we turn it off). From owner-mpi-pt2pt@CS.UTK.EDU Mon Aug 23 07:16:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA06624; Mon, 23 Aug 93 07:16:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00767; Mon, 23 Aug 93 07:15:07 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 23 Aug 1993 07:15:05 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00710; Mon, 23 Aug 93 07:15:00 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA16922 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 23 Aug 1993 10:19:24 +0100 Date: Mon, 23 Aug 1993 10:19:24 +0100 From: James Cownie Message-Id: <199308230919.AA16922@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA02009; Mon, 23 Aug 93 10:18:09 BST To: mpi-pt2pt@cs.utk.edu In-Reply-To: <9308192347.AA22792@nipmuc.fsl.noaa.gov> (hart@nipmuc.fsl.noaa.gov) Subject: Re: Error handling (really Environmental ?) Content-Length: 1064 Leslie, > I like your comments regarding error handlers (I voted in favor of > retaining communicators because of the reasons you gave). Thanks. > I would > like to see that error statuses are returned if MPI_NULL_HANDLER > (or some such name) is given as the handler. With the inheritance > you suggest, this would allow an easy way for the user to always > get statuses by assigning the null handler to MPI_COMM_ALL. I would perhaps expect this to be the default anyway... It's somewhat unclear to me whether there are actually any useful levels of erros in MPI, or does anything unsuccessful cause total failure ? I think the way it is at the moment any erroneous condition is fatal. This is probably reasonable ... Anyone any other ideas ? We MUST get this done next time ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Aug 25 11:01:20 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA24378; Wed, 25 Aug 93 11:01:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01068; Wed, 25 Aug 93 10:56:22 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 25 Aug 1993 10:56:21 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01041; Wed, 25 Aug 93 10:56:11 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA11115 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Wed, 25 Aug 1993 15:56:06 +0100 Date: Wed, 25 Aug 1993 15:56:06 +0100 From: James Cownie Message-Id: <199308251456.AA11115@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00653; Wed, 25 Aug 93 15:54:46 BST To: mpi-pt2pt@cs.utk.edu Subject: Cacheing Content-Length: 3610 At the last meeting the context committee presented a proposal for a cacheing facility within MPI to allow the user to associate data with two classes of MPI objects (groups and comunicators). This proposal was greeted with less than total enthusiasm, particularly (it appeared) because of concerns about store allocation issues. Let me explain why I am in favour of a cacheing proposal, and why I don't believe that store allocation is a (large) issue. Recap on the extant proposal ============================ The proposal provides the ability for user code to be given unique keys which it can use to store a void * on any group or communicator. If you like there is a symbol table associating the pair (MPI_OBJ,key) with the void * value. (A better implementation might be to have the lookup table stored on the MPI object, and actually implement it as a flat table indexed by the key, but this is not necessary for the scheme to work) In addition it associates with each key value two functions, a destructor function called by MPI when an MPI object which has a value associated with this key is destroyed, and a replicator function called by MPI when an MPI object which has a value associated with this key is replicated. (e.g. MPI_GROUP_DUP). Note that these functions are ALWAYS called synchronously to the execution of the user code, since they are called when the user is making an MPI call. (MPI_xxx_DUP, or MPI_xxx_FREE), they are NEVER called by the MPI system "in an interrupt routine". What does this proposal buy us ? ================================ 1) It makes it much easier to support layering of functionality on top of MPI. e.g. Topology information needs to be associated with MPI group objects, and copied or deleted as the groups are changed. User collective operations may need to have a separate communicator from that passed to them (so that they operate in their own context) They can now associate this with the communicator they are passed, AND (because of the call back) correctly free it when the parent communicator is deleted by the user code. 2) It allows the user to write collective functions which behave EXACTLY like the MPI collective functions. Without the callbacks this is impossible, as there is no point at which the user code gains control to free its data structures when MPI objects are deleted. The major gain comes from the call back functions. The cacheing is merely a convenient place to hang the callback functions, and know which ones should be called. What does it cost ? =================== I don't think the cost is large. Indeed it could be implemented using the function re-naming scam on top of existing MPI without any problem. As I point out above, all of the store allocation can be done (should be done ?) in user space on the way into or out of the system. There is no need for this to impact anything "inside the kernel" at all. The user functions are only called while we're on the user stack anyway, as a result of a user call to MPI. They're also only called at non-time critical points (how often does a good code create or replicate groups compared with how many times it communicates ?). Conclusion ========== I think is buys us a lot (in expansibility) and costs us a little. I think we shold have it ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Sep 2 14:25:25 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA17205; Thu, 2 Sep 93 14:25:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08598; Thu, 2 Sep 93 14:21:49 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 2 Sep 1993 14:21:47 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08542; Thu, 2 Sep 93 14:20:40 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Thu, 2 Sep 93 11:13 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA20197; Thu, 2 Sep 93 11:10:48 PDT Date: Thu, 2 Sep 93 11:10:48 PDT From: d39135@snacker.pnl.gov Subject: safe buffering proposal To: als@cs.umd.edu, babb@cs.du.edu, bkg@llnl.gov, bob@lanl.gov, dcheng@nas.nasa.gov, dongarra@cs.utk.edu, elster@cs.cornell.edu, evdv@ama.caltech.edu, feenyj@vnet.endicott.ibm.com, gropp@mcs.anl.gov, gst@ornl.gov, hart@fsl.noaa.gov, heller@shell.com, hender@fsl.noaa.gov, howell@zach.fit.edu, igl@ecs.soton.ac.uk, jim@meiko.co.uk, joelw@convex.com, knighten@ssd.intel.com, lederman@super.org, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, moose@think.com, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu, nessett@llnl.gov, otto@cse.ogi.edu, peter@sun.math.usfca.edu, pmadams@ncube.com, rj_littlefield@pnlg.pnl.gov, snir@watson.ibm.com, tony@cs.msstate.edu, walker@msr.epm.ornl.gov, wkh@almaden.ibm.com Cc: d39135@snacker Message-Id: <9309021810.AA20197@snacker.pnl.gov> X-Envelope-To: dongarra@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu PROPOSAL TO SUPPORT SAFE MPI MESSAGE BUFFERING IN USER-PROVIDED SPACE This is to formally submit the buffering proposal that I outlined at the last MPI meeting. I have copied it to all attendees of the last MPI meeting to make sure that all those who requested individual copies get them. People with long memories may note that an earlier version of this proposal appeared in mpi-pt2pt and mpi-envir on April 8. Relevant discussion occurred on Feb 8 and Apr 8-11 in mpi-pt2pt. The current proposal extends the April 8 version by associating buffer space with a communicator. This extension allows to . cancel MPI access to the user-provided buffer (by freeing the communicator), and . specify multiple buffers (by associating them with separate communicators). The complete proposal follows. 1. INTERFACE An application program can optionally provide space for MPI's use in buffering message data via the following call: MPI_USER_PROVIDES_BUFFER (comm,len,buffer) where IN comm is a communicator with which the buffer is to be associated; IN len is the length of 'buffer', in bytes; OUT buffer is a scratch array of len bytes for MPI's use in buffering messages. 2. FUNCTIONALITY By default, application programs are "unsafe" if they assume buffering. However, an application can make it safe to assume a specified amount of data buffering, by providing buffer space via the MPI_USER_PROVIDES_BUFFER call. If MPI_USER_PROVIDES_BUFFER is called, then for subsequent regular sends using the specified communicator, MPI must provide as much safety *as if* outgoing message data were buffered by the sending process, in the provided buffer space, using a circular contiguous-space allocation policy. That is, user-provided buffer space may be consumed by outgoing messages, but not by incoming ones, and MPI is not required to split any message if it must be copied to the buffer. Following the call to MPI_USER_PROVIDES_BUFFER, the application is not permitted to access the buffer until the corresponding communicator has been freed. Access to the provided buffer space is not inherited by communicators derived from the initially specified one. 3. POSSIBLE IMPLEMENTATION One approach is for MPI to simply transform all blocking sends on the specified communicator into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) No doubt many optimizations within MPI are possible -- the proposal just requires that MPI provide at least this much safety. 4. DISCUSSION This proposal does not provide guarantees on the *number* of outstanding sends, only their aggregate message length. Not guaranteeing number is consistent with MPI's general stance of treating limits as a quality-of-implementation issue. Guaranteeing the length is nominally at odds with MPI's general stance, but is felt to be an important compromise in order to support programs that assume current "common practice". It is unreasonable to simply declare existing programs to be "unsafe", when a simple method exists to make them safe. ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Sep 2 20:33:44 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA18902; Thu, 2 Sep 93 20:33:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01717; Thu, 2 Sep 93 20:31:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 2 Sep 1993 20:31:23 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01681; Thu, 2 Sep 93 20:30:44 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Thu, 2 Sep 93 17:30 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA28225; Thu, 2 Sep 93 17:28:02 PDT Date: Thu, 2 Sep 93 17:28:02 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: safe buffering proposal To: hender@macaw.fsl.noaa.gov, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309030028.AA28225@snacker.pnl.gov> X-Envelope-To: mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu In non-reflected email, Tom Henderson wrote: > In general, I like your proposal. One thing is confusing me: > > > MPI_USER_PROVIDES_BUFFER (comm,len,buffer) > > > > where IN comm is a communicator with which the > > buffer is to be associated; > > > > IN len is the length of 'buffer', in bytes; > > > > OUT buffer is a scratch array of len bytes for MPI's > > use in buffering messages. > > Why is buffer an OUT argument? (I assume that the routine malloc()'s buffer > in user space.) Why would a user want a pointer to a scratch array s/he is > not allowed to use? > ... > Feel free to pass this on to pt2pt if you like. The 'buffer' array is allocated by the caller of MPI_USER_PROVIDES_BUFFER, in whatever way the caller finds convenient -- static, automatic, or malloc'd. The caller then passes a pointer to this array, to MPI_USER_PROVIDES_BUFFER. Trying to describe 'buffer' as IN or OUT may be more confusing than helpful in this case. MPI_USER_PROVIDES_DATA does not care about the contents on entry, and the application cannot use the contents later. It would be better to call the buffer SCRATCH. > If we have MPI_USER_PROVIDES_BUFFER(), do we still need the "synchronous" > communication mode? (As I recall the definition, synchronous mode is where the send is guaranteed not to return, until the matching receive is sure to complete also. Assuming that definition is right, then...) MPI_USER_PROVIDES_BUFFER complements synchronous mode, but does not replace it. Synchronous mode guarantees that an unsafe program will fail. MPI_USER_PROVIDES_BUFFER allows an unsafe program to be made safe, assuming that you know how much buffering is required. My proposal did not do so, but I suppose the interaction between synchronous mode and MPI_USER_PROVIDES_BUFFER should be specified. Possible specifications include: 1. Synchronous mode cannot be used with communicators for which MPI_USER_PROVIDES_BUFFER has been called. 2. The behavior of mixing synchronous mode and MPI_USER_PROVIDES_BUFFER is explicitly undefined. 3. On a communicator for which MPI_USER_PROVIDES_BUFFER has been called, a synchronous send will block waiting for a matching receive if and only if the user-provided buffer space has been exhausted, assuming a circular contiguous-space allocation policy for that buffer. Specification 3 is intended to guarantee failure of a program that is still unsafe, even though MPI_USER_PROVIDES_BUFFER has been called. (I.e., for which not enough buffer space has been provided to guarantee safety.) HOWEVER, I suspect that specification 3 is inadequate, and I would not be surprised even to find that its intent is impossible to accomplish for all possible programs. The reason is that any sort of buffering at least partially uncouples the dependencies between processes, and I imagine that it could be very difficult or impossible to guarantee safety of all possible execution sequences by observing only one. Personally, I would propose for now to use specification 2: explicitly undefined behavior. That way it can be fixed later if anyone cares and also figures out what it should be. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Fri Sep 3 07:52:05 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20623; Fri, 3 Sep 93 07:52:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17651; Fri, 3 Sep 93 07:50:18 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 3 Sep 1993 07:50:17 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17618; Fri, 3 Sep 93 07:50:07 -0400 Via: uk.ac.southampton.ecs; Fri, 3 Sep 1993 12:48:01 +0100 Via: brewery.ecs.soton.ac.uk; Fri, 3 Sep 93 12:38:47 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Fri, 3 Sep 93 12:50:03 BST Date: Fri, 3 Sep 93 12:50:04 BST Message-Id: <13242.9309031150@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: safe buffering proposal > From: d39135@gov.pnl.snacker > That is, user-provided buffer space may be consumed by outgoing > messages, but not by incoming ones, and MPI is not required > to split any message if it must be copied to the buffer. Could you explain what the bit about MPI not being required to split messages means please? Ian From owner-mpi-pt2pt@CS.UTK.EDU Fri Sep 3 08:09:33 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20703; Fri, 3 Sep 93 08:09:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18769; Fri, 3 Sep 93 08:07:47 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 3 Sep 1993 08:07:45 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18734; Fri, 3 Sep 93 08:07:41 -0400 Via: uk.ac.southampton.ecs; Fri, 3 Sep 1993 13:04:30 +0100 Via: brewery.ecs.soton.ac.uk; Fri, 3 Sep 93 12:55:15 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Fri, 3 Sep 93 13:06:30 BST Date: Fri, 3 Sep 93 13:06:33 BST Message-Id: <13253.9309031206@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: safe buffering proposal > From: rj_littlefield@gov.pnl.pnlg > My proposal did not do so, but I suppose the interaction between > synchronous mode and MPI_USER_PROVIDES_BUFFER should be specified. > > Possible specifications include: > > 1. Synchronous mode cannot be used with communicators for which > MPI_USER_PROVIDES_BUFFER has been called. > > 2. The behavior of mixing synchronous mode and MPI_USER_PROVIDES_BUFFER > is explicitly undefined. > > 3. On a communicator for which MPI_USER_PROVIDES_BUFFER has been > called, a synchronous send will block waiting for a matching > receive if and only if the user-provided buffer space has been > exhausted, assuming a circular contiguous-space allocation > policy for that buffer. How about: 4. The behaviour of synchronous mode communications is unaffected by calls to MPI_USER_PROVIDES_BUFFER. > Personally, I would propose for now to use specification 2: > explicitly undefined behavior. That way it can be fixed later > if anyone cares and also figures out what it should be. Why complicate things? The whole point of the synchronous mode is that it has a clean semantics that is unaffected by the vagaries of system buffering (or system administered buffering). I actually WANT my synchronous sends to block until a matching receive commits. This is not, as option 3 implies, an unwanted side-effect of the mode. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Fri Sep 3 12:04:04 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22244; Fri, 3 Sep 93 12:04:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04866; Fri, 3 Sep 93 12:01:56 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 3 Sep 1993 12:01:54 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04858; Fri, 3 Sep 93 12:01:52 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Fri, 3 Sep 93 09:00 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA01789; Fri, 3 Sep 93 08:57:27 PDT Date: Fri, 3 Sep 93 08:57:27 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: safe buffering proposal To: igl@ecs.soton.ac.uk, mpi-pt2pt@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309031557.AA01789@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Ian Glendinning writes: > How about: > > 4. The behaviour of synchronous mode communications is unaffected by calls > to MPI_USER_PROVIDES_BUFFER. > > Why complicate things? The whole point of the synchronous mode is > that it has a clean semantics that is unaffected by the vagaries of > system buffering (or system administered buffering). I actually WANT > my synchronous sends to block until a matching receive commits. This > is not, as option 3 implies, an unwanted side-effect of the mode. Offline, Robert Harrison commented: > I must disagree with what I think your intent is on synchronous > mode combined with user-provides buffer. The semantics of synchronous > mode (send blocks until receive issued) require no buffering and > under some circumstances (e.g., a necessary synchronization) is > a very useful mode. Why should the availability of buffering when > none is needed change this? Rik replies: I will happily yield on this point. My personal interpretation of synchronous mode is that its main purpose is to facilitate writing safe programs. Since the definition of safety is changed by MPI_USER_PROVIDES_BUFFER, it seems to me conceptually cleaner for the two to interact as I had previously outlined. However, as I commented in my earlier posting, it is not obvious that my preferred definition (option 3) can actually be implemented. Ian and Robert apparently have a different interpretation, with more emphasis on forced synchronization than safety. The resulting option 4 is easy to explain and implement, and I'm surely in favor of that! Anybody else care? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Fri Sep 3 14:07:27 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23520; Fri, 3 Sep 93 14:07:27 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14587; Fri, 3 Sep 93 14:05:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 3 Sep 1993 14:05:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14569; Fri, 3 Sep 93 14:05:28 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Fri, 3 Sep 93 11:00 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA02431; Fri, 3 Sep 93 10:57:26 PDT Date: Fri, 3 Sep 93 10:57:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: safe buffering proposal To: igl@ecs.soton.ac.uk, mpi-pt2pt@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309031757.AA02431@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Ian Glendinning writes: > > From: rj_littlefield@pnl.gov: > > > That is, user-provided buffer space may be consumed by outgoing > > messages, but not by incoming ones, and MPI is not required > > to split any message if it must be copied to the buffer. > > Could you explain what the bit about MPI not being required to split messages > means please? The intent was to simplify handling of buffer fragmentation. (Fragmentation can result from receives being issued in a different order from sends.) My gut feeling is that if MPI has to be able to utilize arbitrary noncontiguous buffer fragments, then the guarantee will become difficult to explain and/or implement. The key point of my proposal is that an implementation be allowed to allocate buffer space circularly, strictly in terms of send order. Thus, the required amount of buffer space is equal to the combined data length of all messages sent after (and including) the first one that has not yet been received. I do not see any simpler guarantee. However, the words "not required to split any message" may be too strong, because the simple guarantee requires that a message be able to wrap from the end to the beginning of the buffer. If MPI is not required to wrap, then the guarantee would have to become something like "the required amount of buffer space is equal to the combined data length of all messages ... plus an extra copy of the current message". I find this distasteful, and would rather require wrapping. (Wrapping imposes essentially no extra burden on the rest of MPI, which already has to be prepared to handle noncontiguous data.) If someone can argue convincingly that fragmentation is NOT, in fact, difficult to handle, then I would be happy to delete the restriction altogether. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Mon Sep 6 06:12:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01524; Mon, 6 Sep 93 06:12:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23120; Mon, 6 Sep 93 06:09:43 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 6 Sep 1993 06:09:42 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23112; Mon, 6 Sep 93 06:09:40 -0400 Date: Mon, 6 Sep 93 11:09:19 BST Message-Id: <19937.9309061009@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: safe buffering proposal To: Ian Glendinning In-Reply-To: Ian Glendinning's message of Fri, 3 Sep 93 13:06:33 BST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Just a wee one to say that I basically agree with Ian. If there is buffering then synchronous mode can just ignore such buffering. Cheers Lyndon > Why complicate things? The whole point of the synchronous mode is that it has > a clean semantics that is unaffected by the vagaries of system buffering (or > system administered buffering). I actually WANT my synchronous sends to > block until a matching receive commits. This is not, as option 3 implies, an > unwanted side-effect of the mode. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Sep 6 09:41:47 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02008; Mon, 6 Sep 93 09:41:47 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04644; Mon, 6 Sep 93 09:39:35 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 6 Sep 1993 09:39:34 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04636; Mon, 6 Sep 93 09:39:32 -0400 Via: uk.ac.southampton.ecs; Mon, 6 Sep 1993 14:38:05 +0100 Via: brewery.ecs.soton.ac.uk; Mon, 6 Sep 93 14:28:44 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Mon, 6 Sep 93 14:40:04 BST Date: Mon, 6 Sep 93 14:40:05 BST Message-Id: <15553.9309061340@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: RE: safe buffering proposal Cc: rj_littlefield@pnlg.pnl.gov Rik, I asked what you meant in your buffering proposal when you said "MPI is not required to split any message if it must be copied into the buffer.", and you replied, clarifying your intent. I now understand roughly what you are trying to achieve, but unfortunately I'm no clearer about the meaning of the original sentence. Perhaps I should have been more specific; What do you mean by splitting a message? It's rather hard to make sense of the rest without knowing that. Ian From owner-mpi-pt2pt@CS.UTK.EDU Mon Sep 6 17:43:19 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03268; Mon, 6 Sep 93 17:43:19 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02925; Mon, 6 Sep 93 17:40:45 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 6 Sep 1993 17:40:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02912; Mon, 6 Sep 93 17:40:42 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Mon, 6 Sep 93 14:40 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA11724; Mon, 6 Sep 93 14:37:53 PDT Date: Mon, 6 Sep 93 14:37:53 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: safe buffering proposal To: igl@ecs.soton.ac.uk, mpi-pt2pt@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309062137.AA11724@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Ian writes: > I asked what you meant in your buffering proposal when you said "MPI is > not required to split any message if it must be copied into the buffer.", and > you replied, clarifying your intent. I now understand roughly what you are > trying to achieve, but unfortunately I'm no clearer about the meaning of the > original sentence. Perhaps I should have been more specific; What do you > mean by splitting a message? It's rather hard to make sense of the rest > without knowing that. "Splitting a message" merely meant copying it to a noncontiguous set of memory addresses within the buffer. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 7 11:15:44 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02353; Tue, 7 Sep 93 11:15:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16213; Tue, 7 Sep 93 11:14:52 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 7 Sep 1993 11:14:50 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16205; Tue, 7 Sep 93 11:14:48 -0400 Via: uk.ac.southampton.ecs; Tue, 7 Sep 1993 16:14:19 +0100 Via: brewery.ecs.soton.ac.uk; Tue, 7 Sep 93 16:04:59 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Tue, 7 Sep 93 16:16:20 BST Date: Tue, 7 Sep 93 16:16:22 BST Message-Id: <16762.9309071516@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: RE: safe buffering proposal > From: rj_littlefield@gov.pnl.pnlg > "Splitting a message" merely meant copying it to a noncontiguous > set of memory addresses within the buffer. Thanks for that clarification. Now I'm really sure I don't understand the original sentence: "MPI is not required to split any message if it must be copied into the buffer." Why "not required to split"? Don't you mean "required not to split"? As written it doesn't seem to impose any constraint. It might split, or it might not. Still confused, Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 7 14:08:55 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03766; Tue, 7 Sep 93 14:08:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28491; Tue, 7 Sep 93 14:08:03 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 7 Sep 1993 14:08:02 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28483; Tue, 7 Sep 93 14:08:00 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Tue, 7 Sep 93 11:03 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA18807; Tue, 7 Sep 93 11:00:57 PDT Date: Tue, 7 Sep 93 11:00:57 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: safe buffering proposal To: igl@ecs.soton.ac.uk, mpi-pt2pt@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309071800.AA18807@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Ian Glendinning writes: > > From: rj_littlefield@gov.pnl.pnlg > > > "Splitting a message" merely meant copying it to a noncontiguous > > set of memory addresses within the buffer. > > Thanks for that clarification. Now I'm really sure I don't understand the > original sentence: > > "MPI is not required to split any message if it must be copied into the > buffer." > > Why "not required to split"? Don't you mean "required not to split"? As > written it doesn't seem to impose any constraint. It might split, or > it might not. Still confused, > Ian > -- Ian is fine; the clause is confused. Actually, it's pretty much nonsense, not even worthy of trying to explain. Instead, let me try again to say what I should have meant in the first place. Here is the complete "functionality" part of the proposal, as modified in light of the several emails that Ian and I have exchanged. Major changes are marked with bars. ---- partial spec follows -------------------- 2. FUNCTIONALITY By default, application programs are "unsafe" if they assume buffering. However, an application can make it safe to assume a specified amount of data buffering, by providing buffer space via the MPI_USER_PROVIDES_BUFFER call. If MPI_USER_PROVIDES_BUFFER is called, then for subsequent regular sends using the specified communicator, MPI must provide as much safety *as if* outgoing message data were buffered by the sending process, in the provided buffer space, using a circular contiguous-space allocation policy. | That is, a "send" on a communicator is guaranteed not to | block indefinitely IF the program has provided buffer space | that is equal to or longer than the combined data lengths of: | . the message currently being sent, | . the earliest outstanding (unreceived) message | that was sent on the same communicator, and | . all intervening messages sent on the same communicator. Buffer space is not consumed by incoming messages. Access to the provided buffer space is not inherited by communicators derived from the initially specified one. Following the call to MPI_USER_PROVIDES_BUFFER, the application is not permitted to access the buffer until the corresponding communicator has been freed. ---- end partial spec -------------------- Is this better? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 7 14:23:04 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03930; Tue, 7 Sep 93 14:23:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29644; Tue, 7 Sep 93 14:22:29 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 7 Sep 1993 14:22:28 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29636; Tue, 7 Sep 93 14:22:25 -0400 Via: uk.ac.southampton.ecs; Tue, 7 Sep 1993 19:21:23 +0100 Via: brewery.ecs.soton.ac.uk; Tue, 7 Sep 93 19:12:00 BST From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Tue, 7 Sep 93 19:23:20 BST Date: Tue, 7 Sep 93 19:23:21 BST Message-Id: <16949.9309071823@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: RE: safe buffering proposal Cc: rj_littlefield@pnlg.pnl.gov Rik Littlefield writes: > Ian is fine; the clause is confused. Actually, it's pretty much > nonsense, not even worthy of trying to explain. > > Instead, let me try again to say what I should have meant in the > first place. [Explanation deleted] > Is this better? Yes, much better. Thanks, Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 14 22:06:22 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA16108; Tue, 14 Sep 93 22:06:22 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12618; Tue, 14 Sep 93 22:04:07 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 14 Sep 1993 22:04:02 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12609; Tue, 14 Sep 93 22:03:44 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0719; Tue, 14 Sep 93 22:03:45 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 0551; Tue, 14 Sep 1993 22:03:36 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 14 Sep 93 22:03:29 EDT Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA20700; Tue, 14 Sep 1993 22:03:18 -0400 Date: Tue, 14 Sep 1993 22:03:18 -0400 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9309150203.AA20700@snir.watson.ibm.com> To: mpi-pt2pt@cs.utk.edu This is the current draft of the point to point section. The nontrivial changes are marked with margin marks, and should be discussed at coming meeting. Also need final approval of the derived datatypes. **************** delete above line ************ %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: pt2pt-v8.dvi %%Pages: 51 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 5 118 df<387CE0C0C0CCEC7C3806097E880B>99 D110 D<3078CCCCCCCCCC783006097E880B>I<303030FEFE3030363636 3C18070C7F8B0B>116 DI E /Fc 2 50 df<1E007F806180C0C0C0C0C0C0C0C0C0C0C0C0C0C061807F801E000A0D7E8C0E> 48 D<18F8F81818181818181818FFFF080D7D8C0E>I E /Fd 1 106 df<04040000000038FCD8 181832333E3C080F7E8E0B>105 D E /Fe 5 118 df<0F803FC071C06080E000C000C000C000E0 0060C071C03F800F000A0D7E8C0F>99 D110 D<1E003F0073806180C0C0C0C0C0C0C0C0C0C0618073803F001E00 0A0D7E8C0F>I<0C000C000C00FFC0FFC00C000C000C000C000C000C000C600C600CE00FC00380 0B107F8F0F>116 DI E /Ff 2 81 df<00001C00003C0000F80001E00003C0000780000F00000E00001E00003C 00003C00003C000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 0000780000F00000F00000F00001E00001E00003C0000380000700000E00001C0000780000E000 00E000007800001C00000E000007000003800003C00001E00001E00000F00000F00000F0000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 00007800007800007800007800007800007800007800007800007800007800007800003C00003C 00003C00001E00000E00000F000007800003C00001E00000F800003C00001C167C7B8121>40 D80 D E /Fg 3 111 df<03000380030000000000000000001C003E006600C600CC000C000C0018001980318033003F 001C00091480930C>105 D<0060007000600000000000000000038007C00CE018C018C000C000 C0018001800180018003000300030003006600EE00FC0078000C1A81930E>I<387C007CFE006F 8600CF0600CE06000E06000C06001C0C00180CC01818C0181980301F80300E00120D808C15> 110 D E /Fh 3 62 df<0FC01FE0387070386018E01CE01CE01CE01CE01CE01CE01CE01CE01C60 18703838701FE00FC00E137F9211>48 D<06001E00FE00EE000E000E000E000E000E000E000E00 0E000E000E000E000E000E00FFE0FFE00B137D9211>I<7FFFE0FFFFF000000000000000000000 0000000000000000FFFFF07FFFE0140A7E8B19>61 D E /Fi 25 123 df<007E01FE07800E001E 003C003C0078007FF87FF8F000F000F00070007000700038001C180FF807E00F147E9312>15 D<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C181838306040060E7C840D>I62 D<0FFFFFFC1FFFFFF81E 03C0781803C0383803C01830078018300780186007801860078030C00F0030C00F0030000F0000 000F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000007800 0000780000007800000078000000F0000000F0000000F0000001F000007FFFC0007FFFC0001E1F 7F9E1B>84 D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00 F00E00F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015147E9318>97 D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F 700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207E9F14>I<0000780003F8 0003F00000700000700000700000E00000E00000E00000E00001C00001C000F1C003FDC0078F80 0E07801C07803C07803C0700780700780700780700F00E00F00E00F00E00F00E18F01C30F03C30 707C3078FC603FCFC00F078015207E9F18>100 D<007C01FE07870E031E033C033C06781C7FF8 7FC0F000F000F000F000700070027007381E1FF807E010147E9315>I<00007C0000FE00019E00 039E00030C000700000700000700000700000E00000E00000E0000FFF001FFF0000E00001C0000 1C00001C00001C00001C0000380000380000380000380000380000700000700000700000700000 700000E00000E00000E00000E00000C00001C00031C000798000F300007E00003C000017297E9F 16>I<007000F000F00060000000000000000000000000000007001F8039C031C061C061C0C380 03800380070007000E000E000E181C301C301C601CE00FC007000D1F7F9E10>105 D<0000C00001E00001E00001C0000000000000000000000000000000000000000000003E00007F 0000C380018380030380030380060700000700000700000700000E00000E00000E00000E00001C 00001C00001C00001C0000380000380000380000380000700000700030700078E000F3C0007F80 003E00001328819E13>I<01E0000FE0000FC00001C00001C00001C00003800003800003800003 80000700000700000701C00707E00E0C300E18700E30F00E60F01CC0E01F80001F80001FE00038 F000383800383800381C307038607038607038407018C0E01F80600F0014207E9F18>I<03C01F C01F8003800380038007000700070007000E000E000E000E001C001C001C001C00380038003800 38007000700070007180E300E300E300E6007E003C000A207E9F0E>I<0F03E03E001F8FF0FF00 31DC39C38061F03B038061E01E01C061C01C01C0C3C03C03800380380380038038038003803803 800700700700070070070007007007000700700E0C0E00E00E180E00E01C180E00E01C300E00E0 0C701C01C00FE00C00C0078026147F9328>I<0F07C0001F8FE00031D8700061F0380061E03800 61C03800C3C070000380700003807000038070000700E0000700E0000700E0000701C1800E01C3 000E0383000E0386000E018E001C01FC000C00F00019147F931B>I<007C0001FF000383800F01 C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E 003838001FF00007C00013147E9316>I<03C1E007E7F80C7E3C187C1C18781E18701E30E01E00 E01E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E0007 00000700000E00000E00000E00000E00001C0000FFC000FFC000171D819317>I<0F0F001F9F80 31F1C061E1C061E3C061C3C0C3C3800380000380000380000700000700000700000700000E0000 0E00000E00000E00001C00000C000012147F9315>114 D<007C01FE0387070F070F070E0F0007 8007F803FC01FE001E000F700FF00EF00EE01C70383FF01FC010147E9315>I<00C000E001C001 C001C001C003800380FFF8FFF807000700070007000E000E000E000E001C001C001C001C183830 3830386038E01FC00F000D1C7F9B10>I<0780301FC03838C07030E07060E07061C070C1C0E003 80E00380E00380E00701C00701C00701C00701C3070386070386070786070F8C03F9F801F0F018 147F931A>I<07C3C00FE7E0187C703038706038F06038F0C070E000700000700000700000E000 00E00000E00070E030F1C060F1C060E1C0C0E3E1807E7F003C3E0014147E931A>120 D<0780301FC03838C07030E07060E07061C070C1C0E00380E00380E00380E00701C00701C00701 C00701C0070380070380070780070F8003FF0001F700000700000700000E003C0C003C1C003838 0030F0001FE0000F8000151D7F9316>I<01E06003F0C007FFC00E1F800C0300000600000C0000 180000300000600000C0000180000300000600C00C01801803803F87007FFE0061FC00C0700013 147E9315>I E /Fj 2 4 df0 D<040006000600C660F7E03F801F 003F80F7E0C6600600060004000B0D7E8D11>3 D E /Fk 13 122 df<70F8F8F8700000000000 00000070F8F8F8700512789116>58 D<1FE0007FF8007FFC00783C00301E00000E00003E0007FE 003FFE007FCE00F80E00E00E00E00E00F01E00F83E007FFFE03FF7E00FC3E013127E9116>97 D<03F80FFE1FFE3C1E780C7000F000E000E000E000E000F000700778073E0F1FFE0FFC03F01012 7D9116>99 D<003F00007F00003F0000070000070000070000070003E7000FFF003FFF003C1F00 780F00700F00F00700E00700E00700E00700E00700F00F00F00F00781F007C3F003FFFE01FF7F0 07C7E014197F9816>I<07E00FF81FFC3C3E780E700FF007FFFFFFFFFFFFE000F000700778073E 1F1FFE0FFC03F010127D9116>I<7E0000FE00007E00000E00000E00000E00000E00000E7C000F FF000FFF800F87800F03800F03800E03800E03800E03800E03800E03800E03800E03800E03800E 03807FC7F0FFE7F87FC7F01519809816>104 D<7E7C00FFFF007FFF800F87800F03800F03800E 03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F015128091 16>110 D<03E0000FF8001FFC003C1E00780F00700700E00380E00380E00380E00380E00380F0 0780700700780F003C1E001FFC000FF80003E00011127E9116>I114 D<0FEC3FFC7FFCF03CE01CE01CF0007F801FF007FC003EE00EE00EF0 0EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700000700000700007FFF00FFFF00FFFF 0007000007000007000007000007000007000007000007010007038007038007078007878003FF 0003FE0000F80011177F9616>I<7E1F80FE3F807E1F800E03800E03800E03800E03800E03800E 03800E03800E03800E03800E03800E07800F0F800FFFF007FFF803F3F01512809116>I<7F1FC0 FF9FE07F1FC01C07000E07000E0E000E0E00070E00071C00071C00039C00039C0003980001B800 01B80000F00000F00000F00000E00000E00000E00001C00079C0007BC0007F80003F00003C0000 131B7F9116>121 D E /Fl 7 104 df0 D<70F8F8F87005057C8D0D>I<018001C001800180C183E187F99F7DBE1FF807E007E01FF87DBE F99FE187C1830180018001C0018010147D9417>3 D<07E01FF83FFC7FFE7FFEFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E010127D9317>15 D 91 D<001F003F00F800E001C001C001C001C001C001C001C001C001C001C001C001C001C001C0 01C003800700FE00F800FE000700038001C001C001C001C001C001C001C001C001C001C001C001 C001C001C001C000E000F8003F001F102D7DA117>102 DI E /Fm 42 122 df<000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600000006000000060000FFFFFFE0FFFFFFE0000600000006 000000060000000600000006000000060000000600000006000000060000000600000006000000 060000000600001B1C7E9720>43 D45 D<001C0000003E0000003E 0000002E0000006700000067000000E7800000C7800000C3800001C3C0000183C0000181C00003 81E0000381E0000700F0000700F0000600F0000E0078000FFFF8000FFFF8001FFFFC001C003C00 18003C0038001E0038001E0070001F0070000F0070000F00E0000780191D7F9C1C>65 DI<003FC000FFF003FFF007C0700F80001E00003E00003C 0000780000780000780000F00000F00000F00000F00000F00000F00000F00000F00000F0000078 00007800007800003C00003E00001E00000F800807C07803FFF800FFF0003F80151F7D9D1B>I< FFFC00FFFF00FFFF80F00FE0F003F0F000F0F00078F00078F0003CF0003CF0001CF0001EF0001E F0001EF0001EF0001EF0001EF0001EF0001EF0003CF0003CF0007CF00078F000F0F001F0F007E0 FFFFC0FFFF00FFFC00171D7C9C1E>III<003F8001FFF003FFF807C0F80F00181E00003E00003C0000780000780000 780000F00000F00000F00000F00000F00000F00000F007F8F007F8F007F8780038780038780038 3C00383E00381E00380F003807C07803FFF801FFF0003F80151F7D9D1C>III76 DII<003F000001FFE00003FFF00007C0F800 0F807C001E001E003E001F003C000F00780007807800078078000780F00003C0F00003C0F00003 C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C078000780780007807C000F803C00 0F003E001F001F003E000F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>II82 D<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F00000F800007C00007F00 003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C00003C00003C00003C0C007 80E00780FC1F007FFE001FFC0007F000121F7E9D17>IIII89 D<0FC03FF07FF87038401C001C001C00FC0F FC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 DI<07E00FF81FFC3C1C70047000E000E000E000E000E000E00070 0070043C1C1FFC0FF807E00E127E9112>I<000E000E000E000E000E000E000E000E000E000E00 0E0F8E1FEE3FFE7C3E700E700EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D 7E9C15>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF8 07E00E127E9112>I<00FC01FC03FC07000E000E000E000E000E000E000E00FFE0FFE0FFE00E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E1D809C0D>I<03C3C00F FFC01FFFC01C3800381C00381C00381C00381C00381C001C38001FF8001FF0003BC00038000038 00001FFC001FFF003FFF80700780E001C0E001C0E001C0F003C07C0F803FFF001FFE0007F80012 1B7F9115>III107 DIII<03F0000FFC001FFE003C0F00780780700380E0 01C0E001C0E001C0E001C0E001C0F003C07003807807803C0F001FFE000FFC0003F00012127F91 15>II114 D<1FC03FF07FF0F030E000 E000F0007F003FC01FE000F0003800388038F078FFF07FE01FC00D127F9110>I<1C001C001C00 1C001C001C00FFE0FFE0FFE01C001C001C001C001C001C001C001C001C001C001C001C201FF00F F007C00C187F970F>I118 D121 D E /Fn 15 118 df<387CFEFEFE7C3800000000387CFEFEFE7C3807127D910D>58 D68 D73 D<0FF8001FFE003E1F803E 07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07C0FC07C0FC07C0FC0FC07E1FC03F FBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F007C1F007C0E00FC0000FC0000FC00 00FC0000FC0000FC00007C00007E01803E03801F07000FFE0003F80011127E9115>99 D<01FC000FFF001F0F803E07C07C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC0000FC00007C00 007E00603E00C01F81C00FFF0001FC0013127F9116>101 D<1E003F003F007F003F003F001E00 00000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F 001F00FFE0FFE00B1E7F9D0E>105 D108 DII<01FC000FFF801F07C03E03E07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01 F8FC01F87C01F07C01F03E03E01F07C00FFF8001FC0015127F9118>II<1FD83F F87038E018E018F000FF807FE07FF01FF807FC007CC01CC01CE01CF038FFF0CFC00E127E9113> 115 D<030003000300070007000F000F003F00FFFCFFFC1F001F001F001F001F001F001F001F00 1F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913>II E /Fo 6 79 df<000E0000001F0000001F0000003F8000003F8000003F80000067 C0000067C00000C3E00000C3E00001C3F0000181F0000381F80003FFF80003FFF80006007C0006 007C000C003E000C003E00FF00FFE0FF00FFE01B157F941F>65 D<007F8103FFF70FE03F1F000F 3E00077C00077C0003780003F80000F80000F80000F80000F800007800037C00037C00033E0006 1F000C0FE03803FFF0007FC018157E941E>67 D69 D<007F810003FFF7000FE03F001F000F003E0007007C0007007C00 030078000300F8000000F8000000F8000000F8000000F803FFE07803FFE07C001F007C001F003E 001F001F001F000FE01F0003FFFF00007FE3001B157E9420>71 D I78 D E /Fp 78 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8 F870000000000070F8F8F870051C779B18>33 D<03C00007E0000FF0001E78001C38001C38001C 38001C38001C7BF01CF3F01EE3F00FE7800FC7000F87000F0F001F0E003F8E007B9E0073DC00F1 DC00E1FC00E0F800E07870F0F87071FE707FFFF03FCFE01F03C0141C7F9B18>38 D<007000F001F003C007800F001E001C003C003800780070007000F000E000E000E000E000E000 E000E000E000F00070007000780038003C001C001E000F00078003C001F000F000700C24799F18 >40 D<6000F00078003C001E000F000780038003C001C001E000E000E000F00070007000700070 007000700070007000F000E000E001E001C003C0038007800F001E003C007800F00060000C247C 9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007F00007F0001FFC 007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718>I<00600000F00000 F00000F00000F00000F00000F00000F0007FFFC0FFFFE0FFFFE07FFFC000F00000F00000F00000 F00000F00000F00000F00000600013147E9718>I<3C7E7F7F7F3F0F0E1E7CF870080C788518>I< 7FFF00FFFF80FFFF807FFF0011047D8F18>I<78FCFCFCFC780606778518>I<000300000780000F 80000F80000F00001F00001F00003E00003E00003C00007C00007C0000F80000F80000F00001F0 0001F00003E00003E00007C00007C0000780000F80000F80001F00001F00001E00003E00003E00 007C00007C0000780000F80000F80000F0000060000011247D9F18>I<01F00007FC000FFE001F 1F001C07003803807803C07001C07001C0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E0 00E0E000E0F001E07001C07001C07803C03803801C07001F1F000FFE0007FC0001F000131C7E9B 18>I<01800380078007800F803F80FF80FB806380038003800380038003800380038003800380 03800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>I<07F8001FFE003FFF007C0F80 7003C0F001E0F000E0F000E0F000E00000E00000E00001E00001C00003C0000780000F00001E00 003C0000780000F00003E00007C0000F80001E00E03C00E07FFFE0FFFFE07FFFE0131C7E9B18> I<07F8001FFE003FFF007C0F807803C07801C03001C00001C00003C0000780000F8003FF0003FE 0003FF00000F800003C00001C00001E00000E00000E0F000E0F001E0F001C0F003C07C0F803FFF 001FFE0007F800131C7E9B18>I<001F00003F00007F0000770000F70001E70001C70003C70007 87000707000E07001E07003C0700380700780700F00700FFFFF8FFFFF8FFFFF800070000070000 0700000700000700000700007FF000FFF8007FF0151C7F9B18>I<1FFF803FFF803FFF80380000 3800003800003800003800003800003800003800003BFC003FFE003FFF003E07801803C00001E0 0000E00000E06000E0F000E0F001E0E001C0F003C07C0F803FFF001FFE0007F800131C7E9B18> I<007E0001FF0007FF800FC3C01F03C03C03C0380180780000700000700000F3F800EFFE00FFFF 00FE0F80F803C0F001C0F001E0F000E0F000E0F000E07000E07001E07801C03C03C01E0F800FFF 0007FE0003F800131C7E9B18>II<03F8000FFC001FFE00 3E0F00780780F003C0E001C0E001C0E001E0E001E0E001E0F001E07803E03E0FE01FFFE00FFEE0 03F8E00001E00001C00001C00003C0300380780780780F00783E003FFC001FF8000FE000131C7E 9B18>57 D<78FCFCFCFC78000000000000000078FCFCFCFC780614779318>I<3C7E7E7E7E3C00 00000000000000387C7E7E7E3E0E1E3C78F060071A789318>I<000300000F80001F80003F0000 FE0001FC0003F00007E0001FC0003F80007E0000FC0000FC00007E00003F80001FC00007E00003 F00001FC0000FE00003F00001F80000F8000030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE0 000000000000000000000000FFFFE0FFFFE0FFFFE07FFFC0130C7E9318>I<00700000F80000F8 0000D80000D80001DC0001DC0001DC00018C00038E00038E00038E00038E000306000707000707 000707000707000FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C 7F9B18>65 DI<01FCE003FEE007FFE00F07E01E03E03C01E07800 E07000E07000E0F00000E00000E00000E00000E00000E00000E00000E00000E00000F000007000 E07000E07800E03C01E01E01C00F07C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007F FF001C0F801C03C01C03C01C01E01C00E01C00E01C00F01C00701C00701C00701C00701C00701C 00701C00701C00701C00F01C00E01C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF80014 1C7F9B18>III<01F9C007FFC00F FFC01F0FC01C03C03C03C07801C07001C07001C0F00000E00000E00000E00000E00000E00000E0 0FF0E01FF0E00FF0F001C07001C07003C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C014 1C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C0 1C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C0 1C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F 07F01C03C01C07801C07001C0F001C1E001C3C001C38001C78001CF0001DF0001DF8001FF8001F BC001F1C001E1E001E0E001C0F001C07001C07801C03801C03C01C01C07F03F0FF87F87F03F015 1C7F9B18>75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00 700E00707FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF8 7F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C0 1C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0 151C7F9B18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380F007807007 00780F007FFF003FFE000FF800111C7D9B18>II<0FF8003FFE00 7FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00 001C00001E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03 801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03 801C03801C03801C03801C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07 F3801FFF803FFF807C1F80700780F00380E00380E00380E00000F000007800003F00001FF0000F FE0001FF00001F800003C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FF FF00E7FC00131C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E07038007000007000 007000007000007000007000007000007000007000007000007000007000007000007000007000 00700000700000700007FF0007FF0007FF00151C7F9B18>IIII<7F8FE07F9FE07F8FE00E07000F07 00070E00078E00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC 00039E00038E00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>II<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00 001C00003C0000780000700000F00001E00001C00003C0000780000700000F00001E00E01C00E0 3C00E07800E07000E0FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF 0011047D7F18>95 D<1FE0003FF8007FFC00783E00300F0000070000070001FF000FFF003FFF00 7F0700780700F00700E00700E00700F00F00783F007FFFF03FFBF00FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000EFF800FFFC00FE3E00F80 F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFF C00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300780000700000F00000E0 0000E00000E00000E00000F000007000007801C03C01C03F07C01FFF8007FF0001FC0012147D93 18>I<001F80003F80001F8000038000038000038000038000038003F3800FFB801FFF803E1F80 780F80700780F00780E00380E00380E00380E00380E00380E00780F00780700780780F803E3F80 1FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F00780780700380F003C0E001 C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01FFF8007FF0001FC001214 7D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C0007FFFC0FFFFC0FFFFC001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F303C0F00380700380700 3807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00001FFF001FFFC03FFFE0 7801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF8003FE00151F7F9318> I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000EFF800FFFC00FE1E00F80 E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3 FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C00003800000000000000000000000 00007FC000FFC0007FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C00 3800000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C003C6038F078FFF07FE03FC00E277E9C18>II<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFFFC001F1F1C 001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C 1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E3F00FEFF80 7FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001 C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE 0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00380E00380E 00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00000E00000E 00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<7F87E0FF9FF87FBFF8 03FC7803F03003E00003C00003C00003C000038000038000038000038000038000038000038000 0380007FFE00FFFF007FFE0015147F9318>114 D<0FF7003FFF007FFF00F81F00E00700E00700 F007007C00007FF0001FFC0007FE00001F00600780E00380F00380F00780FC0F00FFFF00FFFE00 E7F80011147D9318>I<0180000380000380000380000380007FFFC0FFFFC0FFFFC00380000380 000380000380000380000380000380000380000380000380400380E00380E00381E003C3C001FF C000FF80007E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F07E007FFFC03FFFE01FCFC17148093 18>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E0380070700070700070700038E00038E00 038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318>II<7F8FF07F9FF07F8FF0070700078E00039E0001 DC0001F80000F80000700000F00000F80001DC00039E00038E000707000F07807F8FF0FF8FF87F 8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E03800E0380070380070700070700038700 038600038E0001CE0001CE0000CC0000CC0000DC00007800007800007800007000007000007000 00F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001 E07003C0700780000F00001E00007C0000F80001F00003E0000780000F00701E00703C00707800 70FFFFF0FFFFF0FFFFF014147F9318>I<0007E0003FE0007FE000FC0000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00001E0007FC000FF8000FF80007FC00001 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000FC0000 7FE0003FE00007E013247E9F18>I<7C0000FF8000FFC00007E00000E00000E00000E00000E000 00E00000E00000E00000E00000E00000E00000E00000F000007FC0003FE0003FE0007FC000F000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00007E000FFC000 FF80007C000013247E9F18>125 D E /Fq 18 122 df<7FF0FFE0FFE00C037D8A10>45 D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00F00E 00F00E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F00070007 0007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01E E01EE03CE03CE038607071E03FC01F0010207B9F15>I<007C0001FF000783000F01801E01803C 01803C0300780E007FFC007FE000F00000F00000F00000F000007000007002007807003C1E001F F80007E00011147C9315>101 D<0000F80001FC0003BC00033C000718000700000700000E0000 0E00000E00000E00000E0001FFE001FFE0001C00001C00001C0000380000380000380000380000 380000700000700000700000700000700000700000E00000E00000E00000E00001C00001C00001 C0000180003380007B8000F300007E00003C00001629829F0E>I<003C6000FF7001E3E00381E0 0701E00F01E00F01C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C0F001C1F00 1E3F000FFE0003CE00000E00000E00001C00001C00301C00783800F0F0007FE0003F8000141D7E 9315>I<006000F000F000E000000000000000000000000000000F001F80318031C063806380C3 800700070007000E000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>105 D<01E0000FE0000FC00001C00001C00001C0000380000380000380000380000700000700000703 C00707E00E0C600E10E00E21E00E61E01CC1C01F80001F00001FC00039E0003870003870003838 607070C07070C07070C0703180E03F00601E0013207D9F15>107 D<03C01FC01F800380038003 8007000700070007000E000E000E000E001C001C001C001C003800380038003800700070007000 7180E300E300E300E6007E003C000A207C9F0C>I<1E07C0F8003F1FE1FC0033B8730E0063E076 070063C03C07006380380700C780780E000700700E000700700E000700700E000E00E01C000E00 E01C000E00E01C000E00E038601C01C038C01C01C038C01C01C071801C01C031803803803F0018 01801E0023147D9325>I<1E07C03F1FE033B87063E07063C038638038C7807007007007007007 00700E00E00E00E00E00E00E01C31C01C61C01C61C038C1C018C3801F81800F018147D931A>I< 007C0001FF000383800F01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0 F00780F00700700F00701E003838001FF00007C00013147C9317>I<1E0F003F3F8033F1C063C1 C063C3C06383C0C783800700000700000700000E00000E00000E00000E00001C00001C00001C00 001C000038000018000012147D9313>114 D<00FC03FE07070E0F0E0F0E0E1E000F800FF007F8 03FC003E001E701EF01CF01CE03860703FE01F8010147D9313>I<018001C00380038003800380 07000700FFF0FFF00E000E000E000E001C001C001C001C0038003800380038307060706070C071 803F001E000C1C7C9B0F>I<0F01C01F83C03183E031C1E06380E06380E0C380C00700C00700C0 0700C00E01800E01800E01800E03000E03000E06000E06000F0C0007F80001E00013147D9315> 118 D<0787800FCFC018F8E03070E06071E06071E0C0E1C000E00000E00000E00001C00001C000 01C00071C060F380C0F380C0E38180C7C3007CFE00387C0013147D9315>120 D<0F00601F80703180E031C0E06380E06380E0C381C00701C00701C00701C00E03800E03800E03 800E03800E07000E07000E0F000E1F0007FE0003EE00000E00000E00001C00781C007838007070 0060E0003FC0001F0000141D7D9316>I E /Fr 48 122 df45 DI<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0FFFFFFFF FFFF10227CA118>49 D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E001E06001 E02001E00001E00001E00001E00003C00003C0000780000780000F00001E00003C0000780000F0 0001E00001C0000380000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118 >I<01F00007FC001FFF003E0F003807807003C02003C02003C00003C00003C00003C000078000 0780000F00001E0003FC0003F80003FE00000F000007800003C00003C00001E00001E00001E000 01E00001E08001E0C003C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F00 001F00002F00002F00006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F00 0E0F001E0F003C0F003C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00 000F00000F00000F00000F00000F00000F0015217FA018>I<3FFF803FFF803FFF803C00003C00 003C00003C00003C00003C00003C00003C00003C00003CF8003FFE003FFF003F0F803E07803C03 C03803C00001E00001E00001E00001E00001E00001E00001E04003C04003C0E003C07007807C1F 003FFE000FFC0003F00013227EA018>I<007E0001FF0003FF0007C1000F00001E00001E00003C 00003C000078000078000078F800F3FE00F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F0 01E0F001E0F001E0F001E07001E07001E07801E07803C03803C03C03801C07801F0F000FFE0007 FC0001F00013237EA118>I I<01F00007FC000FFE001E0F003C07803C07807803C07803C07803C07803C07803C03803803C07 801E0F000F1E0007FC0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F001E0F001 E0F001E0F001E07803C07803C03C07803E0F801FFF0007FC0001F00013237EA118>I<01F00007 FC000FFE001E0F003C07003807807803807803C0F003C0F001C0F001E0F001E0F001E0F001E0F0 01E0F001E07003E07803E07803E03C07E03E1FE01FFDE00FF9E003E3C00003C00003C000038000 0780000700000F00101E00183C003FF8001FF00007C00013237EA118>I<001F0000001F000000 3F8000003F8000003B8000007BC0000073C0000071C00000F1E00000F1E00000E0E00001E0F000 01E0F00001C0F00003C0780003C078000380780007803C0007803C0007003C000F001E000F001E 000FFFFE001FFFFF001FFFFF001C000F003C0007803C00078038000780780003C0780003C07000 03C0F00001E0F00001E0E00001E01B237EA220>65 DI<000FF000007FFC0000FFFF0001F01F0003C007000780 02000F0000001E0000003E0000003C0000003C000000780000007800000078000000F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F0000000F00000007800000078000000 780000003C0000003C0000003E0000001E0000000F0000800780018003C0038001F00F8000FFFF 00007FFC00000FF00019257DA31F>III76 DII<001FC000007FF00001FFFC0003 F07E0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000F0 70000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F0000078F00000 78780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007C00F800F8007C0 1F0003F07E0001FFFC00007FF000001FC0001D257DA324>II82 D<00FF0003FFC007FFF00F83F01E00F03C00203C00007800 007800007800007800007800007C00003C00003F00001FC0000FFC0007FF0001FF80003FC00007 E00001F00000F00000F8000078000078000078000078000078400078E000F0F000F0F801E07F07 C03FFF800FFF0001FC0015257EA31B>I85 D<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3F FF7F0F780FF00FF00FF00FF80F7C3F7FFF3FFF1F8F10167E9517>97 DI<01FC0007FF000FFF801F07803C0180 780000780000700000F00000F00000F00000F00000F00000F000007800007800007800403C00C0 1F07C00FFFC007FF8001FC0012167E9516>I<0003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFF C00FFBC003E3C012237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0FF FFC0FFFFC0FFFFC0F00000F00000F000007000007800007800003C00801F07800FFF8007FF0001 F80012167E9516>I<003F00FF01FF03C1038007800780078007800780078007800780FFF8FFF8 FFF807800780078007800780078007800780078007800780078007800780078007800780078007 80102380A20F>I<01F0F807FFF80FFFF81F1F801E0F003C07803C07803C07803C07803C07801E 0F001F1F001FFE001FFC0039F0003800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F8 00F8F00078F00078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>III<00 F000F000F000F000000000000000000000000000000000000000F000F000F000F000F000F000F0 00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000 F080F0E3E0FFE07FC01F000C2D83A20D>IIIII<01FC0007FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F00078 F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F9518> II114 D<07F01FFE3FFF3C0F7803780078007C003F003FF01FF80FFC01FE001F000F000F400FE00FF81E FFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F000F00 0F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12>IIII<7801F07C01E03E03C01E07C00F0780078F 0007DE0003FC0001FC0000F80000700000F80001FC0003DC00039E00078F000F07801E07801E03 C03C01E07800F0F800F81516809516>II E /Fs 61 124 df<003F0F0000FFBF8003C3F3C00703E3C00703C1800E01C0000E01C000 0E01C0000E01C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01C0000E01C0000E01C0 000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01 C0007F87FC007F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C00703C00E01800E0000 0E00000E00000E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D809C17>I<003FC000FF C003C3C00703C00701C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0FFFFC00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCF F87FCFF8151D809C17>I<003F03F00000FFCFF80003C0FC1C000701F03C000701F03C000E00E0 18000E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC00FFFFFFFC000E00 E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E 00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF807FC7FCFF80 211D809C23>I<7070F8F8FCFCFCFC7C7C0C0C0C0C0C0C181818183030606040400E0D7F9C15> 34 D<00C00180030006000E000C001C0018003800300030007000700060006000E000E000E000 E000E000E000E000E000E000E000E000E000600060007000700030003000380018001C000C000E 0006000300018000C00A2A7D9E10>40 DI<70F0F8F87818181830 30706040050D7D840C>44 DI<70F8F8F87005057D840C>I<030007 003F00FF00C7000700070007000700070007000700070007000700070007000700070007000700 07000700070007000700FFF8FFF80D1C7C9B15>49 D<07E01FF03838301C781E781E781E381E00 1E003C0038007007E007E00038001C001E000E000F000F700FF80FF80FF80EF01E601C38381FF0 07C0101D7E9B15>51 D<001C00001C00003C00007C00007C0000DC0001DC00019C00039C00071C 00061C000E1C000C1C00181C00381C00301C00601C00E01C00FFFFC0FFFFC0001C00001C00001C 00001C00001C00001C0001FFC001FFC0121C7F9B15>I<00F003FC070C0E0E1C1E381E380C7800 70007000F3F0F7F8FC1CF81EF80EF00EF00FF00FF00FF00FF00F700F700F700E380E381C1C380F F003E0101D7E9B15>54 D<70F8F8F870000000000000000070F8F8F87005127D910C>58 D<70F8F8F870000000000000000070F0F8F8781818183030706040051A7D910C>I<0006000000 0F0000000F0000000F0000001F8000001F8000001F8000001F80000033C0000033C0000033C000 0061E0000061E0000061E00000C0F00000C0F00000C0F000018078000180780001FFF80003FFFC 0003003C0003003C0006001E0006001E0006001E001F001F00FFC0FFF0FFC0FFF01C1D7F9C1F> 65 D<001F808000FFE18003F0338007801B800F000F801E0007801C0003803C00038078000380 7800018070000180F0000180F0000000F0000000F0000000F0000000F0000000F0000000F00000 007000018078000180780001803C0001801C0003001E0003000F00060007800C0003F0380000FF F000001F8000191E7E9C1E>67 D69 DI<001F808000FFE18003F0338007801B800F000F801E0007801C0003803C0003807800038078 00018070000180F0000180F0000000F0000000F0000000F0000000F0000000F000FFF0F000FFF0 7000078078000780780007803C0007801C0007801E0007800F00078007800F8003F0398000FFF0 80001FC0001C1E7E9C21>III77 DI<003F800000FFE0 0003E0F80007803C000E000E001E000F003C00078038000380780003C0780003C0700001C0F000 01E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0780003C0780003C078 0003C03C0007803C0007801E000F000F001E0007803C0003E0F80000FFE000003F80001B1E7E9C 20>II82 D<07E0801FF9803C1F80700780700380E0 0380E00180E00180E00180F00000F000007C00007FC0003FF8001FFE0007FF0000FF80000F8000 03C00003C00001C0C001C0C001C0C001C0E00180E00380F00300FC0E00CFFC0083F800121E7E9C 17>I<7FFFFFC07FFFFFC0780F03C0700F01C0600F00C0E00F00E0C00F0060C00F0060C00F0060 C00F0060000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F000003FFFC0003FFFC001B1C 7F9B1E>II87 D89 D<08081818303060606060C0C0C0C0C0C0F8F8FCFCFCFC7C7C38380E0D 7B9C15>92 D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE001FFE003E0E00780E00 F00E00F00E60F00E60F01E60783E603FFFC01F878013127F9115>97 DI<03F00FF81E3C383C78187000F000F000F000F000F000F000780078063C061E0C0F F803E00F127F9112>I<001F80001F800003800003800003800003800003800003800003800003 8000038003E3800FFB801E0F80380780780380700380F00380F00380F00380F00380F00380F003 807003807803803807801E1F800FFBF007E3F0141D7F9C17>I<03E00FF01C38381C781E700EFF FEFFFEF000F000F000F000700078063C061E0C0FF803E00F127F9112>I<007801FC039E071E0E 0C0E000E000E000E000E000E00FFE0FFE00E000E000E000E000E000E000E000E000E000E000E00 0E000E000E007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3DC0381C00781E00781E0078 1E00781E00381C003C3C003FF00037E0007000007000003000003FFC001FFF003FFF80700780E0 01C0E001C0E001C0E001C07003803C0F001FFE0007F800121C7F9215>II<18003C007C003C001800000000000000000000000000FC00FC001C001C001C00 1C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<01C003E003E0 03E001C00000000000000000000000000FE00FE000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E000E000E000E000E060E0F1C0F1C07F803E000B25839C0D>III< FC7E07E000FDFF9FF8001F83B838001E01E01C001E01E01C001C01C01C001C01C01C001C01C01C 001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C0 1C00FF8FF8FF80FF8FF8FF8021127F9124>II<03F0000FFC001E1E00380700780780700380F003C0F003C0F003C0F003C0F003C0F003C070 03807807803807001E1E000FFC0003F00012127F9115>II<03E1800FF9801E1F 803C0780780780780380F00380F00380F00380F00380F00380F003807803807807803C07801E1F 800FFB8007E380000380000380000380000380000380000380001FF0001FF0141A7F9116>II<1F903FF07070E030E030E030F8007F803FE00FF000F8C038C038E038E038F070DFE08FC0 0D127F9110>I<0C000C000C000C000C001C001C003C00FFE0FFE01C001C001C001C001C001C00 1C001C001C301C301C301C301C301E600FC007800C1A7F9910>IIII<7F8FF07F8FF0 0F0780070600038E0001DC0001D80000F00000700000780000F80001DC00038E00030E00060700 0F0380FF8FF8FF8FF81512809116>II<7FFC7FFC7838707060F060E061C063C0 0380070C0F0C0E0C1C1C3C1838187078FFF8FFF80E127F9112>II E /Ft 63 122 df<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F01420809F18>13 D<007000E001C00380078007000E001E001E003C003C003C0078007800780078007000F000F000 F000F000F000F000F000F000F000F000F000F000700078007800780078003C003C003C001E001E 000E0007000780038001C000E000700C2E7EA112>40 DI<00060000000600000006000000060000000600000006000000060000000600000006 0000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF00006000000 060000000600000006000000060000000600000006000000060000000600000006000000060000 000600000006000000060000000600001C207D9A23>43 D<787878781830306060E0050A7D830D >III<03F00007F8001FFE001E1E003C0F 00380700780780780780700380F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003 C0F003C0F003C0F003C0F003C0F003C07807807807807807803807003C0F001E1E001FFE000FFC 0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0FFFEFFFEFFFE0F1F7C9E 17>I<07F0000FFC001FFE00383F00700F00600780E00780E003C04003C04003C00003C00003C0 0003C0000780000780000F00000E00001C00003C0000780000E00001C0000380000700000E0000 1C0000380000700000FFFFC0FFFFC0FFFFC0121F7E9E17>I<03F0000FFC001FFE003C1F00780F 00300780200780000780000780000780000F00000F00003E0003FC0003F80003FC00001E00000F 000007800003800003C00003C00003C00003C08003C0C003C0C00780700F807C1F003FFE000FFC 0003F00012207E9E17>I<003E00003E00005E00005E0000DE0001DE00019E00039E00039E0007 9E00071E000F1E000E1E001E1E003C1E003C1E00781E00781E00F01E00FFFFF0FFFFF0FFFFF000 1E00001E00001E00001E00001E00001E00001E00001E00141E7F9D17>I58 D61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1E00000E1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007803C000780780007C0 780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C0000780000 780000780000F00000F00000F00000F00000F00000F00000F00000F00000780000780000780000 7C00003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FF F801FFFE03E03E07800E0F00001E00003E00003C00007C0000780000780000780000F00000F000 00F00000F00000F00000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E00 1E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E>III<003C003C003C003C003C003C003C003C003C003C003C003C003C00 3C003C003C003C003C003C003C003C003C003C003C003C003C003C003CC078E078FFF07FE00FC0 0E217E9F15>I76 DII<003F000000FFC00003FFF00007E1F8000F807C001F003E001E001E003C000F003C00 0F00780007807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F0 0003C0F00003C0F00003C0F80007C07800078078000780780007803C000F003C000F001E001E00 1F003E000F807C0007E1F80003FFF00000FFC000003F00001A227DA021>II82 D<01FC0007FF800FFFC01F03C03C00C03C00007800007800007800007800007800007C00 003C00003F00001FE0000FFC0007FE0001FF00003F800007C00003C00003E00001E00001E00001 E00001E00001E00001C0C003C0F007C0FC0F807FFF001FFE0003F80013227EA019>IIIII<780007807C000F 003E001F001E001E000F003C000F807C000780780003C0F00003E1F00001F1E00000F3C000007F C000007F8000003F0000001F0000001E0000003F0000007F8000007FC00000F3C00001F1E00001 E0F00003C0F80007C0780007803C000F003E001F001E001E000F003C000F807C0007C0780003C0 F00003E01B207F9F1E>II91 D93 D<07E03FF87FFC701E401F000F000F000F003F07FF1F FF7E0FF80FF00FF00FF00FF83F7FFF3FEF1F8F10147E9316>97 DI<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F000 780078003C013E0F1FFF0FFE03F010147E9314>I<000780000780000780000780000780000780 00078000078000078000078000078000078007C7800FF7801FFF803E1F807C0780780780F80780 F00780F00780F00780F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C780 11207E9F17>I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F000 00F00000F000007000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01FE 03FE078007000F000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F20809F0E>I<03E0F00FFFF01FFFF03E3E 003C1E00780F00780F00780F00780F00780F003C1E003E3E001FFC003FF80033E0003000003800 003FFE003FFF801FFFC03FFFE07803F0F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC 00141E7F9317>I I I<01E001E001E001E00000000000000000000000000000000001E001E001E001E001E001E001E0 01E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E0C3C0FF C0FF803F000B29839F0C>IIIII<01F80007FE001FFF803F0FC03C03C07801E07801E0F000F0 F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F80014147F 9317>II114 D<07F01FFC3FFC780C78 00780078007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E 001E001E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E001E001E001E00 1E001E001E201FF00FF007C00C1A7F9910>IIII<7801E07C03C03E07801E0F000F0F 00079E0003FC0003F80001F80000F00001F00001F80003FC00079E000F0F000E0F001E07803C03 C07801E0F801F01414809315>II E /Fu 13 86 df<3078F8787005057C840D>46 D<000C001C00FC0FF80F380038003800380038 00700070007000700070007000E000E000E000E000E000E001C001C001C001C001C001C003C07F FEFFFE0F1E7C9D17>49 D<0000180000003800000038000000780000007C000000FC000000FC00 0001BC000001BC0000033C0000033E0000061E0000061E00000C1E00000C1E0000181E0000181F 0000300F0000300F0000600F00007FFF0000FFFF0000C00F000180078001800780030007800300 078006000780060007801F0007C0FFC07FFCFFC07FFC1E207E9F22>65 D<0003F808000FFE1800 3F0738007801F801F000F803C000F007C00070078000700F0000701F0000701E0000703E000060 3C0000607C0000007C0000007C0000007C000000F8000000F8000000F8000000F8000000F80000 C0780000C0780001807C0001803C0003003C0003001E0006001E000C000F80180007E0700001FF C000007F00001D217B9F21>67 D<07FFFFF807FFFFF8003C00F8003C0078007800380078003800 780038007800380078003800780C3000F0183000F0180000F0180000F0380000FFF80000FFF800 01E0700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C0006003C000 E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>69 D<07FFE007FFE0003C0000 3C0000780000780000780000780000780000780000F00000F00000F00000F00000F00000F00001 E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003C00007C000FF FC00FFFC00131F7F9E10>73 D<07FC0000FFC007FC0001FFC0003E0001F800003E00037800006E 0003F000006E0006F000006E0006F000006E000CF0000067000CF00000670018F00000C70019E0 0000C70031E00000C70031E00000C70061E00000C38061E00000C380C1E000018380C3C0000183 8183C00001838183C0000181C303C0000181C303C0000181C603C0000301C60780000301CC0780 000301CC0780000300F80780000300F80780000700F00780000F80F00F80007FF0E0FFF800FFF0 E1FFF8002A1F7E9E2A>77 D<07FC03FFC007FC03FFC0003E007C00003E003800006F003000006F 003000006F003000006780300000678030000063C0300000C3C0600000C3C0600000C1E0600000 C1E0600000C0F0600000C0F060000180F0C000018078C000018078C00001803CC00001803CC000 01803CC00003001F800003001F800003000F800003000F800003000F800007000780000F800700 007FF0030000FFF0030000221F7E9E22>I<0003F800001FFE00003C1F0000F0078001E003C003 C001E0078001E00F8000F00F0000F01F0000F01E0000F83E0000F83C0000F87C0000F87C0000F8 7C0000F87C0000F8F80001F0F80001F0F80001F0F80001F0F80003E0780003E0780007C07C0007 C07C000F803C000F003E001E001E003C000F00780007C1F00003FFC00000FE00001D217B9F23> I<07FFFF0007FFFFC0003C03E0003C01F0007800F0007800F8007800F8007800F8007800F80078 00F800F001F000F001F000F001E000F003C000F00F8000FFFE0001FFF80001E0000001E0000001 E0000001E0000001E0000003C0000003C0000003C0000003C0000003C0000003C0000007C00000 7FFC0000FFFC00001D1F7E9E1F>I<003F04007FCC01E0FC03807C03003C0700380600180E0018 0E00180E00180E00000F00000F00000FE00007FE0003FF8001FFC0007FE00007E00001E00000E0 0000F00000F06000E06000E06000E06000E07001C0700180F80380FE0F00C7FC0081F80016217D 9F19>83 D<1FFFFFF81FFFFFF81E03C0F83803C038380780383007803870078018600780186007 803860078030C00F0030000F0000000F0000000F0000000F0000000F0000001E0000001E000000 1E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C0000003C0000 007C00001FFFF0003FFFF0001D1F7B9E21>II E /Fv 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F801F 000B107F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF80 FF8009107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FFC0 FFC00B107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F80 1F000B107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001FE0 1FE00B107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F80 3F000B107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F80 1F000B107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C00 1C0008000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060C0 3F801F000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C07180 7F003E000B107F8F0F>I E /Fw 30 122 df<000FE0FE00007FF3FF0000F87F8F8001E0FF1F80 03E0FE1F8007C0FE1F8007C07E0F0007C03E000007C03E000007C03E000007C03E000007C03E00 00FFFFFFF000FFFFFFF00007C03E000007C03E000007C03E000007C03E000007C03E000007C03E 000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007C0 3E000007C03E000007C03E00003FF9FFE0003FF9FFE0002120809F1E>11 D45 D<07F8001FFE00381F80780F80FC0FC0FC0FC0FC0FC078 0FC0301F80001F00003E00007C0000700000E00000E00000C00000C00000C00000C00000C00000 C00000000000000000000000000001C00007F00007F00007F00007F00007F00001C00012207D9F 19>63 D<0000E000000000E000000001F000000001F000000001F000000003F800000003F80000 0007FC00000007FC0000000FFE0000000CFE0000000CFE000000187F000000187F000000307F80 0000303F800000703FC00000601FC00000601FC00000C01FE00000C00FE00001FFFFF00001FFFF F000018007F000030003F800030003F800060003FC00060001FC000E0001FE00FFE01FFFE0FFE0 1FFFE0231F7E9E28>65 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E 3F00000E3F00000E7F0000067E0000067E000006FE000000FE000000FE000000FE000000FE0000 00FE000000FE0000007E0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E0 003803F0007000FE01C0003FFF800007FC001F1F7D9E26>67 D 70 D<03FC080FFF381E03F83C00F8780078780038F80038F80018FC0018FC0000FF0000FFF800 7FFF007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC0000FCC0007CC0007CC0007C E00078E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83 D<07FC001FFF003F0F803F07C03F 03E03F03E00C03E00003E001FFE00FFFE03F83E07E03E07C03E0F803E0F803E0F803E0FC07E07E 0DE03FF9FE07E07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC0000FC0000FC0000FC0000FC 00007C00007E00003E00603F00C01F81C007FF0001FC0013147E9317>I<0007F80007F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F801F8F80FFEF81F83F83E01F8 7E00F87C00F87C00F8FC00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00F87C00F87E00F83E01F8 1F07F80FFEFF03F8FF18207E9F1D>I<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00 F8FFFFF8FFFFF8FC0000FC0000FC00007C00007E00003E00183F00380F807007FFE000FF801514 7F9318>I<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800F FF0019FC001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C70003EF0001EF0001EF0 001E78003C78003C3F01F80FFFE001FF00171E7F931A>103 DI<1C003F007F007F007F003F001C00000000000000000000000000FF 00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE0 0B217EA00E>I107 DIII<01FF0007FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007E FC007EFC007EFC007EFC007E7C007C7C007C3E00F83E00F81F83F007FFC001FF0017147F931A> II<01F81807FE381F87783F01F83E01F87E00F87C00F8FC 00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00F87E00F87E00F83F01F81F87F80FFEF803F8F800 00F80000F80000F80000F80000F80000F80000F80007FF0007FF181D7E931C>II<0FE63FFE701E600EE006E006F800FFC07F F83FFC1FFE03FE001FC007C007E007F006F81EFFFCC7F010147E9315>I<018001800180038003 80038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F860F86 0F860F8607CC03F801F00F1D7F9C14>IIIIII E /Fx 44 122 df45 DI<007E0001FF8007FFE00FFFF01FC3F81F00F83E007C3E 007C3C003C7C003E7C003E78001E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8 001FF8001FF8001FF8001FF8001FF8001FF8001FF8001F78001E7C003E7C003E7C003E7C003E3E 007C3F00FC1F00F81FC3F80FFFF007FFE001FF80007E0018297EA71D>48 D<00180000380000F80007F800FFF800FFF800FFF800F8F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFF F07FFFF014287CA71D>I<00FE0003FF800FFFC01FFFE01F07F03C01F87800FC78007C70007CF0 003EF0003E60003E20003E00003E00003E00003E00007C00007C0000780000F80001F00001E000 03E00007C0000F80001F00003E00007C0000F80001F00003E00003C0000780000F00001E00003C 00007FFFFE7FFFFE7FFFFE7FFFFE17287EA71D>I<007E0003FF8007FFE00FFFF01F81F83E00F8 7C007C78007C30007C10007C00007C00007C0000780000F80000F80001F00007E000FFC000FF80 00FFC000FFF00001F80000FC00007C00003E00003E00001F00001F00001F00001F00001F80001F C0003EE0003EF0007E7C00FC3F01F81FFFF00FFFE003FFC000FE0018297EA71D>I<0007E0000F E0000BE0001BE0001BE0003BE0003BE00073E00073E000F3E000F3E001E3E001E3E003E3E003C3 E007C3E00F83E00F83E01F03E01F03E03E03E03E03E07C03E07C03E0F803E0FFFFFFFFFFFFFFFF FFFFFFFF0003E00003E00003E00003E00003E00003E00003E00003E00003E00003E018277EA61D >I<3FFFF83FFFF83FFFF83FFFF83E00003E00003E00003E00003E00003E00003E00003E00003E 00003E00003E3E003EFF803FFFE03FFFF03FC1F03F80F83F007C3E007C00007C00003E00003E00 003E00003E00003E00003E00003E20003E60007C70007CF800F87C01F87F07F03FFFE00FFFC007 FF0001FC0017287EA61D>I<000FE0003FF800FFF801FFF803F81807E0000F80000F80001F0000 3E00003E00003C00007C00007C02007C3FC0F8FFE0F9FFF0FBFFF8FF80FCFE007CFE003EFC003E FC003FFC001FF8001FF8001FF8001FF8001F78001F7C001F7C001F7C001E3C003E3E003E1E007C 1F00FC0FC1F807FFF003FFE001FF80007E0018297EA71D>II<007E0003FFC007 FFE00FFFF01F81F83F00FC3E007C7E007E7C003E7C003E7C003E7C003E7C003E3E007C3E007C1F 00F80FC3F007FFE003FFC003FFC00FFFF01F81F83F00FC3E007C7C003E7C003EF8001FF8001FF8 001FF8001FF8001FF8001F7C003E7C003E7E007E3F00FC1F81F80FFFF007FFE003FFC0007E0018 297EA71D>I<007E0001FF8007FFC00FFFE01F83F01F00F83E00787C007C7C003C78003EF8003E F8003EF8001EF8001FF8001FF8001FF8001FF8003FFC003F7C003F7C007F3E007F3F01FF1FFFDF 0FFF9F07FF1F03FC1E00403E00003E00003E00007C00007C0000F80000F80001F01803E01E0FC0 3FFF801FFF000FFE0003F80018297EA71D>I66 D<0003FE00000FFFC0003FFFF000FFFFF001FE03F003F8006007E0002007C000000F8000001F80 00001F0000003E0000003E0000007C0000007C0000007C0000007C000000F8000000F8000000F8 000000F8000000F8000000F8000000F8000000F8000000F8000000F80000007C0000007C000000 7C0000007C0000003E0000003E0000001F0000001F8000000F80000007C0000007E0000803F800 3801FE01F800FFFFF8003FFFF0000FFFC00003FE001D2C7CAA25>II73 D77 DI80 D<007FC00001FFF80007FFFE000FFFFF001FC07F003F000F007E0006007C0000007C 000000F8000000F8000000F8000000F8000000F8000000FC0000007E0000007F0000003F800000 1FF800000FFF000007FFE00003FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F 8000000FC0000007C0000007C0000007C0000007C0000007C0000007C000000F8060000F80F000 1F00FC003F00FF80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>83 DII<01FC000FFF003FFF803FFFC03C07E03003E00001F00001F00001F00001F00001 F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0F801F0F801F0F801F0FC03F07E0FF07FFF F03FFFF01FF9F00FE1F0141B7E9A1C>97 DI<007F8001FF E007FFF80FFFF81FC0783F00103E00007C00007C00007C0000F80000F80000F80000F80000F800 00F80000F800007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FFE0007F80151B 7E9A1A>I<00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00 007C00007C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C7C007C7C007CFC 007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C7E00FC3F00FC1F 83FC1FFFFC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFFC01F87E03F01E0 3E00F07C00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800007800007C0000 7C00003E00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<001FC0007FC000FF C001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007C00007C000FFFE 00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 00122A7FA912>I<00F80F0003FE7F800FFFFF801FFFFF801F07C0003E03E0003E03E0007C01F0 007C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0003FFF80003BFE 000038F8000078000000780000007C0000003FFF80003FFFF0001FFFF8001FFFFC003FFFFE007C 00FE007C003F00F8001F00F8001F00F8001F00F8001F00FC003F007E007E003F81FC001FFFF800 0FFFF00007FFE00000FF000019287E9A1D>III<00F800F800F800F800F8000000000000000000000000000000000000000000F800 F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F8 00F800F800F800F800F800F800F800F800F800F800F800F840F8E3F0FFF0FFE07FC01F800D3683 A90F>IIIII<007E0001FF8007FFE00FFFF01F 81F83F00FC3E007C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001F7C 003E7C003E7C003E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E00181B7E9A1D>II114 D<07F8001FFF003FFF807FFF 807C0780F80100F80000F80000F80000FC00007F00007FF0003FFC001FFE0007FF0000FF80001F C0000FC00007C00007C04007C0E007C0FC1F80FFFF807FFF001FFE0007F800121B7E9A16>I<07 C00007C00007C00007C00007C00007C00007C000FFFF80FFFF80FFFF80FFFF8007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C08007E38003FFC003FFC001FF0000FC0012227FA115>I II121 D E /Fy 10 118 df<000000007FFC00000E0000 000FFFFFC0001E0000007FFFFFF8003E000003FFFFFFFE007E00000FFFFFFFFF80FE00003FFFF8 007FC1FE0000FFFF80000FF3FE0003FFFC000003FFFE0007FFF0000001FFFE000FFFC00000007F FE001FFF800000003FFE003FFF000000001FFE007FFC000000000FFE00FFFC000000000FFE01FF F80000000007FE03FFF00000000003FE03FFE00000000003FE07FFE00000000001FE0FFFC00000 000001FE0FFFC00000000000FE1FFF800000000000FE1FFF8000000000007E3FFF800000000000 7E3FFF0000000000007E3FFF0000000000003E7FFF0000000000003E7FFF0000000000003E7FFF 0000000000003E7FFE000000000000007FFE00000000000000FFFE00000000000000FFFE000000 00000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000000000 00FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE 00000000000000FFFE00000000000000FFFE000000000000007FFE000000000000007FFE000000 000000007FFF000000000000007FFF0000000000003E7FFF0000000000003E3FFF000000000000 3E3FFF0000000000003E3FFF8000000000003E1FFF8000000000007E1FFF8000000000007C0FFF C000000000007C0FFFC00000000000FC07FFE00000000000FC03FFE00000000000F803FFF00000 000001F801FFF80000000003F000FFFC0000000003F0007FFE0000000007E0003FFF000000000F C0001FFF800000001F80000FFFC00000003F000007FFF0000000FE000003FFFC000001FC000000 FFFF80000FF80000003FFFF8007FE00000000FFFFFFFFFC000000003FFFFFFFF00000000007FFF FFFC00000000000FFFFFE00000000000007FFE00000047497AC754>67 D80 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFF F8000003FF003FFC000007FF800FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007 FF8001FFC00007FF8001FFC00003FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FF E00000000000FFE00000000000FFE00000000000FFE00000000000FFE000000000FFFFE0000000 7FFFFFE0000007FFFFFFE000003FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE0 000FFF0000FFE0001FFE0000FFE0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF800 00FFE000FFF00000FFE000FFF00000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000 FFF80003FFE0007FFC0007FFE0007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC 3FFFF003FFFFF01FFFF0007FFFC00FFFF00007FE0003FFF0342E7DAD38>97 D<00001FFF80000001FFFFF8000007FFFFFE00001FFFFFFF00007FF801FF8000FFE003FFC003FF C003FFC007FF8003FFC007FF0003FFC00FFE0003FFC01FFE0003FFC01FFC0001FF803FFC0000FF 003FFC00007E007FFC000000007FF8000000007FF800000000FFF800000000FFF800000000FFF8 00000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF8000000007FFC000000007FFC000000007FFC000000003FFC 000000003FFE000000001FFE000001F01FFF000001F00FFF000003F007FF800007E007FFC00007 C003FFE0000FC000FFF8003F80007FFF00FF00001FFFFFFC000007FFFFF0000001FFFFC0000000 1FFC00002C2E7CAD34>99 D<00FC0001FF0003FF8007FF800FFFC00FFFC00FFFE00FFFE00FFFE0 0FFFC00FFFC007FF8003FF8001FF0000FC00000000000000000000000000000000000000000000 000000000000000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497C C820>105 D<00FF8003FF000003FF000000FFFF801FFFE0001FFFE00000FFFF807FFFF8007FFF F80000FFFF81FFFFFC01FFFFFC0000FFFF83F01FFE03F01FFE0000FFFF87C007FF07C007FF0000 03FF8F0007FF8F0007FF800001FF9E0003FF9E0003FF800001FFBC0003FFBC0003FF800001FFF8 0001FFF80001FFC00001FFF80001FFF80001FFC00001FFF00001FFF00001FFC00001FFE00001FF E00001FFC00001FFE00001FFE00001FFC00001FFE00001FFE00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000FFFFFF80FFFFFF80FF FFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80 FFFFFF80FFFFFF80FFFFFF80592E7BAD62>109 D<00FF8007FE0000FFFF803FFFC000FFFF80FF FFF000FFFF81FFFFFC00FFFF87F01FFC00FFFF8FC00FFE0003FF9F0007FF0001FF9E0007FF0001 FFBC0007FF0001FFF80003FF8001FFF80003FF8001FFF00003FF8001FFE00003FF8001FFE00003 FF8001FFE00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0 0003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80 01FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80FFFFFF81 FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFF382E7BAD41>I<00 000FFE0000000001FFFFF000000007FFFFFC0000001FFFFFFF0000007FF803FFC00000FFE000FF E00001FF80003FF00003FF00001FF80007FE00000FFC000FFE00000FFE000FFC000007FE001FFC 000007FF003FFC000007FF803FF8000003FF803FF8000003FF807FF8000003FFC07FF8000003FF C07FF8000003FFC0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF800 0003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0 7FF8000003FFC07FF8000003FFC07FF8000003FFC07FF8000003FFC03FFC000007FF803FFC0000 07FF801FFC000007FF001FFC000007FF000FFE00000FFE0007FF00001FFC0003FF00001FF80001 FFC0007FF00000FFE000FFE000007FFC07FFC000001FFFFFFF00000007FFFFFC00000001FFFFF0 000000001FFF000000332E7DAD3A>I<0001F000000001F000000001F000000001F000000001F0 00000001F000000003F000000003F000000003F000000007F000000007F000000007F00000000F F00000000FF00000001FF00000003FF00000003FF00000007FF0000001FFF0000003FFF000000F FFFFFFC0FFFFFFFFC0FFFFFFFFC0FFFFFFFFC0FFFFFFFFC0007FF00000007FF00000007FF00000 007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF000 00007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF0 0000007FF00000007FF00000007FF00000007FF00000007FF001F0007FF001F0007FF001F0007F F001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF003F0003FF803E000 3FF807E0001FFC07C0001FFE0F80000FFFFF800003FFFE000000FFFC0000001FE00024427EC12D >116 D<007FC00000FF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80 FFFFC001FFFF8003FFC00007FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC000 03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003 FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00007FF8001FFC00007FF8001FFC00007FF8001FFC0000FFF8001FFC0001FFF8000FFC0 003FFF8000FFE0007FFF80007FE000FBFFC0007FF803F3FFFF003FFFFFE3FFFF000FFFFF83FFFF 0003FFFE03FFFF00007FF003FFFF382E7BAD41>I E /Fz 8 117 df<00001E000000003E000000 00FE00000003FE0000003FFE0000FFFFFE0000FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE00007FFFFFFF807FFFFFFF807FFFFFFF807FFFFFFF80213879B7 30>49 D<0003FF000700001FFFE00F0000FFFFFC0F0001FFFFFE1F0007FE00FFBF000FF0001FFF 001FE00007FF001FC00003FF003F800001FF003F800000FF007F0000007F007F0000007F007F00 00003F00FF0000003F00FF0000001F00FF8000001F00FF8000001F00FFC000000F00FFC000000F 00FFE000000F00FFF800000000FFFE000000007FFFE00000007FFFFF0000003FFFFFF000003FFF FFFE00001FFFFFFF80000FFFFFFFE0000FFFFFFFF80007FFFFFFFC0001FFFFFFFE0000FFFFFFFE 00003FFFFFFF00000FFFFFFF800000FFFFFF80000007FFFFC00000007FFFC000000007FFC00000 0001FFE000000000FFE0000000007FE0F00000007FE0F00000003FE0F00000003FE0F00000001F E0F00000001FE0F80000001FE0F80000001FE0FC0000001FC0FC0000001FC0FE0000003FC0FF00 00003F80FF8000007F80FFC000007F00FFE00000FE00FFFC0003FC00FDFFC00FF800F87FFFFFF0 00F01FFFFFC000F003FFFF0000E0003FF800002B3D7BBB36>83 D<0001FFF000000FFFFE00003F FFFF8000FFC03FC003FF007FE007FE007FE00FFC007FE01FF8007FE01FF8007FE03FF0003FC03F F0001F807FF0000F007FE00000007FE0000000FFE0000000FFE0000000FFE0000000FFE0000000 FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00000007FE00000007FF00000 007FF00000003FF00000003FF80000F01FF80000F01FFC0001F00FFC0003E007FE0007C003FF80 0FC000FFE03F00003FFFFE00000FFFF8000001FFC00024267DA52B>99 D<0003FF8000001FFFF0 00007FFFFC0000FF83FF0003FE00FF8007FC003F800FF8003FC01FF8001FE01FF0001FE03FF000 1FF03FF0000FF07FE0000FF07FE0000FF87FE0000FF8FFE0000FF8FFE0000FF8FFFFFFFFF8FFFF FFFFF8FFFFFFFFF8FFE0000000FFE0000000FFE0000000FFE0000000FFE00000007FE00000007F E00000007FF00000003FF00000783FF00000781FF80000F80FF80000F007FC0001F003FE0003E0 01FF000FC000FFC07F80003FFFFE00000FFFFC000001FFC00025267DA52C>101 D<01F00003FC0007FE000FFF000FFF001FFF001FFF001FFF000FFF000FFF0007FE0003FC0001F0 0000000000000000000000000000000000000000000000000000000000000000FF00FFFF00FFFF 00FFFF00FFFF0007FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF 0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF0003FF 0003FF0003FF0003FF0003FF0003FF00FFFFF8FFFFF8FFFFF8FFFFF8153D7DBC1B>105 D<00FE003FE00000FFFE01FFFC0000FFFE07FFFE0000FFFE0F81FF8000FFFE3C00FF800007FE38 00FFC00003FE7000FFC00003FEE0007FE00003FEC0007FE00003FFC0007FE00003FF80007FE000 03FF80007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF0000 7FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003 FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007F E00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF 00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF8031267DA536> 110 D<0001FFC00000000FFFF80000007FFFFF000000FF80FF800003FE003FE00007FC001FF000 0FF8000FF8000FF00007F8001FF00007FC003FF00007FE003FE00003FE007FE00003FF007FE000 03FF007FE00003FF007FE00003FF00FFE00003FF80FFE00003FF80FFE00003FF80FFE00003FF80 FFE00003FF80FFE00003FF80FFE00003FF80FFE00003FF80FFE00003FF807FE00003FF007FE000 03FF007FE00003FF003FF00007FE003FF00007FE001FF00007FC001FF00007FC000FF8000FF800 07FC001FF00003FE003FE00000FF80FF8000007FFFFF0000000FFFF800000001FFC0000029267D A530>I<000F0000000F0000000F0000000F0000000F0000001F0000001F0000001F0000001F00 00003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00001FFFFFF0FFFF FFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF000003FF000003 FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF0000 03FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF00 3C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377EB626>116 D E /FA 78 124 df<000FC3E0007FEFF000E07C7801C0F8780380F83007007000070070000700 700007007000070070000700700007007000FFFFFF80FFFFFF8007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 0700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381 E00381E00701E0070000070000070000070000070000070000FFFFE0FFFFE00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E00380E00700E00700E00700E007 00E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E07FE7FE7FE7FE1720809F19>I< 000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700F00F00070070000007007000 000700700000070070000007007000000700700000FFFFFFFF00FFFFFFFF000700700700070070 070007007007000700700700070070070007007007000700700700070070070007007007000700 7007000700700700070070070007007007000700700700070070070007007007007FE3FE3FF07F E3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06180C180C381C301860304020 0F0E7E9F17>34 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D>39 D<006000C00180030007 0006000C001C00180038003800300070007000700060006000E000E000E000E000E000E000E000 E000E000E000E000E0006000600070007000700030003800380018001C000C0006000700030001 8000C000600B2E7DA112>II<018001C00180 0180C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C0018010147DA117 >I<00060000000600000006000000060000000600000006000000060000000600000006000000 0600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF00006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 000006000000060000000600001C207D9A23>I<70F8FCFC7C0C0C0C181838306040060E7C840D> II<70F8F8F87005057C840D>I<00030003000700060006000E000C 000C001C0018001800380030003000700060006000E000C000C001C00180018001800380030003 000700060006000E000C000C001C0018001800380030003000700060006000E000C000C000102D 7DA117>I<03F0000FFC001E1E001C0E00380700780780700380700380700380F003C0F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C070038070038070 03807807803807001C0E001E1E000FFC0003F000121F7E9D17>I<018003801F80FF80E3800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 8003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E00300F00600F80C00780F007C0F807 C0F803C0F803C02003C00007C0000780000780000F00000E00001C0000380000700000E00001C0 000380000300000600C00C00C01800C03001807FFF80FFFF80FFFF80121E7E9D17>I<03F0000F FC001C1E00300F00780F80780F80780780380F80000F80000F00000F00001E00003C0003F80003 F000001C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780600F8070 0F003C1E001FFC0003F000121F7E9D17>I<000E00000E00001E00003E00003E00006E0000EE00 00CE00018E00038E00030E00060E000E0E000C0E00180E00380E00300E00600E00E00E00FFFFF0 FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE000FFE0141E7F9D17>I<3803 003FFF003FFE003FF8003FE00030000030000030000030000030000030000031F00037FC003E0E 003C07003807803003800003800003C00003C00003C06003C0F003C0F003C0F00380C007806007 00700E003C1E000FF80007E000121F7E9D17>I<007C0001FE000783000E03800C07801C078038 0300380000780000700000700000F3F800F7FE00FC0E00F80700F80380F80380F003C0F003C0F0 03C0F003C0F003C07003C07003C07003803803803807001C07000E1E0007FC0001F000121F7E9D 17>I<6000007FFFC07FFFC07FFF80600180C00300C00600C00600000C00001800001800003000 00600000600000E00000C00001C00001C00001C000038000038000038000038000078000078000 078000078000078000078000078000030000121F7D9D17>I<03F0000FFC001E1E003807003007 007003807003807003807003807807003E07003F0E001FDC000FF80003F80007FC001E7E00383F 00700F80700780E003C0E001C0E001C0E001C0E001C0F001807003803807001E0E000FFC0003F0 00121F7E9D17>I<03F00007F8001E1C00380E00380700700700700380F00380F00380F003C0F0 03C0F003C0F003C0F003C07007C07007C0380FC01C0FC01FFBC007F3C000038000038000078000 0700300700780E00780E00701C003078001FF0000FC000121F7E9D17>I<70F8F8F87000000000 00000000000070F8F8F87005147C930D>I<70F8F8F8700000000000000000000070F8F8F87818 1818183030606040051D7C930D>I<7FFFFFE0FFFFFFF000000000000000000000000000000000 00000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018 F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000 C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0 0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 DI<001FC04000 7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0 780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8 1C00007FF000001FC0001A217D9F21>IIII<000FC020007FF86001F81CE003E006E0078003 E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C 0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F 24>III76 DII<001F800000FFF00001E0780007C03E000F80 1F000F000F001E0007803C0003C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F8 0001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E0 3C0003C03E0007C01E0007800F000F000F801F0007C03E0001F0F80000FFF000001F80001C217D 9F23>II82 D<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F000007800 007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0C001 C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE0780F 01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C1F7E 9E21>II II<7FF83FF87FF83FF807C00F8003C0060001E00E0001F00C0000F0180000783800007C 3000003C7000003E6000001EC000000FC000000F8000000780000007C0000007E000000DE00000 1DF0000018F8000038780000307C0000603C0000E01E0000C01F0001800F0003800780030007C0 0F8007C0FFE01FFEFFE01FFE1F1F7F9E22>I91 D<0804180C3018703860306030C060C060C060F87CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE0003FF000783800781C00300E00000E00000E00000E0003FE001FFE 003E0E00780E00F80E00F00E30F00E30F00E30F01E30783F703FEFE00F878014147E9317>97 D<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3F 000EFF800FC3C00F00E00E00F00E00700E00780E00780E00780E00780E00780E00780E00780E00 700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19>I<03F80FFC1E1E3C1E380C780070 00F000F000F000F000F000F0007000780038033C031F0E0FFC03F010147E9314>I<000380003F 80003F8000038000038000038000038000038000038000038000038000038003E3800FFB801E0F 803C0780780380780380700380F00380F00380F00380F00380F00380F003807003807003807807 803807801E1F800FFBF803E3F815207E9F19>I<03F0000FFC001E1E003C0F0038070078070070 0380F00380FFFF80FFFF80F00000F00000F000007000007800003801801C03800F070007FE0001 F80011147F9314>I<003C00FE01CF038F03060700070007000700070007000700FFF0FFF00700 0700070007000700070007000700070007000700070007000700070007007FF07FF01020809F0E >I<0001E003E3F00FFF701C1C703C1E00380E00780F00780F00780F00780F00380E003C1E001C 1C003FF80033E0003000003000003800003FFE001FFF803FFFC07003E07000F0E00070E00070E0 0070F000F07801E03E07C00FFF0003FC00141F7F9417>I<0E0000FE0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E3E000EFF000FC3800F01C00F01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC FFE7FC16207F9F19>I<1C001E003E001E001C000000000000000000000000000E007E007E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A1F809E0C> I<00E001F001F001F000E0000000000000000000000000007007F007F000F00070007000700070 00700070007000700070007000700070007000700070007000700070007000706070F0E0F1C07F 803E000C28829E0E>I<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18000E30000E70000EF0000FF8000F38 000E1C000E1E000E0E000E07000E07800E03800E03C0FFCFF8FFCFF815207F9F18>I<0E00FE00 FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE0FFE00B20809F0C>I<0E3F03F000FEFF8FF800FF C1DC1C000F80F80E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E 00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00FFC3800F01C00F01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FCFFE7FC 16147F9319>I<01F80007FE001E07803C03C03801C07000E07000E0F000F0F000F0F000F0F000 F0F000F0F000F07000E07801E03801C03C03C01E078007FE0001F80014147F9317>I<0E3F00FE FF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E00780E00780E00780E00780E00700E 00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E00000E00000E00000E00000E0000FF E000FFE000151D7F9319>I<03E1800FF9801E1F803C0780780780780380700380F00380F00380 F00380F00380F00380F003807003807803807807803C0F801E1F800FF38003E380000380000380 000380000380000380000380000380003FF8003FF8151D7E9318>I<0E78FEFCFF9E0F1E0F0C0F 000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00F147F9312>I<1F903F F07070E030E030E030F00078007F803FE00FF000F0C078C038C038E038E038F070DFE08F800D14 7E9312>I<06000600060006000E000E001E003E00FFF8FFF80E000E000E000E000E000E000E00 0E000E000E000E180E180E180E180E18073007E003C00D1C7F9B12>I<0E01C0FE1FC0FE1FC00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00E 03C0070FC007FDFC01F1FC16147F9319>III<7FC7FC7FC7FC0703E007038003830001C70000EE0000EC 00007800003800003C00007C0000EE0001C7000187000303800701C00F01E0FF87FEFF87FE1714 809318>II<3FFF3FFF380E301C703C6038607060F001E0 01C0038007830F030E031C073C063806701EFFFEFFFE10147F9314>II E /FB 45 122 df<70F8FCFC7C0C0C0C0C181830306040060F7C840E>44 DI<70F8F8F87005057C840E>I<01F00007FC000E0E001C07003803 803803807803C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001 E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07 000E0E0007FC0001F00013227EA018>48 D<018003800F80FF80F3800380038003800380038003 800380038003800380038003800380038003800380038003800380038003800380038003800380 0380FFFEFFFE0F217CA018>I<03F0000FFC001C1F003007807007807807C07807C07803C03807 C00007C0000780000780000F00000E00003C0003F80003F800001E00000F000007800007C00003 C00003E02003E07003E0F803E0F803E0F003C06003C06007803007801C1F000FFC0003F0001322 7EA018>51 D<000E00000E00001E00001E00003E00003E00006E0000EE0000CE0001CE00018E00 030E00070E00060E000E0E000C0E00180E00180E00300E00700E00600E00E00E00FFFFF8FFFFF8 000E00000E00000E00000E00000E00000E00000E0001FFF001FFF015217FA018>I<1000801C07 801FFF001FFE001FFC001FF00018000018000018000018000018000018000018F8001BFE001F0F 001C07801803801803C00001C00001E00001E00001E00001E06001E0F001E0F001E0F001C0C003 C06003C06003803007001C1E000FFC0003F00013227EA018>I<007E0001FF0003C1800701C00E 03C01C03C01C0180380000380000780000700000702000F1FC00F3FE00F60700FC0380F803C0F8 01C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001E07001E03801C03803C01803801C 07000E0E0007FC0001F00013227EA018>I<01F00007FC000E0F001807803803803001C07001C0 7001C07001C07801C07803803E03803F07001FCE000FFC0003F80003FC000F7F001C3F80380F80 7007C07003C0E001E0E000E0E000E0E000E0E000E0E000E07001C07001C03803801E0F000FFE00 03F00013227EA018>56 D<01F00007FC000E0E001C0700380380780380700380F001C0F001C0F0 01C0F001E0F001E0F001E0F001E0F001E07001E07003E07803E03807E01C0DE00FF9E007F1E000 81C00001C00003C0000380000380300700780700780E00701C003838001FF0000FC00013227EA0 18>I<000180000003C0000003C0000003C0000007E0000007E0000007E000000FF000000CF000 000CF000001CF800001878000018780000383C0000303C0000303C0000601E0000601E0000601E 0000C00F0000C00F0000C00F0001FFFF8001FFFF8001800780030003C0030003C0030003C00600 01E0060001E0060001E00E0000F01F0001F0FFC00FFFFFC00FFF20237EA225>65 D<000FE010003FF83000F81C7001E0067003C003F0078001F00F0000F01E0000F03E0000703C00 00707C0000707C0000307800003078000030F8000030F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000030780000307C0000307C0000303C0000603E0000601E000060 0F0000C0078000C003C0018001E0030000F80E00003FF800000FE0001C247DA223>67 D69 DI73 D77 DI80 D82 D<03F0200FFC601C0EE03803E07001E07001E0E000E0E000E0E00060E00060E00060F00000 F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000F00000F0000070 C00070C00070C00070C00070E00060E000E0F000C0F801C0EF0380C7FF0081FC0014247DA21B> I<7FFFFFF87FFFFFF87C0780F8700780386007801860078018E007801CC007800CC007800CC007 800CC007800CC007800C0007800000078000000780000007800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078000 00078000000780000007800003FFFF0003FFFF001E227EA123>I<0FE0001FF8003C1C003C0E00 180700000700000700000F0003FF000FFF003F07007C0700780700F00700F00718F00718F00F18 780F187C3FB83FF3F00FC3C015157E9418>97 D<0E0000FE0000FE00001E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FE00FC0F00F00780E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0F00380F80700F C1F00C7FC00C1F0017237FA21B>I<01FE0007FF000F07801C0780380300780000700000F00000 F00000F00000F00000F00000F00000F000007800007800C03C00C01E01800F030007FE0001F800 12157E9416>I<0000E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E003F0E007FEE01F07E03C01E03800E07800E07000E0F000E0F000E0F000 E0F000E0F000E0F000E0F000E07000E07800E03801E03C03E01E0EF00FFCFE03F0FE17237EA21B >I<01FC0007FF000F07801C03C03801C07801E07000E0FFFFE0FFFFE0F00000F00000F00000F0 0000F000007800007800603C00601E00C00F838007FF0000FC0013157F9416>I<003C00FE01CF 038F038F070007000700070007000700070007000700FFF8FFF807000700070007000700070007 0007000700070007000700070007000700070007007FF87FF8102380A20F>I<0000F001F1F807 FFB80F1F381E0F001C07003C07803C07803C07803C07803C07801C07001E0F000F1E001FFC0019 F0001800001800001C00001FFF000FFFC01FFFE03801F0700070E00038E00038E00038E0003870 00707800F01E03C00FFF8001FC0015217F9518>I<0E0000FE0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FC00FC1E00F80F00F0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E0070FFE7FFFFE7FF18237FA21B>I<1C001E003E001E001C0000000000000000000000000000 0000000E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E00FFC0FFC00A227FA10E>I<01C003E003E003E001C000000000000000000000000000000000 01E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E000E060C0F1C0F1807F003E000B2C82A10F>I<0E0000FE0000FE0000 1E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FFC0E0FFC 0E07E00E03800E07000E0E000E18000E30000E78000EF8000F9C000F1E000E0E000E07000E0780 0E03C00E01C00E01E00E01F0FFE3FEFFE3FE17237FA21A>I<0E00FE00FE001E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E00FFE0FFE00B237FA20E>I<0E1FC07F00FE7FE1FF80FEC0F303C0 1F807E01E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038 00E0FFE3FF8FFEFFE3FF8FFE27157F942A>I<0E1F80FE7FC0FFC1E01F80F00F00700E00700E00 700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7 FFFFE7FF18157F941B>I<01FC0007FF000F07801C01C03800E07800F0700070F00078F00078F0 0078F00078F00078F00078F000787000707800F03800E01C01C00F078007FF0001FC0015157F94 18>I<0E1F80FE7FE0FFC1F00F00780E00780E003C0E003C0E001E0E001E0E001E0E001E0E001E 0E001E0E001E0E003C0E003C0F00780F80700FC1F00E7FC00E1F000E00000E00000E00000E0000 0E00000E00000E00000E0000FFE000FFE000171F7F941B>I<0E3CFEFEFFCF1F8F0F060F000E00 0E000E000E000E000E000E000E000E000E000E000E000E00FFF0FFF010157F9413>114 D<0F883FF87078E038E018E018E018F0007F003FE01FF001F8003CC01CC01CE01CE01CF018F878 DFF08FC00E157E9413>I<060006000600060006000E000E000E001E003E00FFF8FFF80E000E00 0E000E000E000E000E000E000E000E000E0C0E0C0E0C0E0C0E0C0E08071803F001E00E1F7F9E13 >I<0E0070FE07F0FE07F01E00F00E00700E00700E00700E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00F00E01F007037803FE7F01F87F18157F941B>III121 D E /FC 20 118 df45 D68 D73 D77 D80 D<007F802001FFE06007C0F8600F001CE01E000F E03C0007E0380003E0780001E0700001E0700000E0F00000E0F00000E0F0000060F0000060F000 0060F8000060F80000007C0000007E0000003F0000003FC000001FF800000FFF800007FFF80003 FFFC0000FFFF00000FFF800000FFC000001FE0000007E0000003F0000001F0000000F0000000F8 000000F8C0000078C0000078C0000078C0000078C0000078E0000078E0000070F00000F0F00000 E0F80000E0FC0001C0FE000380E7800700C1F01E00C07FFC00800FF0001D337CB125>83 D<01FE00000FFFC0001C03E0003C00F0003E0078003E003C003E001C0008001E0000001E000000 1E0000001E0000001E000001FE00003FFE0001FF1E0007F01E000F801E001F001E003E001E007C 001E007C001E00F8001E0CF8001E0CF8001E0CF8003E0CF8003E0C7C007E0C3E00DF1C1F038FB8 0FFF07F003F803C01E1F7D9E21>97 D<003FC000FFF803E01C07801E0F003E1E003E1E003E3C00 087C00007C0000780000F80000F80000F80000F80000F80000F80000F80000F80000F800007C00 007C00007C00003E00031E00031F00060F800E07C01C03F03800FFE0003F80181F7D9E1D>99 D<000001E000003FE000003FE0000003E0000001E0000001E0000001E0000001E0000001E00000 01E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E000 3F81E000FFE1E003E079E007800DE00F0007E01E0003E01E0001E03C0001E07C0001E07C0001E0 780001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001 E0780001E07C0001E07C0001E03C0001E03E0003E01E0007E00F000FE007801DE003E071F001FF E1FF003F01FF20327DB125>I<003F800000FFE00003E0F80007803C000F001E001E001E001E00 0F003C000F007C000F007C000F8078000780F8000780FFFFFF80FFFFFF80F8000000F8000000F8 000000F8000000F8000000F8000000780000007C0000003C0000003C0001801E0001800F000300 0F80060007C00E0001F03C0000FFF000001FC000191F7E9E1D>I<0003E0001FF8003C3800707C 00F07C01E07C01E03803C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C000FFFFC0FFFFC003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00007E0007FFF007FFF0016327FB114>I<000000F0007F03F801FFCF 1C03C1FC1C0780F8180F0078001F007C001E003C003E003E003E003E003E003E003E003E003E00 3E003E003E003E003E001E003C001F007C000F0078000780F0000FC1E0000DFFC0001C7F000018 000000180000001C0000001C0000001E0000000FFFF8000FFFFF0007FFFF800FFFFFC01E0007E0 380001F07000007070000078E0000038E0000038E0000038E0000038E000003870000070780000 F03C0001E01E0003C007C01F0001FFFC00003FE0001E2F7E9F21>I<07000F801F801F800F8007 00000000000000000000000000000000000000000000000780FF80FF800F800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800FC0FFF8FFF80D307EAF12>105 D<0781FE003FC000FF87FF80FFF000FF8E07C1C0F8000F98 03E3007C0007B001E6003C0007E000FC001E0007C000F8001E0007C000F8001E0007C000F8001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF034 1F7E9E38>109 D<0781FE0000FF87FF8000FF8E07C0000F9803E00007B001E00007E000F00007 C000F00007C000F00007C000F000078000F000078000F000078000F000078000F000078000F000 078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F000078000F0000FC001F800FFFC1F FF80FFFC1FFF80211F7E9E25>I<001FC00000FFF80001E03C0007800F000F0007801E0003C01E 0003C03C0001E03C0001E0780000F0780000F0780000F0F80000F8F80000F8F80000F8F80000F8 F80000F8F80000F8F80000F8F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003 C00F00078007C01F0001F07C0000FFF800001FC0001D1F7E9E21>I<0783E0FF8FF8FF9C7C0FB0 7C07F07C07E03807C00007C00007C0000780000780000780000780000780000780000780000780 000780000780000780000780000780000780000780000780000780000780000780000FC000FFFE 00FFFE00161F7E9E19>114 D<01FC100FFF301E03F03800F0700070E00070E00030E00030E000 30F00030F800007E00003FF0001FFF000FFF8003FFE0003FF00003F8000078C0003CC0003CC000 1CE0001CE0001CF0001CF00018F80038FC0070EF01E0C3FFC080FE00161F7E9E1A>I<00C00000 C00000C00000C00000C00001C00001C00001C00003C00003C00007C0000FC0001FC000FFFFE0FF FFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C03003C03003C03003C03003C03003C03003C03003C03001E06001 E06000F0C0007F80001F00142C7FAB19>I<078000F000FF801FF000FF801FF0000F8001F00007 8000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F000 078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078001F000078001F000078003F000038007F00003C00E F00001F03CF80000FFF0FF80003FC0FF80211F7E9E25>I E /FD 5 85 df<0000000060000000 0000E00000000000E00000000001E00000000001F00000000003F00000000003F00000000007F0 000000000FF0000000000FF0000000001BF0000000001BF00000000033F00000000033F0000000 0063F00000000063F000000000C3F000000001C3F00000000183F00000000303F00000000303F0 0000000603F80000000603F80000000C01F80000000C01F80000001801F80000003801F8000000 3001F80000006001F80000006001F8000000C001F8000000C001F80000018001F8000001FFFFF8 000003FFFFF80000070001F80000060001F800000C0001F800000C0001F80000180001FC000018 0001FC0000300000FC0000300000FC0000600000FC0000E00000FC0000C00000FC0001C00000FC 0003C00000FC000FE00003FC00FFFC003FFFE0FFF8003FFFE02B337CB234>65 D<000FFFFFFC00000FFFFFFF8000007F000FC000007E0003E000007E0001F000007E0000F80000 7E00007C0000FC00007C0000FC00003E0000FC00003E0000FC00003E0001F800001E0001F80000 1F0001F800001F0001F800001F0003F000001F0003F000001F0003F000001F0003F000001F0007 E000003F0007E000003F0007E000003F0007E000003F000FC000003E000FC000007E000FC00000 7E000FC000007E001F8000007C001F800000FC001F800000FC001F800000F8003F000001F8003F 000001F0003F000003F0003F000003E0007E000007E0007E000007C0007E00000F80007E00000F 0000FC00001F0000FC00003E0000FC00007C0000FC0000F80001F80001F00001F80007C00001F8 000F800003F8007E0000FFFFFFF80000FFFFFFC0000030317BB035>68 D<000FFFFFFFF8000FFF FFFFF800007F0003F800007E0000F800007E00007800007E00007800007E0000300000FC000030 0000FC0000300000FC0000300000FC0000300001F80000300001F80000300001F80000300001F8 0000600003F00180600003F00180000003F00180000003F00180000007E00300000007E0030000 0007E00700000007E01F0000000FFFFE0000000FFFFE0000000FC01E0000000FC00E0000001F80 0C0000001F800C0000001F800C0000001F800C0000003F00180000003F00180000003F00000000 003F00000000007E00000000007E00000000007E00000000007E0000000000FC0000000000FC00 00000000FC0000000000FC0000000001F80000000001F80000000001F80000000003F800000000 FFFFF0000000FFFFF00000002D317BB02E>70 D<000FFFFFE000000FFFFFFC0000007F003F0000 007E000F8000007E0007C000007E0003E000007E0003F00000FC0001F00000FC0001F00000FC00 01F00000FC0001F00001F80003F00001F80003F00001F80003F00001F80003E00003F00007E000 03F00007C00003F0000FC00003F0001F800007E0003F000007E0007C000007E000F8000007E007 E000000FFFFF8000000FFFFF0000000FC00FC000000FC007E000001F8003F000001F8001F00000 1F8001F800001F8001F800003F0001F800003F0001F800003F0001F800003F0001F800007E0003 F000007E0003F000007E0003F000007E0003F00000FC0007E00000FC0007E00000FC0007E00000 FC0007E01801F80007E03001F80007E03001F80007E06003F80003E060FFFFE001F1C0FFFFC000 FF80000000003E002D327BB033>82 D<07FFFFFFFFF007FFFFFFFFF00FE007F007F00F8007E001 F00E0007E000E00C0007E000E01C0007E000E018000FC000E018000FC000E030000FC000C03000 0FC000C030001F8000C060001F8000C060001F8000C060001F8000C0C0003F00018000003F0000 0000003F00000000003F00000000007E00000000007E00000000007E00000000007E0000000000 FC0000000000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F80000 000001F80000000003F00000000003F00000000003F00000000003F00000000007E00000000007 E00000000007E00000000007E0000000000FC0000000000FC0000000000FC0000000000FC00000 00001F80000000001F80000000001F80000000003F800000007FFFFF8000007FFFFF8000002C31 74B032>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 799 911 a FD(D)25 b(R)g(A)g(F)g(T)225 1002 y FC(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)621 1196 y FB(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum)766 1323 y(Septem)o(b)q(er)g(14,)h(1993)77 1381 y(This)h(w)o(ork)f(w)o(as)h(supp) q(orted)g(in)f(part)h(b)o(y)e(ARP)l(A)h(and)h(NSF)e(under)h(gran)o(t)h (ASC-9310330,)i(the)192 1439 y(National)d(Science)f(F)l(oundation)i(Science)e (and)i(T)l(ec)o(hnology)f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)76 1497 y(Agreemen)o(t)e(No.)22 b(CCR-8809615,)d(and)e(b)o(y)e(the)h(Commission) e(of)j(the)f(Europ)q(ean)i(Comm)o(unit)n(y)654 1555 y(through)f(Esprit)f(pro) s(ject)g(P6643.)p eop %%Page: 1 2 bop 166 49 a FA(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g(run)g(of)g (a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF)f(Final)75 106 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop %%Page: 1 3 bop 75 361 a Fz(Section)32 b(1)75 568 y Fy(P)m(oin)m(t)39 b(to)g(P)m(oin)m(t) g(Comm)m(u)o(nication)75 809 y Fx(1.1)59 b(Intro)r(duction)75 910 y FA(This)17 b(c)o(hapter)e(is)i(a)f(draft)f(of)h(the)g(curren)o(t)g (prop)q(osal)g(for)g(p)q(oin)o(t-to-p)q(oin)o(t)g(comm)o(unication.)23 b(Sending)75 967 y(and)c(receiving)h(of)e(messages)g(b)o(y)g(pro)q(cesses)h (is)g(the)g(basic)g(MPI)g(comm)o(unication)g(mec)o(hanism.)31 b(All)75 1023 y(other)20 b(comm)o(unication)h(functions)f(can)h(b)q(e)g (implemen)o(ted)g(on)f(top)g(of)g(this)g(basic)h(comm)o(unication)75 1080 y(la)o(y)o(er)15 b(\(although)g(a)g(more)g(direct)h(implemen)o(tation)g (ma)o(y)f(lead)h(to)e(greater)g(e\016ciency\).)166 1136 y(The)19 b(basic)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unication)g(op)q(erations)g (are)f Fw(send)g FA(and)h Fw(receiv)o(e)p FA(.)29 b(A)18 b Fw(send)75 1193 y FA(op)q(eration)13 b(creates)f(and)h(sends)g(a)g(message.) 18 b(The)13 b(op)q(eration)g(sp)q(eci\014es)i(a)d Fw(send)i(bu\013er)f FA(in)h(the)e(sender)75 1249 y(memory)f(from)g(whic)o(h)h(the)g(message)f (data)g(is)h(tak)o(en.)18 b(In)12 b(addition,)h(the)f(send)g(op)q(eration)g (asso)q(ciates)f(an)75 1306 y Fw(en)o(v)o(elop)q(e)16 b FA(with)g(the)g (message.)22 b(This)16 b(en)o(v)o(elop)q(e)h(sp)q(eci\014es)h(the)e(message)g (destination)h(and)f(con)o(tains)75 1362 y(distinguishing)h(information)d (that)f(can)i(b)q(e)g(used)f(b)o(y)g(the)g Fw(receiv)o(e)g FA(op)q(eration)g(to)g(select)h(a)e(particular)75 1418 y(message.)166 1475 y(A)22 b Fw(receiv)o(e)f FA(op)q(eration)h(consumes)g(a)f(message.)39 b(The)22 b(message)f(to)g(b)q(e)i(receiv)o(ed)f(is)h(selected)75 1531 y(according)13 b(to)f(the)g(v)m(alue)i(on)e(its)g(en)o(v)o(elop)q(e,)i (and)f(the)f(message)g(data)g(is)h(put)f(in)o(to)g(the)h Fw(receiv)o(e)g (bu\013er)p FA(.)166 1588 y(The)f(next)g(sections)h(describ)q(e)g(the)g (basic)f(\(blo)q(c)o(king\))h(send)g(and)f(receiv)o(e)h(op)q(erations.)19 b(W)l(e)12 b(discuss)75 1644 y(send,)21 b(receiv)o(e,)g(basic)f(comm)o (unication)g(seman)o(tics,)h(t)o(yp)q(e)e(matc)o(hing)h(requiremen)o(ts,)h(t) o(yp)q(e)e(con)o(v)o(er-)75 1701 y(sion)14 b(in)h(heterogeneous)f(en)o (vironmen)o(ts,)f(and)h(more)g(general)g(comm)o(unication)g(mo)q(des.)20 b(Non)o(blo)q(c)o(king)75 1757 y(comm)o(unication)c(is)h(addressed)f(next,)f (follo)o(w)o(ed)h(b)o(y)g(c)o(hannel-lik)o(e)i(constructs)e(and)f (send-receiv)o(e)j(op-)75 1814 y(erations.)i(W)l(e)15 b(then)g(consider)h (general)f(datat)o(yp)q(es)f(that)g(allo)o(w)h(one)g(to)g(transfer)f (heterogeneous)h(and)75 1870 y(noncon)o(tiguous)f(data,)f(and)h(conclude)h (with)f(a)g(description)h(of)e(an)h(implemen)o(tation)h(of)e(MPI)g(p)q(oin)o (t)i(to)75 1927 y(p)q(oin)o(t)h(comm)o(unication)g(using)g(a)e(small)i(n)o (um)o(b)q(er)g(of)f(primitiv)o(es.)75 2070 y Fx(1.2)59 b(Basic)20 b(send)g(op)r(eration)75 2171 y FA(The)15 b(syn)o(tax)g(of)g(the)g(simplest)h (send)g(op)q(eration)f(is)h(giv)o(en)g(b)q(elo)o(w.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 2 4 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(2)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(SEND)15 b(\(sta)o(rt,)g(count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(\))117 126 y Fs(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 202 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)g(in)g(send)h(bu\013er)h (\(nonnegativ)o(e)e(in)o(te-)905 258 y(ger\))117 333 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h (bu\013er)g(elemen)o(t)117 408 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 483 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 559 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))75 748 y Fr(1.2.1)49 b(Message)16 b(data)75 834 y FA(The)i(send)h(bu\013er)f(sp)q(eci\014ed)i(b)o(y)d(the)h Ft(MPI)p 828 834 V 16 w(SEND)h FA(op)q(eration)f(consists)g(of)f Ft(count)i FA(successiv)o(e)h(en)o(tries)75 891 y(of)e(the)g(t)o(yp)q(e)g (indicated)h(b)o(y)f Ft(datat)o(yp)q(e)p FA(,)i(starting)e(with)g(the)g(en)o (try)f(at)h(address)g Ft(sta)o(rt)p FA(.)28 b(Note)18 b(that)f(w)o(e)75 947 y(sp)q(ecify)d(the)e(message)g(length)h(in)g(terms)e(of)h(n)o(um)o(b)q (er)h(of)e Fq(elements)p FA(,)h(not)f(n)o(um)o(b)q(er)i(of)f Fq(bytes)p FA(.)18 b(The)13 b(former)75 1003 y(is)j(mac)o(hine)g(indep)q (enden)o(t)h(and)f(closer)f(to)g(the)g(application)i(lev)o(el.)166 1060 y(The)g(data)f(part)g(of)g(the)g(message)h(consists)f(of)h(a)f(sequence) i(of)e Ft(count)i FA(v)m(alues,)f(eac)o(h)g(of)f(the)h(t)o(yp)q(e)75 1116 y(indicated)j(b)o(y)e Ft(datat)o(yp)q(e)p FA(.)31 b Ft(count)20 b FA(ma)o(y)d(b)q(e)i(zero,)g(in)g(whic)o(h)g(case)g(the)f(data)g(part)f(of)h (the)h(message)e(is)75 1173 y(empt)o(y)l(.)30 b(The)18 b(basic)h(datat)o(yp)q (es)f(that)g(can)h(b)q(e)g(sp)q(eci\014ed)h(for)e(message)g(data)g(v)m(alues) i(corresp)q(ond)f(to)75 1229 y(the)14 b(basic)h(datat)o(yp)q(es)f(of)f(the)i (host)e(language.)20 b(The)15 b(p)q(ossible)g(v)m(alues)h(of)e(this)g (parameter)g(for)f(F)l(ortran)75 1286 y(and)i(the)h(corresp)q(onding)g(F)l (ortran)e(t)o(yp)q(es)h(are)g(listed)h(b)q(elo)o(w)p 481 1448 989 2 v 480 1505 2 57 v 506 1488 a(MPI)f(datat)o(yp)q(e)p 1037 1505 V 277 w(F)l(ortran)f(datat)o(yp)q(e)p 1468 1505 V 481 1506 989 2 v 480 1563 2 57 v 506 1546 a Ft(MPI)p 591 1546 14 2 v 15 w(INTEGER)p 1037 1563 2 57 v 271 w Fp(INTEGER)p 1468 1563 V 480 1619 V 506 1602 a Ft(MPI)p 591 1602 14 2 v 15 w(REAL)p 1037 1619 2 57 v 349 w Fp(REAL)p 1468 1619 V 480 1676 V 506 1659 a Ft(MPI)p 591 1659 14 2 v 15 w(DOUBLE)p 1037 1676 2 57 v 281 w Fp(DOUBLE)23 b(PRECISION)p 1468 1676 V 480 1732 V 506 1715 a Ft(MPI)p 591 1715 14 2 v 15 w(COMPLEX)p 1037 1732 2 57 v 247 w Fp(COMPLEX)p 1468 1732 V 480 1788 V 506 1772 a Ft(MPI)p 591 1772 14 2 v 15 w(DOUBLE)p 785 1772 V 17 w(COMPLEX)p 1037 1788 2 57 v 51 w Fp(DOUBLE)g(COMPLEX)p 1468 1788 V 480 1845 V 506 1828 a Ft(MPI)p 591 1828 14 2 v 15 w(LOGICAL)p 1037 1845 2 57 v 275 w Fp(LOGICAL)p 1468 1845 V 480 1901 V 506 1884 a Ft(MPI)p 591 1884 14 2 v 15 w(CHARA)o(CTER)p 1037 1901 2 57 v 195 w Fp(CHARACTER)p 1468 1901 V 480 1958 V 506 1941 a Ft(MPI)p 591 1941 14 2 v 15 w(BYTE)p 1037 1958 2 57 v 1468 1958 V 481 1959 989 2 v 1875 2136 a Fo(C)n(H)-6 b(A)h(N)f(G)h(E)75 2143 y FA(The)13 b(p)q(ossible)h(v)m(alues)f(for)f(this)g(parameter)g(for)g(C)g (and)g(the)h(corresp)q(onding)g(C)f(t)o(yp)q(es)g(are)g(listed)i(b)q(elo)o (w.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 3 5 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(3)p 466 5 1019 2 v 465 62 2 57 v 491 45 a(MPI)15 b(datat)o(yp)q(e)p 1004 62 V 259 w(C)g(datat)o(yp)q(e)p 1483 62 V 466 63 1019 2 v 465 120 2 57 v 491 103 a Ft(MPI)p 576 103 14 2 v 16 w(CHAR)p 1004 120 2 57 v 321 w Fp(signed)23 b(char)p 1483 120 V 465 176 V 491 159 a Ft(MPI)p 576 159 14 2 v 16 w(SHORT)p 1004 176 2 57 v 291 w Fp(signed)g(short)p 1483 176 V 465 233 V 491 216 a Ft(MPI)p 576 216 14 2 v 16 w(INT)p 1004 233 2 57 v 365 w Fp(signed)g(int)p 1483 233 V 465 289 V 491 272 a Ft(MPI)p 576 272 14 2 v 16 w(LONG)p 1004 289 2 57 v 321 w Fp(signed)g(long)g(int)p 1483 289 V 465 346 V 491 329 a Ft(MPI)p 576 329 14 2 v 16 w(UNSIGNED)p 815 329 V 17 w(CHAR)p 1004 346 2 57 v 81 w Fp(unsigned)f(char)p 1483 346 V 465 402 V 491 385 a Ft(MPI)p 576 385 14 2 v 16 w(UNSIGNED)p 815 385 V 17 w(SHORT)p 1004 402 2 57 v 51 w Fp(unsigned)g(short)i(int)p 1483 402 V 465 459 V 491 442 a Ft(MPI)p 576 442 14 2 v 16 w(UNSIGNED)p 1004 459 2 57 v 218 w Fp(unsigned)e(int)p 1483 459 V 465 515 V 491 498 a Ft(MPI)p 576 498 14 2 v 16 w(UNSIGNED)p 815 498 V 17 w(LONG)p 1004 515 2 57 v 81 w Fp(unsigned)g(long)i(int)p 1483 515 V 465 571 V 491 555 a Ft(MPI)p 576 555 14 2 v 16 w(FLO)o(A)l(T)p 1004 571 2 57 v 301 w Fp(float)p 1483 571 V 465 628 V 491 611 a Ft(MPI)p 576 611 14 2 v 16 w(DOUBLE)p 1004 628 2 57 v 262 w Fp(double)p 1483 628 V 465 684 V 491 667 a Ft(MPI)p 576 667 14 2 v 16 w(LONG)p 712 667 V 16 w(DOUBLE)p 1004 684 2 57 v 126 w Fp(long)f(double)p 1483 684 V 465 741 V 491 724 a Ft(MPI)p 576 724 14 2 v 16 w(BYTE)p 1004 741 2 57 v 1483 741 V 466 742 1019 2 v 166 913 a FA(The)16 b(datat)o(yp)q(e)f Ft(MPI)p 536 913 14 2 v 15 w(BYTE)i FA(do)q(es)e(not)g(corresp)q(ond)h(to)f(a)g(F)l (ortran)g(or)g(C)g(datat)o(yp)q(e.)20 b(A)c(v)m(alue)g(of)75 970 y(t)o(yp)q(e)i Ft(MPI)p 265 970 V 15 w(BYTE)h FA(consists)f(of)f(8)g (binary)h(digits.)28 b(A)18 b(b)o(yte)f(is)h(unin)o(terpreted)h(and)f(is)g (di\013eren)o(t)g(from)75 1026 y(a)f(c)o(haracter.)25 b(Di\013eren)o(t)17 b(mac)o(hines)g(ma)o(y)g(ha)o(v)o(e)g(di\013eren)o(t)g(represen)o(tations)g (for)g(c)o(haracters,)f(or)h(ma)o(y)75 1083 y(use)g(more)g(than)g(one)g(b)o (yte)g(to)g(represen)o(t)g(c)o(haracters.)25 b(On)18 b(the)f(other)g(hand,)g (a)g(b)o(yte)g(has)g(the)g(same)75 1139 y(binary)f(v)m(alue)g(on)f(all)h(mac) o(hines.)166 1272 y Fn(Discussion:)166 1322 y Fs(One)h(goal)e(of)h(the)h (design)g(is)f(to)g(allo)o(w)f(for)h(MPI)h(to)f(b)q(e)h(implemen)o(ted)d(as)j (a)f(library)m(,)g(with)g(no)g(need)h(for)75 1371 y(additional)d(prepro)q (cessing)k(or)e(compilation.)22 b(Th)o(us,)16 b(one)g(cannot)h(assume)e(that) h(a)g(comm)o(unication)d(call)i(has)75 1421 y(information)g(on)k(the)g(datat) o(yp)q(e)f(of)g(v)n(ariables)g(in)g(the)h(comm)o(unicatio)o(n)d(bu\013er;)21 b(this)d(information)e(m)o(ust)h(b)q(e)75 1471 y(supplied)h(b)o(y)f(an)h (explicit)f(parameter.)29 b(The)18 b(need)g(for)g(suc)o(h)g(datat)o(yp)q(e)g (information)d(will)h(b)q(ecome)h(clear)h(in)75 1521 y(Section)c(1.6.)75 1725 y Fr(1.2.2)49 b(Message)16 b(envelop)q(e)75 1811 y FA(In)g(addition)g (to)e(the)h(data)g(part,)f(messages)g(con)o(tain)h(information)h(that)e(can)h (b)q(e)h(used)f(to)g(distinguish)75 1867 y(messages)h(and)g(selectiv)o(ely)i (receiv)o(e)f(them.)23 b(This)17 b(information)f(is)h(con)o(tained)g(in)g(a)f (\014xed)h(n)o(um)o(b)q(er)f(of)75 1924 y(\014xed-format)f(\014elds,)h(whic)o (h)g(w)o(e)f(collectiv)o(ely)i(call)g(the)e Fw(message)i(en)o(v)o(elop)q(e)p FA(.)i(These)d(\014elds)g(are)75 2018 y Fw(source)75 2112 y(destination)75 2205 y(tag)75 2299 y(con)o(text)166 2393 y FA(The)c(message)g(source)g(is)h (implicitly)i(determined)f(b)o(y)e(the)g(iden)o(tit)o(y)h(of)f(the)g(message) g(sender.)20 b(The)75 2449 y(other)15 b(\014elds)h(are)f(sp)q(eci\014ed)i(b)o (y)f(parameters)e(in)i(the)f(send)h(op)q(eration.)166 2506 y(The)k(in)o(teger-v)m(alued)i(message)d(tag)g(is)i(sp)q(eci\014ed)h(b)o(y)d (the)h Ft(tag)g FA(parameter.)33 b(This)21 b(in)o(teger)f(can)75 2562 y(b)q(e)d(used)g(b)o(y)g(the)f(program)f(to)h(distinguish)j(di\013eren)o (t)d(t)o(yp)q(es)h(of)f(messages.)23 b(The)17 b(range)f(of)g(v)m(alid)i(tag) 75 2619 y(v)m(alues)e(is)g(implemen)o(tation)g(dep)q(enden)o(t)g(and)f(can)g (b)q(e)h(queried)g(using)g(the)f Ft(T)l(agRange)g FA(en)o(vironmen)o(tal)75 2675 y(inquiry)i(function,)e(as)g(describ)q(ed)i(in)f(Chapter)f Fw(??)p FA(.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 6 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(4)166 49 y(The)20 b(con)o(text)f(of)g(the)g(message)h(sen)o(t) f(is)h(sp)q(eci\014ed)i(b)o(y)d(the)h Ft(comm)13 b FA(parameter.)33 b(The)19 b(message)75 106 y(carries)c(the)h(con)o(text)e(asso)q(ciated)i (with)f(this)h(comm)o(unicator)f(\(see)g(Chapter)g Fw(??)p FA(\).)166 162 y(The)e(message)f(destination)h(is)h(sp)q(eci\014ed)g(b)o(y)f (the)f Ft(dest)j FA(parameter)c(as)i(a)f(rank)g(within)i(the)f(pro)q(cess)75 219 y(group)j(asso)q(ciated)h(with)g(that)f(same)h(comm)o(unicator)f (\(remote)g(pro)q(cess)h(group,)f(for)g(in)o(tercomm)o(uni-)75 275 y(cators\).)i(The)d(range)f(of)f(v)m(alid)j(v)m(alues)g(is)e Fp(0,)24 b(...)47 b(,)24 b(n-1)p FA(,)14 b(where)g Fp(n)g FA(is)h(the)f(n)o (um)o(b)q(er)h(of)f(pro)q(cesses)g(in)75 332 y(this)k(group.)25 b(Th)o(us,)17 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unications)g(do)f(not)g (use)g(absolute)h(addresses,)f(but)g(only)75 388 y(relativ)o(e)f(ranks)f (within)h(a)f(group.)k(This)d(pro)o(vides)g(imp)q(ortan)o(t)f(mo)q(dularit)o (y)l(.)166 444 y(The)j(message)f(en)o(v)o(elop)q(e)i(w)o(ould)f(normally)g(b) q(e)h(enco)q(ded)g(b)o(y)e(a)g(\014xed-length)j(message)d(header.)75 501 y(Ho)o(w)o(ev)o(er,)c(the)i(actual)f(mec)o(hanism)h(used)g(to)f(asso)q (ciate)g(an)g(en)o(v)o(elop)q(e)i(with)e(a)g(message)g(is)h(implemen-)75 557 y(tation)d(dep)q(enden)o(t:)20 b(some)12 b(of)g(the)g(information)g (\(e.g.,)g(source)g(or)g(destination\))g(ma)o(y)g(b)q(e)h(implicit,)i(and)75 614 y(need)f(not)g(b)q(e)g(explicitly)i(carried)e(b)o(y)g(messages;)f(pro)q (cesses)h(ma)o(y)f(b)q(e)h(iden)o(ti\014ed)i(b)o(y)d(relativ)o(e)h(ranks,)f (or)75 670 y(absolute)j(ids;)f(etc.)75 813 y Fx(1.3)59 b(Basic)20 b(receive)f(op)r(eration)75 915 y FA(The)c(syn)o(tax)g(of)g(the)g(simplest)h (receiv)o(e)g(op)q(eration)g(is)f(giv)o(en)h(b)q(elo)o(w.)75 1019 y Ft(MPI)p 160 1019 14 2 v 16 w(RECV)g(\(sta)o(rt,)f(count,)h(datat)o (yp)q(e,)g(source,)f(tag,)g(comm)m(,)d(status\))117 1096 y Fs(OUT)108 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h (bu\013er)g(\(c)o(hoice\))117 1171 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g (\(in)o(teger\))117 1246 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e (\(state\))117 1321 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j (\(in)o(teger\))117 1396 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 1472 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator) 11 b(\(handle\))117 1547 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 1671 y FA(The)g(receiv)o(e)h(bu\013er)f(consists)h(of)e(the)h (storage)f(con)o(taining)i Ft(count)g FA(consecutiv)o(e)g(elemen)o(ts)g(of)f (the)75 1728 y(t)o(yp)q(e)k(sp)q(eci\014ed)i(b)o(y)e Ft(datat)o(yp)q(e)p FA(,)i(starting)d(at)g(address)h Ft(sta)o(rt)p FA(.)31 b(The)19 b(length)h(of)e(the)h(receiv)o(ed)h(message)75 1784 y(m)o(ust)15 b(b)q(e)i(less)g(then)f(or)f(equal)i(to)e(the)h(length)h(of)f(the)g(receiv)o (e)g(bu\013er.)23 b(I.e.,)15 b(all)i(incoming)g(data)f(m)o(ust)75 1840 y(\014t,)23 b(without)e(truncation,)i(in)o(to)e(the)h(receiv)o(e)g (bu\013er.)39 b(The)22 b Ft(MPI)p 1262 1840 V 16 w(PROBE)g FA(function)h(describ)q(ed)g(in)75 1897 y(Section)16 b(1.9)e(can)i(b)q(e)g (used)f(to)g(receiv)o(e)h(messages)f(of)f(unkno)o(wn)i(length.)166 1953 y(The)j(selection)h(of)f(a)f(message)h(b)o(y)f(a)h(receiv)o(e)h(op)q (eration)f(is)g(done)g(uniquely)i(according)f(to)e(the)75 2010 y(v)m(alue)i(of)f(the)g(message)g(en)o(v)o(elop)q(e.)32 b(The)19 b(receiv)o(e)h(op)q(eration)g(sp)q(eci\014es)h(an)d Fw(en)o(v)o(elop)q(e)k (pattern)p FA(;)f(a)75 2066 y(message)13 b(can)g(b)q(e)g(receiv)o(ed)h(b)o(y) f(that)g(receiv)o(e)h(op)q(eration)f(only)g(if)h(its)f(en)o(v)o(elop)q(e)h (matc)o(hes)f(that)f(pattern.)75 2123 y(A)19 b(pattern)g(sp)q(eci\014es)i(v)m (alues)g(for)e(the)g Ft(source)p FA(,)i Ft(tag)e FA(and)h Ft(context)g FA(\014elds)h(of)e(the)g(message)g(en)o(v)o(elop)q(e.)75 2179 y(The)d(receiv)o(er)h(ma)o(y)e(sp)q(ecify)j(a)e(wildcard)h Fm(MPI)p 889 2179 13 2 v 14 w(ANY)p 988 2179 V 15 w(SOURCE)d FA(v)m(alue)k(for)d Ft(source)p FA(,)h(and/or)g(a)g(wildcard)75 2236 y Fm(MPI)p 152 2236 V 14 w(ANY)p 251 2236 V 15 w(T)m(A)o(G)c FA(v)m(alue)j(for)d Ft(tag)p FA(,)i(indicating)h(that)d(an)o(y)h(source)h (and/or)e(tag)h(are)g(acceptable.)20 b(It)13 b(cannot)75 2292 y(sp)q(ecify)h(a)f(wildcard)h(v)m(alue)g(for)f Ft(context)p FA(.)20 b(Th)o(us,)13 b(a)g(message)f(can)i(b)q(e)f(receiv)o(ed)h(b)o(y)f(a)g (receiv)o(e)h(op)q(eration)75 2349 y(only)23 b(if)g(it)g(is)g(addressed)h(to) e(the)g(receiving)j(pro)q(cess,)f(has)f(a)f(matc)o(hing)h(con)o(text,)g(has)g (matc)o(hing)75 2405 y(source)14 b(unless)g(source=)p Fm(MPI)p 581 2405 V 15 w(ANY)p 681 2405 V 15 w(SOURCE)d FA(in)k(the)e(pattern,)g(and)h (has)g(a)f(matc)o(hing)g(tag)g(unless)i(tag=)75 2461 y Fm(MPI)p 152 2461 V 14 w(ANY)p 251 2461 V 15 w(T)m(A)o(G)f FA(in)i(the)f(pattern.)166 2518 y(The)c(message)g(tag)f(is)i(sp)q(eci\014ed)h(b)o(y)e(the)g Ft(tag)g FA(parameter)f(of)h(the)g(receiv)o(e)h(op)q(eration.)19 b(The)11 b(message)75 2574 y Ft(context)19 b FA(is)e(the)h(con)o(text)e (attac)o(hed)h(with)h(the)f(comm)o(unicator)g(sp)q(eci\014ed)i(b)o(y)e(the)h (parameter)e Ft(comm)m FA(.)75 2631 y(The)h(message)g(source,)h(if)f (di\013eren)o(t)h(from)32 b Fm(MPI)p 929 2631 V 14 w(ANY)p 1028 2631 V 15 w(SOURCE)p FA(,)15 b(is)j(sp)q(eci\014ed)h(as)e(a)g(rank)f (within)j(the)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 5 7 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(5)75 49 y(pro)q(cess)19 b(group)f(asso)q(ciated)h(with)g(that) e(same)h(comm)o(unicator)h(\(remote)e(pro)q(cess)i(group,)g(for)f(in)o(ter-) 75 106 y(comm)o(unicators\).)k(Th)o(us,)17 b(the)f(range)g(of)g(v)m(alid)i(v) m(alues)f(for)f(the)g Ft(source)h FA(parameter)f(is)h Fl(f)23 b Fp(0,)h(...)47 b(,)75 162 y(n-1)23 b Fl(g[f)h Fp(MPI)p 345 162 15 2 v 17 w(ANY)p 434 162 V 17 w(SOURCE)f Fl(g)p FA(,)14 b(where)i Fp(n)f FA(is)g(the)h(n)o(um)o(b)q(er)f(of)g(pro)q(cesses)h(in)g (this)f(group.)166 219 y(Note)j(the)i(asymmetry)d(b)q(et)o(w)o(een)j(send)f (and)g(receiv)o(e)h(op)q(erations:)28 b(A)19 b(receiv)o(e)h(op)q(eration)f (ma)o(y)75 275 y(accept)13 b(messages)f(from)g(an)h(arbitrary)f(sender;)i(on) f(the)g(other)f(hand,)i(a)e(send)h(op)q(eration)g(m)o(ust)g(sp)q(ecify)75 332 y(a)i(unique)j(receiv)o(er.)k(This)16 b(matc)o(hes)f(a)h(\\push")f(comm)o (unication)i(mec)o(hanism,)f(where)g(data)f(transfer)75 388 y(is)i(e\013ected)h(b)o(y)f(the)g(sender)g(\(rather)f(than)h(a)g(\\pull")h (mec)o(hanism,)g(where)f(data)f(transfer)g(is)i(e\013ected)75 444 y(b)o(y)d(the)g(receiv)o(er\).)166 501 y(Source)h(=)f(destination)h(is)g (allo)o(w)o(ed:)k(a)15 b(pro)q(cess)g(can)h(send)g(a)e(message)h(to)g (itself.)75 620 y Fr(1.3.1)49 b(Return)16 b(status)75 705 y FA(The)j(source)f(or)g(the)h(tag)f(of)g(a)g(receiv)o(ed)i(message)e(ma)o(y)g (not)g(b)q(e)h(kno)o(wn)f(if)h(wildcard)h(v)m(alues)g(where)75 762 y(used)d(in)g(the)g(receiv)o(e)g(op)q(eration.)24 b(Also,)16 b(the)h(actual)f(length)i(of)e(the)g(message)g(receiv)o(ed)i(ma)o(y)d(not)h (b)q(e)75 818 y(kno)o(wn.)22 b(Th)o(us,)15 b(this)i(information)f(needs)g(to) g(b)q(e)g(returned)g(b)o(y)g(the)g(receiv)o(e)h(op)q(eration.)22 b(F)l(or)15 b(reasons)75 875 y(that)j(will)j(b)q(ecome)f(clear)g(later,)f(it) h(is)f(preferable)i(to)d(return)h(this)h(information)f(in)h(a)f(separate)f Ft(out)75 931 y FA(v)m(ariable,)d(rather)e(then)h(using)g(the)g Ft(count)p FA(,)h Ft(source)f FA(and)f Ft(tag)h FA(parameters)f(b)q(oth)g (for)g(input)i(and)f(output.)75 988 y(The)i(information)h(is)g(returned)f(b)o (y)g(the)h Ft(status)h FA(parameter)d(of)h(the)g Ft(MPI)p 1357 988 14 2 v 16 w(RECV)h FA(function.)24 b(This)17 b(is)f(a)75 1044 y(parameter)e(of)g(a)h(sp)q(ecial)h(MPI-de\014ned)h(t)o(yp)q(e.)j(The)15 b(status)f(v)m(ariable)i(can)f(b)q(e)g(\\deco)q(ded")h(to)e(retriev)o(e)75 1101 y(the)h Ft(count)p FA(,)h Ft(source)g FA(and)f Ft(tag)h FA(\014elds,)g(using)g(the)f(query)g(functions)h(listed)h(b)q(elo)o(w.)75 1204 y Ft(MPI)p 160 1204 V 16 w(GET)p 264 1204 V 17 w(SOURCE\(status,)h (source\))117 1281 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f(\(Status\))117 1353 y(OUT)108 b Ft(source)468 b Fs(source)15 b(rank)f(\(in)o(teger\))166 1478 y FA(Returns)f(the)g(rank)g(of)f(the)h(message)f(source)h(in)h(the)f (group)f(asso)q(ciated)h(with)g(the)g(comm)o(unicator)75 1534 y(that)h(w)o(as)h(used)h(to)e(receiv)o(e)i(\(remote)f(group,)f(for)h(in)o (tercomm)o(unicators\).)75 1638 y Ft(MPI)p 160 1638 V 16 w(GET)p 264 1638 V 17 w(T)l(A)o(G\(status,)i(tag\))117 1715 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f (\(Status\))117 1787 y(OUT)108 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))166 1912 y FA(Returns)j(the)f(tag)f(of)h(receiv)o(ed)h (message.)75 2015 y Ft(MPI)p 160 2015 V 16 w(GET)p 264 2015 V 17 w(COUNT\(status,)h(count\))117 2092 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f(\(Status\))117 2164 y(OUT)108 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h (elemen)o(ts)f(\(in)o(teger\))166 2289 y FA(Returns)h(the)f(n)o(um)o(b)q(er)h (of)f(elemen)o(ts)h(receiv)o(ed.)21 b(\(Here,)14 b(again,)h(w)o(e)f(coun)o(t) g Fq(elements)p FA(,)f(not)h Fq(bytes)p FA(.\))166 2345 y(The)20 b(information)f(returned)h(b)o(y)g(these)f(query)h(functions)g(is)g(the)g (information)g(last)f(stored)g(in)75 2402 y(the)c Ft(status)i FA(v)m(ariable)f(b)o(y)f(a)f(receiv)o(e)i(function.)21 b(It)15 b(is)g(erroneous)g(to)f(call)i(these)f(query)g(functions)h(if)f(the)75 2458 y Ft(status)i FA(v)m(ariable)g(w)o(as)d(nev)o(er)h(set)g(b)o(y)h(a)e (receiv)o(e.)166 2515 y(Note)h(that)g(it)h(is)g(not)f(mandatory)g(to)g(query) g(the)h(return)g(status)e(after)h(a)g(receiv)o(e.)22 b(The)16 b(receiv)o(er)75 2571 y(will)h(use)e(these)h(calls,)g(or)e(some)h(of)g(them,) g(only)h(when)f(the)h(information)f(they)g(return)g(is)h(needed.)166 2704 y Fn(Discussion:)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 8 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(6)166 49 y Fs(The)17 b(use)g(of)e(a)h(separate)i(status)f (parameter)f(prev)o(en)o(ts)i(errors)f(that)g(are)f(often)h(attac)o(hed)g (with)e(INOUT)75 99 y(parameters)g(\(e.g.,)e(passing)i(the)30 b Fm(MPI)p 706 99 13 2 v 14 w(ANY)p 805 99 V 15 w(T)m(A)o(G)13 b Fs(constan)o(t)i(as)g(the)g(actual)f(argumen)o(t)g(for)g Fm(tag)p Fs(\).)20 b(The)15 b(use)h(of)75 149 y(an)g(explicit)f(user)i(ob)r (ject)g(b)q(ecomes)f(imp)q(ortan)o(t)f(for)g(non)o(blo)q(c)o(king)g(comm)o (unication)e(as)j(it)f(allo)o(ws)g(the)i(receiv)o(e)75 199 y(op)q(eration)c(to)g(b)q(e)h(stateless)h(and,)d(hence,)i(reen)o(tran)o(t.)20 b(This)13 b(prev)o(en)o(ts)h(confusions)f(in)g(the)h(case)g(where)g(m)o (ultiple)75 249 y(receiv)o(es)i(can)e(b)q(e)g(p)q(osted)h(b)o(y)f(a)f(pro)q (cess.)166 464 y Fn(Implemen)o(tati)o(on)f(note:)166 520 y Fs(One)j(exp)q(ects)g(deco)q(de)h(functions)e(to)f(b)q(e)i(in-lined)e(in)g (man)o(y)f(implemen)o(tations.)166 659 y FA(All)18 b(send)f(and)f(receiv)o(e) i(op)q(erations)e(use)h(the)g Ft(sta)o(rt)p FA(,)f Ft(count)p FA(,)i Ft(datat)o(yp)q(e)p FA(,)g Ft(source)p FA(,)f Ft(dest)p FA(,)h Ft(tag)p FA(,)e Ft(comm)75 716 y FA(and)g Ft(status)i FA(parameters)d(in)i(the)f(same)f(w)o(a)o(y)g(as)h(the)g(basic)g Ft(MPI)p 1204 716 14 2 v 16 w(SEND)g FA(and)g Ft(MPI)p 1524 716 V 16 w(RECV)h FA(op)q(erations)75 772 y(describ)q(ed)g(in)f(this)g (section.)75 915 y Fx(1.4)59 b(Semantics)18 b(of)i(p)r(oint-to-p)r(oint)e (communication)75 1017 y FA(A)g(v)m(alid)h(MPI)e(implemen)o(tation)i(guaran)o (tees)e(certain)h(general)g(prop)q(erties)g(of)f(p)q(oin)o(t-to-p)q(oin)o(t)h (com-)75 1073 y(m)o(unication,)e(whic)o(h)g(are)f(describ)q(ed)i(in)f(this)f (section.)75 1193 y Ft(Order)46 b FA(Messages)14 b(are)h Fq(non-overtaking)p FA(,)g(within)h(eac)o(h)f(con)o(text:)20 b(if)c(t)o(w)o(o)d(messages)i(are)g (successiv)o(ely)75 1250 y(sen)o(t)10 b(from)g(the)g(same)g(source,)h(to)e (the)i(same)f(destination,)h(with)g(the)f(same)g(con)o(text,)h(then)f(the)g (messages)75 1306 y(are)17 b(receiv)o(ed)i(in)f(the)f(order)g(they)h(w)o(ere) f(sen)o(t.)26 b(I.e.,)17 b(if)h(the)f(receiv)o(er)i(p)q(osts)e(t)o(w)o(o)f (successiv)o(e)i(receiv)o(es)75 1363 y(that)g(b)q(oth)g(matc)o(h)g(either)i (message,)e(then)h(the)f(\014rst)h(receiv)o(e)g(will)h(receiv)o(e)g(the)e (\014rst)g(message,)h(and)75 1419 y(the)e(second)h(receiv)o(e)h(will)g (receiv)o(e)f(the)f(second)h(message.)26 b(This)18 b(requiremen)o(t)g (facilitates)g(matc)o(hing)75 1476 y(of)e(sends)h(to)f(receiv)o(es.)25 b(It)16 b(guaran)o(tees)g(that)g(message-passing)g(co)q(de)h(is)g (deterministic,)i(if)e(pro)q(cesses)75 1532 y(are)e(single-threaded)i(and)e (wildcard)31 b Fm(MPI)p 831 1532 13 2 v 14 w(ANY)p 930 1532 V 15 w(SOURCE)13 b FA(is)j(not)f(used)g(in)h(receiv)o(es.)166 1589 y(If)h(a)g(pro)q(cess)h(has)f(a)g(single)h(thread)f(of)g(execution,)i (then)e(an)o(y)g(t)o(w)o(o)f(comm)o(unications)i(executed)75 1645 y(b)o(y)e(this)h(pro)q(cess)g(are)f(ordered.)23 b(On)17 b(the)f(other)g(hand,)h(if)g(the)f(pro)q(cess)h(is)g(m)o(ulti-threaded,)g (then)g(the)75 1701 y(seman)o(tics)e(of)f(thread)h(execution)h(ma)o(y)e(not)g (de\014ne)i(a)e(relativ)o(e)i(order)e(b)q(et)o(w)o(een)h(t)o(w)o(o)e(send)j (op)q(erations)75 1758 y(executed)g(b)o(y)f(t)o(w)o(o)f(distinct)j(threads:)i (the)d(op)q(erations)f(are)g(logically)i(concurren)o(t,)e(ev)o(en)h(though)f (one)75 1814 y(ph)o(ysically)22 b(precedes)e(the)g(other.)33 b(In)20 b(suc)o(h)g(a)f(case,)i(the)e(t)o(w)o(o)g(messages)g(sen)o(t)g(can)h (b)q(e)g(receiv)o(ed)h(in)75 1871 y(an)o(y)f(order.)37 b(Similarly)l(,)24 b(if)d(t)o(w)o(o)e(receiv)o(e)j(op)q(erations)f(that)f(are)g(logically)j (concurren)o(t)e(receiv)o(e)g(t)o(w)o(o)75 1927 y(successiv)o(ely)i(sen)o(t)d (messages,)h(then)g(the)g(t)o(w)o(o)f(messages)g(can)h(matc)o(h)f(the)h(t)o (w)o(o)e(receiv)o(es)j(in)f(either)75 1984 y(order.)75 2104 y Ft(Progress)45 b FA(If)13 b(a)g(pair)g(of)f(matc)o(hing)h(send)h(and)f (receiv)o(es)g(ha)o(v)o(e)g(b)q(een)h(initiated)g(on)f(t)o(w)o(o)e(pro)q (cesses,)j(then)75 2160 y(at)k(least)h(one)h(of)e(these)h(t)o(w)o(o)f(op)q (erations)h(will)i(complete,)f(indep)q(enden)o(tly)i(of)c(other)h(actions)g (in)h(the)75 2217 y(system:)e(the)13 b(send)h(op)q(eration)f(will)h (complete,)g(unless)g(the)f(receiv)o(e)h(is)f(satis\014ed)h(b)o(y)e(another)h (message,)75 2273 y(and)h(completes;)g(the)g(receiv)o(e)h(op)q(eration)f (will)h(complete,)g(unless)g(the)f(message)f(sen)o(t)g(is)i(consumed)f(b)o(y) 75 2330 y(another)h(matc)o(hing)g(receiv)o(e)h(that)f(w)o(as)f(p)q(osted)h (at)g(the)g(same)g(destination)h(pro)q(cess.)75 2450 y Ft(F)o(airness)46 b FA(MPI)15 b(mak)o(es)g(no)g(guaran)o(tee)g(of)g Fq(fairness)f FA(in)i(the)g(handling)h(of)e(comm)o(unication.)21 b(Supp)q(ose)75 2506 y(that)c(a)f(send)i(w)o(as)f(p)q(osted.)26 b(Then)18 b(it)f(is)h(p)q (ossible)h(that)e(the)g(destination)h(pro)q(cess)g(rep)q(eatedly)g(p)q(osts) 75 2563 y(a)d(receiv)o(e)h(that)f(matc)o(hes)f(this)i(send,)g(y)o(et)e(the)i (message)f(is)g(nev)o(er)h(receiv)o(ed,)g(b)q(ecause)g(it)g(is)f(eac)o(h)h (time)75 2619 y(o)o(v)o(ertak)o(en)e(b)o(y)i(another)f(message,)g(sen)o(t)h (from)f(another)g(source.)21 b(Similarly)l(,)d(supp)q(ose)f(that)e(a)g (receiv)o(e)75 2675 y(w)o(as)f(p)q(osted)i(b)o(y)f(a)g(m)o(ulti-threaded)h (pro)q(cess.)k(Then)c(it)f(is)h(p)q(ossible)h(that)d(messages)h(that)f(matc)o (h)h(this)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 7 9 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(7)75 49 y(receiv)o(e)20 b(are)g(rep)q(eatedly)g(receiv)o(ed,)i (y)o(et)d(the)g(receiv)o(e)i(is)f(nev)o(er)f(satis\014ed,)i(b)q(ecause)f(it)g (is)g(o)o(v)o(ertak)o(en)75 106 y(b)o(y)15 b(other)h(receiv)o(es)g(p)q(osted) g(at)e(this)i(no)q(de)h(\(b)o(y)e(other)g(executing)h(threads\).)21 b(It)15 b(is)h(the)g(programmer's)75 162 y(resp)q(onsibilit)o(y)i(to)c(prev)o (en)o(t)h(starv)m(ation)g(in)h(suc)o(h)g(situations.)75 282 y Ft(Resource)e(lim)o(itations)44 b FA(The)13 b(curren)o(t)f(practice)h(for)f (man)o(y)g(commercial)i(message-passing)e(libraries)i(is)75 339 y(that)f(\(short\))f(messages)h(are)g(bu\013ered)g(b)o(y)h(the)f(system,) g(th)o(us)g(allo)o(wing)h(blo)q(c)o(king)h(send)f(op)q(erations)g(to)75 395 y(complete)d(ahead)f(of)g(the)h(matc)o(hing)f(receiv)o(es.)19 b(It)11 b(is)f(exp)q(ected)i(that)e(man)o(y)f(MPI)i(implemen)o(tations)g (will)75 452 y(follo)o(w)j(this)g(practice,)g(and)f(pro)o(vide)h(the)g(same)f (lev)o(el)i(of)e(bu\013ering)h(that)f(is)h(a)o(v)m(ailable)h(on)f(the)f (libraries)75 508 y(they)21 b(replace.)37 b(Ho)o(w)o(ev)o(er,)20 b(message)g(bu\013ering)h(is)h(not)e(a)g(univ)o(ersal)i(practice.)36 b(Ev)o(en)21 b(on)f(systems)75 564 y(where)e(bu\013ering)g(o)q(ccur,)g(the)f (amoun)o(t)g(of)g(bu\013er)h(space)f(a)o(v)m(ailable)j(and)d(the)h(w)o(a)o(y) e(it)i(is)g(allo)q(cated)g(is)75 621 y(b)q(ound)e(to)f(b)q(e)h(implemen)o (tation)g(dep)q(enden)o(t.)166 677 y(Therefore,)24 b(message)e(bu\013ering)h (is)g(not)f(mandated)g(b)o(y)g(MPI)h(and)f(is)h(seen)g(as)f(a)g(qualit)o(y)h (of)75 734 y(implemen)o(tation)d(issue.)32 b(A)19 b(v)m(alid)i(MPI)e (implemen)o(tation)h(of)f Ft(MPI)p 1285 734 14 2 v 15 w(SEND)h FA(is)f(to)g(blo)q(c)o(k)g(the)g(sender)75 790 y(un)o(til)k(a)f(matc)o(hing)h (receiv)o(e)g(has)f(b)q(een)h(initiated.)43 b(In)23 b(general,)h(the)e (programmer)f(can)h(mak)o(e)g(no)75 847 y(assumptions)f(on)g(the)g(a)o(v)m (ailabilit)o(y)j(of)c(bu\013er)h(space,)i(and)e(ho)o(w)f(this)i(space)f(is)h (allo)q(cated.)38 b(Th)o(us,)75 903 y(p)q(ortable)14 b(\(safe\))e(MPI)h(co)q (de)h(should)g(w)o(ork)f(under)h(the)f(assumption)g(that)g(an)g(arbitrary)g (subset)g(of)g(the)75 960 y(send)18 b(op)q(erations)f(are)g(going)g(to)g (return)g(b)q(efore)h(a)f(matc)o(hing)g(receiv)o(e)h(is)g(p)q(osted,)g(and)f (the)g(rest)g(will)75 1016 y(blo)q(c)o(k)f(un)o(til)g(a)f(matc)o(hing)g (receiv)o(e)i(is)e(p)q(osted.)166 1073 y(MPI)g(implemen)o(tation)i(will)h (pro)o(vide)e(information)g(on)f(the)h(amoun)o(t)f(of)g(a)o(v)m(ailable)i (bu\013er)f(space)75 1129 y(and)f(on)h(the)f(bu\013ering)h(p)q(olicy)g(via)g (the)f(en)o(vironmen)o(tal)h(enquiries)h(describ)q(ed)g(in)f(Chapter)f Fw(??)p FA(.)166 1186 y(Examples)h(\(in)o(v)o(olving)g(t)o(w)o(o)e(pro)q (cesses)h(with)h(ranks)f(0)f(and)i(1\))166 1242 y(The)f(follo)o(wing)h (program)e(is)i(safe,)f(and)g(should)h(alw)o(a)o(ys)f(succeed.)75 1392 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1449 y(IF)h(\(rank.EQ.0\))123 1505 y(THEN)170 1562 y(CALL)g (MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(ierr\))170 1618 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g (status,)g(ierr\))123 1675 y(ELSE)95 b(!)23 b(rank.EQ.1)170 1731 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))170 1787 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g (0,)h(tag,)f(comm,)g(ierr\))75 1844 y(END)g(IF)166 1938 y FA(The)15 b(follo)o(wing)h(program)e(is)i(erroneous,)f(and)g(should)h(alw)o(a)o(ys)f (fail.)75 2088 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2144 y(IF)h(\(rank.EQ.0\))123 2201 y(THEN)170 2257 y(CALL)g (MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g (ierr\))170 2314 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h (tag,)f(comm,)g(ierr\))123 2370 y(ELSE)95 b(!)23 b(rank.EQ.1)170 2427 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))170 2483 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g (0,)h(tag,)f(comm,)g(ierr\))75 2540 y(END)g(IF)166 2633 y FA(The)15 b(receiv)o(e)g(op)q(eration)g(of)f(the)h(\014rst)f(pro)q(cess)h(m)o(ust)f (complete)i(b)q(efore)f(its)f(send,)h(and)g(can)g(com-)75 2690 y(plete)j(only)g(if)g(the)f(matc)o(hing)g(send)h(of)f(the)g(second)h(pro)q (cessor)f(is)h(executed;)g(the)g(receiv)o(e)g(op)q(eration)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 10 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(8)75 49 y(of)14 b(the)h(second)g(pro)q(cess)g(m)o(ust)f (complete)h(b)q(efore)g(its)f(send)i(and)e(can)h(complete)g(only)g(if)g(the)g (matc)o(hing)75 106 y(send)h(of)f(the)g(\014rst)g(pro)q(cess)g(is)h (executed.)21 b(This)15 b(program)f(will)j(deadlo)q(c)o(k.)166 162 y(The)10 b(follo)o(wing)h(program)f(is)g(unsafe,)h(and)g(ma)o(y)e (succeed)j(or)e(fail,)h(dep)q(ending)i(on)d(implemen)o(tation.)75 325 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 381 y(IF)h(\(rank.EQ.0\))123 438 y(THEN)170 494 y(CALL)g(MPI_SEND\(sendbuf,)d (count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(ierr\))170 551 y(CALL)h (MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g (ierr\))123 607 y(ELSE)95 b(!)23 b(rank.EQ.1)170 664 y(CALL)h (MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g(ierr\))170 720 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))75 777 y(END)g(IF)166 939 y FA(The)11 b(message)g(sen)o(t)g (b)o(y)g(eac)o(h)g(pro)q(cess)g(has)g(to)f(b)q(e)i(copied)g(out)f(b)q(efore)g (the)g(send)h(op)q(eration)f(returns)75 996 y(and)k(the)g(receiv)o(e)h(op)q (eration)f(starts.)k(F)l(or)14 b(the)h(program)f(to)g(complete,)h(it)h(is)f (necessary)g(that)g(at)f(least)75 1052 y(one)i(of)g(the)h(t)o(w)o(o)d (messages)i(sen)o(t)g(is)h(bu\013ered.)23 b(Th)o(us,)16 b(this)h(program)e (can)i(succeed)g(only)g(if)g(the)f(MPI)75 1109 y(implemen)o(tation)e (bu\013ers)f(messages,)g(and)g(the)g(comm)o(unication)h(system)f(has)g (su\016cien)o(t)h(bu\013er)f(space)75 1165 y(to)i(bu\013er)g Ft(count)h FA(w)o(ords)f(of)f(data.)75 1308 y Fx(1.5)59 b(Data)20 b(T)-5 b(yp)r(e)19 b(Matching)75 1410 y FA(One)d(can)f(think)h(of)f(message)g (transmission)g(as)g(consisting)h(of)f(three)g(phases:)131 1504 y(1.)22 b(Data)14 b(is)h(pulled)j(out)d(of)f(the)i(send)f(bu\013er)h (and)f(a)g(message)g(is)g(assem)o(bled)131 1597 y(2.)22 b(A)15 b(message)g(is)g(transferred)g(from)f(sender)i(to)f(receiv)o(er)131 1691 y(3.)22 b(Data)14 b(is)h(pulled)j(from)c(the)h(incoming)i(message)e(and) g(disassem)o(bled)i(in)o(to)e(the)g(receiv)o(e)h(bu\013er)166 1785 y(T)o(yp)q(e)k(matc)o(hing)f(has)h(to)e(b)q(e)j(observ)o(ed)e(at)g(eac)o (h)g(of)h(these)f(three)h(phases:)29 b(The)19 b(t)o(yp)q(e)h(of)f(eac)o(h)75 1841 y(v)m(ariable)i(in)f(the)g(sender)g(bu\013er)g(has)g(to)f(matc)o(h)g (the)g(t)o(yp)q(e)h(sp)q(eci\014ed)i(for)d(that)g(en)o(try)g(b)o(y)h(the)f (send)75 1898 y(op)q(eration;)d(the)f(t)o(yp)q(e)h(sp)q(eci\014ed)i(b)o(y)e (the)f(send)i(op)q(eration)e(has)h(to)f(matc)o(h)g(the)h(t)o(yp)q(e)f(sp)q (eci\014ed)j(b)o(y)e(the)75 1954 y(receiv)o(e)g(op)q(eration;)e(and)h(the)g (t)o(yp)q(e)f(of)h(eac)o(h)f(v)m(ariable)i(in)g(the)e(receiv)o(e)i(bu\013er)f (has)f(to)g(matc)o(h)g(the)h(t)o(yp)q(e)75 2011 y(sp)q(eci\014ed)f(for)e (that)g(en)o(try)g(b)o(y)g(the)g(receiv)o(e)h(op)q(eration.)20 b(A)12 b(program)f(that)h(fails)h(to)e(observ)o(e)h(these)h(three)75 2067 y(rules)j(is)g(erroneous.)166 2124 y(T)l(o)f(de\014ne)i(t)o(yp)q(e)f (matc)o(hing)g(more)f(precisely)l(,)j(w)o(e)d(need)i(to)e(deal)h(with)g(t)o (w)o(o)f(issues:)22 b(matc)o(hing)15 b(of)75 2180 y(t)o(yp)q(es)e(of)g(the)h (host)f(language)g(with)h(t)o(yp)q(es)g(sp)q(eci\014ed)h(in)f(comm)o (unication)h(op)q(erations;)e(and)h(matc)o(hing)75 2237 y(of)h(t)o(yp)q(es)g (at)g(sender)g(and)h(receiv)o(er.)166 2293 y(A)h(t)o(yp)q(e)f(sp)q(eci\014ed) j(for)d(an)h(en)o(try)f(b)o(y)h(a)f(send)h(op)q(eration)g(matc)o(hes)f(the)h (t)o(yp)q(e)g(sp)q(eci\014ed)i(for)d(that)75 2350 y(en)o(try)11 b(b)o(y)h(a)f(receiv)o(e)h(op)q(eration)g(if)g(b)q(oth)f(op)q(erations)h(use) g(iden)o(tical)h(names:)36 b Ft(MPI)p 1500 2350 14 2 v 16 w(INTEGER)12 b FA(matc)o(hes)75 2406 y Ft(MPI)p 160 2406 V 16 w(INTEGER)p FA(,)j Ft(MPI)p 475 2406 V 16 w(REAL)h FA(matc)o(hes)30 b Ft(MPI)p 891 2406 V 15 w(REAL)p FA(,)16 b(and)f(so)g(on.)166 2462 y(The)k(t)o(yp)q(e)g (of)f(a)g(v)m(ariable)i(in)g(a)e(host)h(program)e(matc)o(hes)h(the)h(t)o(yp)q (e)g(sp)q(eci\014ed)i(in)e(the)g(comm)o(u-)75 2519 y(nication)g(op)q(eration) g(if)f(the)h(datat)o(yp)q(e)f(name)g(used)h(b)o(y)f(that)f(op)q(eration)i (corresp)q(onds)f(to)g(the)g(basic)75 2575 y(t)o(yp)q(e)i(of)g(the)g(host)g (program)f(v)m(ariable:)31 b(an)20 b(en)o(try)f(with)i(t)o(yp)q(e)f(name)40 b Ft(MPI)p 1449 2575 V 16 w(INTEGER)21 b FA(matc)o(hes)e(a)75 2632 y(F)l(ortran)c(v)m(ariable)j(of)f(t)o(yp)q(e)f Fp(INTEGER)p FA(,)g(an)g(en)o(try)g(with)h(t)o(yp)q(e)g(name)g Ft(MPI)p 1368 2632 V 16 w(REAL)g FA(matc)o(hes)f(a)g(F)l(ortran)75 2688 y(v)m(ariable)j(of)e(t)o(yp)q(e)g Fp(REAL)p FA(,)f(and)i(so)f(on.)26 b(There)18 b(is)f(one)h(exception)g(to)f(this)h(last)f(rule:)25 b(An)18 b(en)o(try)f(with)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 9 11 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)654 b FA(9)75 49 y(t)o(yp)q(e)18 b(name)37 b Ft(MPI)p 408 49 14 2 v 15 w(BYTE)19 b FA(can)f(b)q(e)h(used)g(to)e(matc)o(h)h(an)o(y)g (b)o(yte)g(of)g(storage)e(\(on)i(a)g(b)o(yte-addressable)75 106 y(mac)o(hine\),)d(irresp)q(ectiv)o(e)i(of)d(the)h(datat)o(yp)q(e)g(of)f (the)h(v)m(ariable)i(that)d(con)o(tains)h(this)g(b)o(yte.)20 b(The)15 b(v)m(alue)h(of)75 162 y(the)f(message)g(en)o(try)g(will)i(b)q(e)f (the)f(binary)h(v)m(alue)g(of)f(the)g(corresp)q(onding)h(b)o(yte)f(in)h (memory)l(.)166 219 y(W)l(e)f(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)143 312 y Fl(\017)23 b FA(Comm)o(unication)11 b(of)f(t)o(yp)q(ed)i(v)m(alues)g (\(e.g.,)e(with)h(datat)o(yp)q(e)g(di\013eren)o(t)g(from)21 b Ft(MPI)p 1586 312 V 16 w(BYTE)p FA(\),)11 b(where)189 369 y(the)k(datat)o(yp)q(es)f(of)g(the)h(corresp)q(onding)h(en)o(tries)f(in)h (the)f(sender)g(program,)e(in)j(the)f(send)g(call,)h(in)189 425 y(the)f(receiv)o(e)h(call)g(and)g(in)g(the)f(receiv)o(er)h(program)e (should)i(all)g(matc)o(h.)143 519 y Fl(\017)23 b FA(Comm)o(unication)16 b(of)g(un)o(t)o(yp)q(ed)h(v)m(alues)g(\(e.g.,)e(of)g(datat)o(yp)q(e)h Ft(MPI)p 1330 519 V 16 w(BYTE)p FA(\),)g(where)g(b)q(oth)g(sender)189 576 y(and)g(receiv)o(er)h(use)f(the)g(datat)o(yp)q(e)g Ft(MPI)p 881 576 V 15 w(BYTE)p FA(.)h(In)f(this)h(case,)f(there)g(are)g(no)f (requiremen)o(ts)i(on)189 632 y(the)d(t)o(yp)q(es)h(of)f(the)g(corresp)q (onding)h(en)o(tries)g(in)h(the)e(sender)h(and)g(the)f(receiv)o(er)h (programs,)e(nor)h(is)189 689 y(it)h(required)h(that)f(they)g(b)q(e)h(the)f (same.)166 782 y(The)g(follo)o(wing)h(examples)g(illustrate)h(these)e(t)o(w)o (o)f(cases.)166 839 y(First)h(program:)75 933 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 989 y(IF\(rank.EQ.0\))123 1046 y(THEN)170 1102 y(CALL)h(MPI_SEND\(a\(1\),)e(10,)h(MPI_REAL,)g(1,)g (tag,)h(comm,)f(ierr\))123 1158 y(ELSE)170 1215 y(CALL)h(MPI_RECV\(a\(1\),)e (15,)h(MPI_REAL,)g(0,)g(tag,)h(comm,)f(status,)g(ierr\))75 1271 y(END)g(IF)166 1365 y FA(This)c(co)q(de)f(is)h(correct)e(if)i(b)q(oth)f (sender)h(and)f(receiv)o(er)h(programs)d(ha)o(v)o(e)i(allo)q(cated)h (consecutiv)o(e)75 1422 y(storage)14 b(for)h(ten)g(real)g(n)o(um)o(b)q(ers,)g (starting)g(from)f Fp(a\(1\))p FA(.)166 1478 y(Second)i(program:)75 1572 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1628 y(IF\(rank.EQ.0\))123 1685 y(THEN)170 1741 y(CALL)h(MPI_SEND\(a\(1\),)e (10,)h(MPI_REAL,)g(1,)g(tag,)h(comm,)f(ierr\))123 1798 y(ELSE)170 1854 y(CALL)h(MPI_RECV\(a\(1\),)e(40,)h(MPI_BYTE,)g(0,)g(tag,)h(comm,)f (status,)g(ierr\))75 1911 y(END)g(IF)166 2004 y FA(This)d(co)q(de)g(is)f (erroneous,)h(since)g(sender)g(and)g(receiv)o(er)g(do)f(not)f(pro)o(vide)i (matc)o(hing)f(datat)o(yp)q(e)75 2061 y(parameters.)166 2117 y(Third)d(program:)75 2211 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h (ierr\))75 2268 y(IF\(rank.EQ.0\))123 2324 y(THEN)170 2381 y(CALL)h(MPI_SEND\(a\(1\),)e(40,)h(MPI_BYTE,)g(1,)g(tag,)h(comm,)f(ierr\))123 2437 y(ELSE)170 2493 y(CALL)h(MPI_RECV\(a\(1\),)e(60,)h(MPI_BYTE,)g(0,)g (tag,)h(comm,)f(status,)g(ierr\))75 2550 y(END)g(IF)166 2644 y FA(This)16 b(co)q(de)f(is)h(correct,)f(irresp)q(ectiv)o(e)h(of)f(the)g(t)o (yp)q(e)h(of)e Fp(a\(1\))h FA(and)g(follo)o(wing)h(v)m(ariables)h(in)f (store.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 12 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(10)166 49 y Fn(Discussion:)166 99 y Fs(The)15 b(b)q(eha)o(vior)f(of)g(erroneous)i(co)q(de)g(is)e(implemen)o(tation-dep)q (enden)o(t.)k(It)d(is)g(desirable)g(but)g(not)f(required)75 149 y(that)i(errors)i(are)e(rep)q(orted)i(at)e(run-time.)24 b(An)16 b(MPI)g(implemen)o(tation)d(ma)o(y)h(decide)j(to)f(a)o(v)o(oid)f(t)o (yp)q(e-c)o(hec)o(king)75 199 y(for)d(the)h(sak)o(e)f(of)g(e\016ciency)m(,)h (in)f(whic)o(h)g(case)h(none)g(of)e(the)i(three)h(programs)d(will)g(rep)q (ort)i(an)f(error.)19 b(On)12 b(a)g(system)75 249 y(where)f(real)g(n)o(um)o (b)q(ers)f(o)q(ccup)o(y)h(four)f(b)o(ytes)h(and)f(b)q(oth)g(sender)i(and)e (receiv)o(er)i(execute)g(in)e(the)h(same)f(en)o(vironmen)o(t,)75 298 y(it)k(is)f(lik)o(ely)g(that)h(all)e(three)k(co)q(des)f(will)d(ha)o(v)o (e)i(the)g(same)f(b)q(eha)o(vior.)75 523 y Fx(1.6)59 b(Data)20 b(conversion)75 625 y FA(One)c(of)f(the)g(goals)g(of)g(MPI)g(is)h(to)e(supp)q (ort)i(parallel)g(computations)f(across)g(heterogeneous)g(en)o(viron-)75 681 y(men)o(ts.)20 b(Comm)o(unication)15 b(in)h(a)f(heterogeneous)g(en)o (vironmen)o(t)h(ma)o(y)e(require)i(data)f(con)o(v)o(ersions.)166 738 y(W)l(e)g(use)h(the)f(follo)o(wing)h(terminology:)75 828 y Fw(t)o(yp)q(e)h(con)o(v)o(ersion)22 b FA(c)o(hanges)16 b(the)h(datat)o(yp)q (e)f(of)g(a)g(v)m(alue,)h(e.g.)23 b(b)o(y)17 b(rounding)g(a)f Ft(REAL)h FA(to)f(an)g Ft(INTE-)189 884 y(GER)p FA(.)75 976 y Fw(represen)o(tation)h(con)o(v)o(ersion)22 b FA(c)o(hanges)16 b(the)h(binary)g(represen)o(tation)f(of)g(a)g(v)m(alue,)h(e.g.)23 b(from)16 b(Hex)189 1033 y(\015oating)f(p)q(oin)o(t)g(to)g(IEEE)g(\015oating) h(p)q(oin)o(t.)166 1123 y(Represen)o(tation)d(con)o(v)o(ersion)g(is)h(needed) g(when)f(data)f(is)i(mo)o(v)o(ed)e(across)g(di\013eren)o(t)h(en)o(vironmen)o (ts,)75 1179 y(e.g.)19 b(di\013eren)o(t)d(mac)o(hines)g(that)e(use)i (di\013eren)o(t)f(binary)h(enco)q(dings)h(for)d(the)i(same)e(datat)o(yp)q(e,) h(or)f(co)q(des)75 1236 y(compiled)k(with)e(di\013eren)o(t)h(compiler)g (options)f(for)g(datat)o(yp)q(e)f(represen)o(tation.)23 b(Represen)o(tation) 17 b(con-)75 1292 y(v)o(ersion)12 b(preserv)o(es,)g(to)f(the)h(greatest)f(p)q (ossible)i(exten)o(t,)f(the)g Fw(v)m(alue)g FA(represen)o(ted.)19 b(Ho)o(w)o(ev)o(er,)11 b(rounding)75 1349 y(errors)k(and)g(o)o(v)o(er\015o)o (w)f(and)i(under\015o)o(w)g(exceptions)g(ma)o(y)f(o)q(ccur)h(during)g (\015oating)f(p)q(oin)o(t)h(con)o(v)o(ersions;)75 1405 y(con)o(v)o(ersion)k (of)f(in)o(tegers)g(ma)o(y)g(also)h(lead)g(to)f(exceptions)i(when)f(a)f(v)m (alue)i(that)d(can)i(represen)o(ted)g(in)75 1462 y(one)h(system)g(cannot)g(b) q(e)g(represen)o(ted)h(in)g(the)f(other)f(system.)37 b(MPI)21 b(do)q(es)h(not)e(sp)q(ecify)j(rules)f(for)75 1518 y(represen)o(tation)15 b(con)o(v)o(ersion.)166 1574 y(The)e(t)o(yp)q(e)g(matc)o(hing)g(rules)g (imply)h(that)e(MPI)h(comm)o(unication)h(nev)o(er)f(en)o(tails)g(t)o(yp)q(e)g (con)o(v)o(ersion.)75 1631 y(On)20 b(the)f(other)f(hand,)i(MPI)f(requires)h (that)e(a)h(represen)o(tation)g(con)o(v)o(ersion)g(b)q(e)h(p)q(erformed)f (when)g(a)75 1687 y(t)o(yp)q(ed)i(v)m(alue)h(is)f(transferred)f(across)g(en)o (vironmen)o(ts)h(that)f(use)h(di\013eren)o(t)f(represen)o(tations)h(for)f (the)75 1744 y(datat)o(yp)q(e)15 b(of)h(this)g(v)m(alue.)24 b(An)16 b(exception)h(o)q(ccurring)g(during)g(represen)o(tation)f(con)o(v)o (ersion)g(results)g(in)75 1800 y(a)f(failure)h(of)f(the)g(comm)o(unication;)g (an)h(error)e(o)q(ccurs)h(either)h(in)g(the)f(send)h(op)q(eration,)f(or)g (the)g(receiv)o(e)75 1857 y(op)q(eration,)g(or)g(b)q(oth.)166 1913 y(If)22 b(a)f(v)m(alue)i(sen)o(t)e(in)h(a)f(message)g(is)h(un)o(t)o(yp)q (ed)h(\(i.e.,)f(of)f(t)o(yp)q(e)h Ft(MPI)p 1370 1913 14 2 v 15 w(BYTE)p FA(\),)g(then)g(the)f(binary)75 1970 y(represen)o(tation)15 b(of)f(the)h(b)o(yte)g(stored)f(at)g(the)h(receiv)o(er)h(is)f(iden)o(tical)i (to)d(the)h(binary)h(represen)o(tation)e(of)75 2026 y(the)f(b)o(yte)g(loaded) h(at)e(the)h(sender.)20 b(This)14 b(holds)f(true,)g(whether)g(sender)h(and)f (receiv)o(er)h(run)f(in)h(the)f(same)75 2083 y(or)i(in)h(distinct)g(en)o (vironmen)o(ts;)f(no)g(represen)o(tation)h(con)o(v)o(ersion)f(is)h(required.) 166 2139 y(Note)d(that)g(no)g(con)o(v)o(ersion)h(ev)o(er)f(o)q(ccurs)h(when)g (an)f(MPI)g(program)g(executes)h(in)g(a)f(homogeneous)75 2195 y(system,)i(where)h(all)g(pro)q(cesses)g(run)g(in)g(the)g(same)f(en)o (vironmen)o(t.)22 b(Also)16 b(note)f(the)h(di\013eren)o(t)f(b)q(eha)o(vior)75 2252 y(of)e Ft(MPI)p 210 2252 V 15 w(BYTE)h FA(and)f(of)g Ft(MPI)p 575 2252 V 15 w(CHAR)p FA(.)h(A)f(bu\013er)g(descriptor)g(en)o(try)g(with)g (datat)o(yp)q(e)g(of)f Ft(MPI)p 1660 2252 V 16 w(CHAR)i FA(can)75 2308 y(only)f(matc)o(h)f(a)g(C)g(v)m(ariable)i(of)e(t)o(yp)q(e)h Fp(CHAR)p FA(;)e(and)i(represen)o(tation)g(con)o(v)o(ersion)f(ma)o(y)g(o)q (ccur)h(when)g(v)m(alues)75 2365 y(of)g(t)o(yp)q(e)h Ft(MPI)p 311 2365 V 15 w(CHAR)h FA(are)e(transferred.,)g(e.g.,)f(from)h(an)g(EBCDIC)h (enco)q(ding)h(to)e(an)g(ASCI)q(I)i(enco)q(ding.)166 2421 y(Consider)h(the)f (previous)h(three)f(examples.)166 2478 y(The)c(\014rst)f(program)f(is)i (correct,)g(assuming)g(b)q(oth)f(sender)i(and)e(receiv)o(er)i(declared)f(ten) g(consecutiv)o(e)75 2534 y(real)19 b(v)m(ariables)h(in)g(storage)e(starting)g (at)g Fp(a\(1\))p FA(.)30 b(If)19 b(the)g(sender)h(and)f(receiv)o(er)g (execute)h(in)g(di\013eren)o(t)75 2591 y(en)o(vironmen)o(ts,)12 b(then)f(the)g(ten)f(real)i(v)m(alues)f(that)g(are)f(fetc)o(hed)h(from)f(the) h(send)h(bu\013er)e(will)j(b)q(e)e(con)o(v)o(erted)75 2647 y(to)g(the)h(represen)o(tation)f(for)g(reals)h(on)f(the)h(receiv)o(er)g(site) g(b)q(efore)g(they)g(are)f(stored)g(in)h(the)g(receiv)o(e)g(bu\013er.)75 2704 y(While)19 b(the)e(n)o(um)o(b)q(er)h(of)f(real)g(elemen)o(ts)h(fetc)o (hed)g(from)f(the)g(send)h(bu\013er)f(equal)i(the)e(n)o(um)o(b)q(er)h(of)f (real)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 11 13 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(11)75 49 y(elemen)o(ts)14 b(stored)e(in)i(the)f(receiv)o(e)h (bu\013er,)f(the)g(n)o(um)o(b)q(er)g(of)g(b)o(ytes)g(stored)f(need)i(not)f (equal)h(the)f(n)o(um)o(b)q(er)75 106 y(of)f(b)o(ytes)g(loaded:)19 b(e.g.)g(the)12 b(sender)h(ma)o(y)e(use)i(a)f(four)g(b)o(yte)g(represen)o (tation)h(and)f(the)g(receiv)o(er)i(an)e(eigh)o(t)75 162 y(b)o(yte)17 b(represen)o(tation)h(for)f(reals.)27 b(If)18 b(the)g(send)g(or)f(receiv)o(e) h(bu\013er)g(do)f(not)h(con)o(tain)f(ten)h(consecutiv)o(e)75 219 y(real)e(v)m(ariables,)g(then)f(the)g(program)g(is)g(erroneous,)g(and)g (its)h(b)q(eha)o(vior)f(is)h(unde\014ned.)166 275 y(The)f(second)h(program)e (is)i(erroneous,)f(and)g(its)g(b)q(eha)o(vior)h(is)g(unde\014ned.)166 332 y(The)e(third)g(program)e(is)i(correct.)19 b(The)13 b(exact)g(same)g (sequence)i(of)e(fort)o(y)f(b)o(ytes)h(that)g(w)o(ere)g(loaded)75 388 y(from)18 b(the)h(send)h(bu\013er)f(will)i(b)q(e)e(stored)g(in)h(the)f (receiv)o(e)h(bu\013er,)f(ev)o(en)h(if)f(sender)h(and)f(receiv)o(er)h(run)75 444 y(in)e(a)g(di\013eren)o(t)f(en)o(vironmen)o(t.)28 b(The)18 b(message)f(sen)o(t)g(has)h(exactly)f(the)h(same)f(length)i(\(in)f(b)o (ytes\))f(and)75 501 y(the)f(same)f(binary)h(represen)o(tation)f(as)g(the)h (message)f(receiv)o(ed.)22 b(If)16 b(the)f(v)m(ariables)i(in)f(the)g(send)g (bu\013er)75 557 y(are)i(of)h(di\013eren)o(t)g(t)o(yp)q(es)g(as)f(the)h(v)m (ariables)h(in)g(the)f(receiv)o(e)g(bu\013er,)h(or)e(they)h(are)f(of)h(the)f (same)h(t)o(yp)q(e)75 614 y(but)e(di\013eren)o(t)g(data)g(represen)o(tations) f(are)h(used,)h(then)f(the)g(bits)h(stored)e(in)i(the)f(receiv)o(e)h (bu\013er)f(ma)o(y)75 670 y(enco)q(de)f(v)m(alues)h(that)d(are)h(di\013eren)o (t)g(from)g(the)g(v)m(alues)h(they)g(enco)q(ded)g(in)g(the)g(send)f (bu\013er.)166 727 y(Data)f(represen)o(tation)i(con)o(v)o(ersion)f(also)h (applies)h(to)e(the)g(en)o(v)o(elop)q(e)i(of)e(a)g(message:)20 b(source,)15 b(des-)75 783 y(tination)h(and)f(tag)f(are)h(all)h(in)o(tegers)g (that)e(ma)o(y)h(need)h(to)e(b)q(e)i(con)o(v)o(erted.)166 840 y(The)k(curren)o(t)g(draft)f(do)q(es)h(not)g(pro)o(vide)g(a)g(mec)o(hanism)h (for)e(in)o(ter-language)i(comm)o(unication:)75 896 y(messages)d(sen)o(t)h(b) o(y)f(F)l(ortran)g(calls)i(should)f(b)q(e)h(receiv)o(ed)g(b)o(y)e(F)l(ortran) g(calls)h(and)g(messages)g(sen)o(t)f(b)o(y)75 953 y(C)e(calls)g(should)h(b)q (e)g(receiv)o(ed)g(b)o(y)e(C)h(calls)h(\(this)f(follo)o(ws)f(from)h(the)f (requiremen)o(ts)i(of)e(t)o(yp)q(e)h(matc)o(hing)75 1009 y(and)g(the)h(fact)f (that)f(F)l(ortran)g(and)i(C)f(datat)o(yp)q(es)g(are)g(distinct\).)24 b(If)16 b(in)o(ter-language)h(comm)o(unication)75 1065 y(is)g(needed)i(then)e (one)g(needs)h(to)e(in)o(v)o(ok)o(e)h(C)f(comm)o(unication)i(routines)f(from) f(a)h(F)l(ortran)f(program)f(or)75 1122 y(F)l(ortran)f(comm)o(unication)i (routines)g(from)e(a)h(C)g(program.)166 1254 y Fn(Implemen)o(tati)o(on)d (note:)166 1304 y Fs(The)k(curren)o(t)i(de\014nition)e(do)q(es)h(not)f (require)h(messages)f(to)g(carry)h(data)f(t)o(yp)q(e)g(information.)22 b(A)16 b(message)75 1354 y(can)c(b)q(e)h(comp)q(osed)f(and)g(sen)o(t)h(using) f(only)g(the)h(information)c(pro)o(vided)j(b)o(y)g(the)h(send)g(call,)e(and)h (can)h(b)q(e)g(receiv)o(ed)75 1404 y(and)k(stored)g(using)g(only)f(the)h (information)d(pro)o(vided)j(b)o(y)g(the)g(receiv)o(e)h(call.)26 b(If)17 b(messages)g(are)g(sen)o(t)h(b)q(et)o(w)o(een)75 1454 y(di\013eren)o(t)c(mac)o(hines)f(then)g(one)h(can)f(either)h(use)g(a)f (\\univ)o(ersal")g(data)g(enco)q(ding)g(for)g(messages,)g(use)h(kno)o(wledge) 75 1503 y(of)g(the)i(receiv)o(er)h(en)o(vironmen)o(t)d(in)g(order)i(to)f(con) o(v)o(ert)h(data)e(at)h(the)h(sender,)h(or)e(use)g(kno)o(wledge)g(of)g(the)g (sender)75 1553 y(en)o(vironmen)o(t)j(in)g(order)i(to)f(con)o(v)o(ert)g(data) g(at)g(the)g(receiv)o(er.)35 b(In)19 b(either)g(case)h(the)g(datat)o(yp)q(e)f (parameter)f(in)75 1603 y(the)f(lo)q(cal)f(call)g(can)g(b)q(e)h(used)h(to)e (deriv)o(e)h(the)g(t)o(yp)q(es)h(of)e(the)h(v)n(alues)f(transferred.)28 b(Ho)o(w)o(ev)o(er,)17 b(additional)d(t)o(yp)q(e)75 1653 y(information)d (carried)k(b)o(y)e(messages)h(can)g(pro)o(vide)g(b)q(etter)h(error)g (detection.)166 1868 y Fn(Discussion:)166 1918 y Fs(MPI)f(do)q(es)h(not)f (handle)g(in)o(ter-language)g(comm)o(unicatio)o(n)e(b)q(ecause)k(there)f (seem)f(to)g(b)q(e)h(no)f(agreed)h(stan-)75 1968 y(dards)f(for)g(in)o (ter-language)f(t)o(yp)q(e)h(con)o(v)o(ersions.)19 b(The)c(curren)o(t)g (design)f(means)f(that)h(in)o(ter-language)f(comm)o(uni-)75 2017 y(cation)h(relies)g(on)g(the)g(v)o(endor)g(sp)q(eci\014c)i(rules)e(for)g (in)o(ter-language)f(calls)g(at)h(eac)o(h)g(site.)75 2243 y Fx(1.7)59 b(Communication)17 b(Mo)r(des)75 2345 y FA(The)12 b(basic)h(send)g(op)q(eration)g(describ)q(ed)h(in)f(Section)g(1.2)e(used)i (the)f Fw(standard)h FA(comm)o(unication)g(mo)q(de.)75 2401 y(In)h(suc)o(h)g(a)f(comm)o(unication)h(mo)q(de,)g(a)f(send)h(op)q(eration)g (can)f(b)q(e)h(started)f(whether)h(or)f(not)g(a)g(matc)o(hing)75 2458 y(receiv)o(e)g(w)o(as)e(p)q(osted.)20 b(The)12 b(completion)h(of)f(the)g (send)h(op)q(eration)f(indicates)i(that)d(the)i(message)e(and)i(its)75 2514 y(en)o(v)o(elop)q(e)j(ha)o(v)o(e)e(b)q(een)i(safely)f(stored)f(a)o(w)o (a)o(y)g(and)g(that)h(the)f(sender)i(is)f(free)g(to)f(access)h(and)g(mo)q (dify)g(the)75 2571 y(sender)g(bu\013er.)k(Th)o(us,)14 b(the)g(op)q(eration)h (is)f Fw(blo)q(c)o(king)p FA(:)21 b(it)14 b(do)q(es)h(not)e(return)i(un)o (til)g(the)f(send)h(op)q(eration)75 2627 y(has)g(completed)h Fw(lo)q(cally)p FA(,)h(on)f(the)f(sender)h(side.)166 2684 y(The)f(completion) i(of)e(a)g(send)h(op)q(eration)f(giv)o(es)h(no)f(indication)i(that)e(the)g (message)g(w)o(as)g(receiv)o(ed)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 14 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(12)75 49 y(on)15 b(the)h(receiv)o(er)g(side.)21 b(A)15 b(blo)q(c)o(king)i(send)f(ma)o(y)e(b)q(e)i(implemen)o(ted)h(so)e(that) g(a)g(blo)q(c)o(king)h(send)g(returns)75 106 y(only)g(after)f(a)g(matc)o (hing)g(receiv)o(e)h(has)g(b)q(een)g(executed)g(on)g(the)f(receiv)o(er)h (side.)22 b(This)16 b(a)o(v)o(oids)f(the)g(need)75 162 y(to)i(bu\013er)g (message)g(data)g(out)g(of)g(sender)h(or)f(receiv)o(er)h(memory)l(.)26 b(In)18 b(this)g(case)f(the)h(send)g(op)q(eration)75 219 y(completes)j(only)f (after)f(the)h(matc)o(hing)g(receiv)o(e)h(has)f(started)f(executing.)35 b(On)20 b(the)g(other)g(hand,)h(it)75 275 y(is)g(also)f(p)q(ossible)i(for)e (MPI)g(to)g(bu\013er)g(messages,)h(so)f(as)g(to)f(allo)o(w)i(the)f(sender)h (to)f(pro)q(ceed)h(ahead)75 332 y(of)e(the)h(receiv)o(er.)35 b(In)20 b(suc)o(h)g(a)f(case)h(the)g(send)g(op)q(eration)g(ma)o(y)g(complete) g(successfully)i(b)q(efore)e(the)75 388 y(message)c(w)o(as)g(receiv)o(ed.)26 b(Th)o(us,)16 b(the)h(basic)h(send)f(op)q(eration)g(describ)q(ed)h(in)g(the)f (previous)g(section)g(is)75 444 y Fw(async)o(hronous)p FA(,)12 b(since)i(its)f(return)g(do)q(es)h(not)f(imply)h(a)f(sync)o(hronization)h (with)f(the)g(\(remote\))f(receiv)o(e)75 501 y(op)q(eration,)j(and)g(do)q(es) h(not)f(imply)h Fw(global)h FA(termination)f(of)f(the)g(comm)o(unication.)166 557 y(There)g(are)g(t)o(w)o(o)f(additional)j(comm)o(unication)f(mo)q(des:)166 614 y(A)h(send)g(that)f(uses)i(the)f Fw(ready)f FA(comm)o(unication)i(mo)q (de)f(ma)o(y)f(b)q(e)h(started)g(only)g(if)g(a)g(matc)o(hing)75 670 y(receiv)o(e)d(is)f(already)g(p)q(osted;)g(otherwise)g(the)g(op)q (eration)g(is)g(erroneous)f(and)h(its)g(outcome)f(is)h(unde\014ned.)75 727 y(In)19 b(some)f(systems,)h(this)f(allo)o(ws)h(the)f(remo)o(v)m(al)h(of)f (a)g(hand-shak)o(e)h(op)q(eration)g(that)e(is)i(otherwise)g(re-)75 783 y(quired,)g(and)f(results)g(in)g(impro)o(v)o(ed)g(p)q(erformance.)27 b(The)18 b(completion)h(of)e(the)h(send)g(op)q(eration)g(do)q(es)75 840 y(not)e(dep)q(end)i(on)e(the)g(status)f(of)h(a)g(matc)o(hing)g(receiv)o (e,)h(and)f(merely)h(indicates)h(that)d(the)h(send)h(bu\013er)75 896 y(can)i(b)q(e)g(reused.)30 b(A)19 b(send)g(op)q(eration)g(that)f(uses)g (the)h(ready)f(mo)q(de)h(has)g(the)f(same)g(seman)o(tics)h(as)f(a)75 953 y(standard)g(send)h(op)q(eration.)29 b(It)18 b(is)h(merely)g(that)f(the)g (sender)h(pro)o(vides)f(additional)i(information)f(to)75 1009 y(the)c(comm)o(unication)h(subsystem,)f(when)h(using)g(the)f(ready)g(mo)q (de.)166 1065 y(A)j(send)g(that)f(uses)g(the)h Fw(sync)o(hronous)e FA(mo)q(de)i(can)g(b)q(e)g(started)f(whether)g(or)g(not)h(a)f(matc)o(hing)75 1122 y(receiv)o(e)12 b(w)o(as)e(p)q(osted.)19 b(Ho)o(w)o(ev)o(er,)10 b(the)h(send)h(will)g(complete)g(successfully)h(only)f(if)f(a)g(matc)o(hing)g (receiv)o(e)g(is)75 1178 y(p)q(osted,)h(and)g(the)g(receiv)o(e)g(op)q (eration)g(has)f(started)g(to)g(receiv)o(e)i(the)e(message)g(sen)o(t)h(b)o(y) f(the)h(sync)o(hronous)75 1235 y(send.)29 b(\(I.e.,)19 b(the)f(receiv)o(e)h (has)f(b)q(een)h(p)q(osted,)g(and)g(the)f(incoming)h(message)f(has)g(b)q(een) i(matc)o(hed)e(to)75 1291 y(this)g(p)q(osted)g(receiv)o(e.\))28 b(Th)o(us,)18 b(the)g(completion)g(of)g(a)f(sync)o(hronous)h(send)g(not)f (only)i(indicates)g(that)75 1348 y(the)g(send)h(bu\013er)f(can)g(b)q(e)h (reused,)g(but)f(also)g(indicates)i(that)d(the)i(receiv)o(er)f(has)g(reac)o (hed)h(a)f(certain)75 1404 y(p)q(oin)o(t)e(in)h(its)f(execution,)h(namely)g (that)e(it)h(has)g(started)f(executing)i(the)f(matc)o(hing)g(receiv)o(e.)26 b(If)17 b(b)q(oth)75 1461 y(sends)f(and)h(receiv)o(es)f(are)g(blo)q(c)o(king) h(op)q(erations)f(then)h(the)f(use)g(of)f(the)h(sync)o(hronous)g(mo)q(de)g (pro)o(vides)75 1517 y(sync)o(hronous)f(comm)o(unication)h(seman)o(tics:)k(a) 14 b(comm)o(unication)i(do)q(es)f(not)g(complete)g(at)g(either)g(ends)75 1574 y(b)q(efore)d(b)q(oth)g(pro)q(cesses)g(\\attend")f(to)g(the)h(comm)o (unication;)h(the)f(completion)h(of)e(a)h(sync)o(hronous)f(send)75 1630 y(is)16 b(a)f Fw(global)i FA(ev)o(en)o(t.)166 1686 y(Tw)o(o)22 b(additional)j(send)e(functions)h(are)f(pro)o(vided)h(for)f(the)g(t)o(w)o(o)f (additional)i(comm)o(unication)75 1743 y(mo)q(des.)32 b(The)19 b(comm)o(unication)h(mo)q(de)f(is)h(indicated)g(b)o(y)f(a)g(one)g(letter)h (pre\014x:)28 b Fp(R)18 b FA(for)h(ready)g(and)g Fp(S)75 1799 y FA(for)c(sync)o(hronous.)166 1856 y(Send)h(in)g(ready)f(mo)q(de)75 1960 y Ft(MPI)p 160 1960 14 2 v 16 w(RSEND)h(\(sta)o(rt,)f(count,)h(datat)o (yp)q(e,)g(dest,)g(tag,)f(comm)m(\))117 2037 y Fs(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 2112 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o(teger\))117 2187 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 2262 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 2337 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2412 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 2537 y FA(Send)16 b(in)g(sync)o(hronous)f(mo)q(de)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 13 15 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(13)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(SSEND)16 b(\(sta)o(rt,)e(count,)j(datat)o(yp)q(e,)f(dest,)g(tag,)f(comm)m(\))117 126 y Fs(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 202 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 277 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 352 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 427 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 502 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 627 y FA(There)16 b(is)g(only)g(one)g(receiv)o(e)h(mo)q(de,)e(whic)o(h)i(can) f(matc)o(h)f(an)o(y)g(of)h(the)f(send)i(mo)q(des.)k(The)16 b(receiv)o(e)75 683 y(op)q(eration)g(describ)q(ed)h(in)g(the)e(last)h (section)g(is)g Fw(blo)q(c)o(king)p FA(:)22 b(it)16 b(returns)f(only)h(after) f(the)h(receiv)o(e)g(bu\013er)75 740 y(con)o(tains)h(the)h(newly)g(receiv)o (ed)g(message.)26 b(It)17 b(is)h Fw(async)o(hronous)p FA(:)23 b(the)17 b(completion)i(of)d(a)h(receiv)o(e)h(is)75 796 y(a)e(lo)q(cal)i(op)q (eration,)f(and)f(a)g(receiv)o(e)i(can)f(complete)g(b)q(efore)g(the)f(matc)o (hing)h(send)g(has)f(completed)i(\(of)75 852 y(course,)d(it)g(can)h(complete) g(only)f(after)g(the)g(matc)o(hing)g(send)h(has)f(started\).)166 909 y(Comm)o(unication)e(imp)q(oses)g(an)f(order)h(on)f(the)h(ev)o(en)o(ts)f (o)q(ccurring)h(at)f(the)h(comm)o(unicating)g(no)q(des.)75 965 y(It)i(is)g(alw)o(a)o(ys)e(the)i(case)g(that)e(the)i(completion)h(of)e(a) g(receiv)o(e)i(o)q(ccurs)e(after)g(the)h(start)e(of)h(the)h(matc)o(hing)75 1022 y(send.)29 b(If)19 b(the)f(sync)o(hronous)g(send)h(mo)q(de)f(is)h(used,) g(then)g(it)f(is)h(also)f(the)g(case)g(that)g(the)g(completion)75 1078 y(of)i(a)h(send)g(o)q(ccurs)g(after)f(the)g(start)g(of)g(the)h(matc)o (hing)f(receiv)o(e.)38 b(Of)20 b(course,)i(on)f(eac)o(h)f(pro)q(cess,)i(a)75 1135 y(comm)o(unication)14 b(completes)h(after)d(it)i(is)g(started.)19 b(No)13 b(other)h(order)f(is)h(imp)q(osed)h(b)o(y)e(MPI.)g(E.g.,)g(if)h(the) 75 1191 y(standard)h(send)g(mo)q(de)h(is)f(used,)h(the)f(send)h(op)q(eration) f(ma)o(y)f(complete)i(b)q(efore)g(the)f(matc)o(hing)g(receiv)o(e)75 1248 y(w)o(as)f(started.)166 1304 y(In)h(a)f(m)o(ulti-threaded)i(implemen)o (tation)g(of)e(MPI,)h(the)f(system)g(ma)o(y)g(desc)o(hedule)j(a)d(thread)h (that)75 1361 y(is)g(blo)q(c)o(k)o(ed)h(on)f(a)g(send)g(or)f(receiv)o(e)i(op) q(eration,)f(and)g(sc)o(hedule)i(another)d(thread)h(for)f(execution)i(in)g (the)75 1417 y(same)f(address)h(space.)21 b(In)16 b(suc)o(h)g(a)f(case)g(it)h (the)g(user)f(resp)q(onsibilit)o(y)j(not)d(to)g(mo)q(dify)h(the)g(send)g (bu\013er)75 1473 y(un)o(til)k(the)f(send)h(completes)f(and)g(not)g(to)f (access)h(or)g(mo)q(dify)g(the)g(receiv)o(e)h(bu\013er)f(un)o(til)h(the)f (receiv)o(e)75 1530 y(completes;)c(otherwise)h(the)f(outcome)g(of)g(the)g (computation)g(is)h(unde\014ned.)166 1662 y Fn(Implemen)o(tati)o(on)10 b(note:)34 b Fs(A)13 b(ready)f(send)h(can)f(b)q(e)h(implemen)o(ted)d(as)i(a)g (standard)h(send;)g(in)e(suc)o(h)i(a)f(case)75 1712 y(there)j(will)e(b)q(e)h (no)g(p)q(erformance)g(adv)n(an)o(tage)f(\(or)h(disadv)n(an)o(tage\))f(for)g (the)i(use)f(of)g(ready)g(send.)166 1762 y(A)f(standard)h(send)h(can)e(b)q(e) h(implemen)o(ted)e(as)h(a)g(sync)o(hronous)i(send.)j(In)c(suc)o(h)g(a)f (case,)h(no)f(data)h(bu\013ering)75 1812 y(is)g(needed.)166 1862 y(A)g(p)q(ossible)g(comm)o(unication)c(proto)q(col)k(for)g(the)g(v)n (arious)f(comm)o(unication)e(mo)q(des)i(is)h(outlined)f(b)q(elo)o(w:)166 1911 y Fk(ready)21 b(send)p Fs(:)c(The)d(message)g(is)f(sen)o(t)i(as)f(so)q (on)g(as)g(p)q(ossible.)166 1961 y Fk(synchronous)19 b(send:)29 b Fs(The)20 b(sender)g(sends)h(a)e(request-to-send)j(message.)34 b(The)19 b(receiv)o(er)i(stores)g(this)75 2011 y(request.)f(When)15 b(a)f(matc)o(hing)e(receiv)o(e)j(is)f(p)q(osted,)h(the)g(receiv)o(er)h(sends) f(bac)o(k)f(a)g(p)q(ermission-to-send)g(message,)75 2061 y(and)g(the)g (sender)i(no)o(w)d(sends)i(the)g(message.)166 2111 y Fk(standard)20 b(send:)e Fs(First)d(proto)q(col)f(ma)o(y)e(b)q(e)j(used)h(for)d(short)i (messages,)f(and)g(second)i(proto)q(col)e(for)g(long)75 2160 y(messages.)166 2210 y(Additional)h(con)o(trol)h(messages)h(migh)o(t)e(b)q(e) i(needed)h(for)e(\015o)o(w)g(con)o(trol)h(and)f(error)i(reco)o(v)o(ery)m(.)27 b(Of)16 b(course,)75 2260 y(there)f(are)f(man)o(y)f(other)h(p)q(ossible)g(c)o (hoices.)75 2486 y Fx(1.8)59 b(Nonblo)r(cking)20 b(communication)75 2587 y FA(One)e(can)g(impro)o(v)o(e)f(p)q(erformance)g(on)h(man)o(y)e (systems)h(b)o(y)h(o)o(v)o(erlapping)f(comm)o(unication)i(and)e(com-)75 2644 y(putation.)35 b(This)21 b(is)g(esp)q(ecially)i(true)d(on)g(systems)g (where)g(comm)o(unication)h(can)g(b)q(e)g(executed)g(au-)75 2700 y(tonomously)12 b(b)o(y)h(an)g(in)o(telligen)o(t)h(comm)o(unication)g (con)o(troller.)19 b(Ligh)o(t-w)o(eigh)o(t)13 b(threads)f(are)h(one)g(mec)o (h-)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 16 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(14)75 49 y(anism)21 b(for)g(ac)o(hieving)h(suc)o(h)g(o)o(v)o (erlap.)37 b(An)21 b(alternativ)o(e)h(mec)o(hanism)g(that)e(often)h(leads)g (to)g(b)q(etter)75 106 y(p)q(erformance)e(is)g(to)f(use)h Fw(non)o(blo)q(c)o (king)j(comm)o(unication)p FA(.)32 b(A)19 b(non)o(blo)q(c)o(king)h(send)f (call)h(initiates)75 162 y(the)15 b(send)h(op)q(eration,)f(but)h(do)q(es)f (not)g(complete)h(it.)k(The)15 b(send)h(call)h(will)f(return)g(b)q(efore)f (the)g(message)75 219 y(w)o(as)d(copied)h(out)f(of)g(the)g(send)h(bu\013er.) 19 b(A)13 b(separate)e(call)j(is)f(needed)g(to)f(complete)h(the)g(comm)o (unication,)75 275 y(i.e.)20 b(to)13 b(v)o(erify)i(that)e(the)h(data)g(has)g (b)q(een)h(copied)g(out)f(of)g(the)g(send)h(bu\013er.)k(With)c(suitable)g (hardw)o(are,)75 332 y(the)k(transfer)f(of)h(data)f(out)h(of)g(the)g(sender)g (memory)g(ma)o(y)f(pro)q(ceed)i(concurren)o(tly)g(with)f(computa-)75 388 y(tions)g(done)g(at)f(the)g(sender)h(after)f(the)h(send)g(w)o(as)f (initiated)i(and)e(b)q(efore)h(it)g(completed.)31 b(Similarly)l(,)75 444 y(a)19 b(non)o(blo)q(c)o(king)h(receiv)o(e)g(initiates)h(the)e(receiv)o (e)h(op)q(eration,)g(but)f(do)q(es)g(not)g(complete)h(it.)32 b(The)19 b(call)75 501 y(will)f(return)f(b)q(efore)g(a)f(message)g(is)i (stored)e(in)o(to)g(the)h(receiv)o(e)h(bu\013er.)24 b(A)17 b(separate)f(call)h(is)h(needed)g(to)75 557 y(complete)f(the)f(receiv)o(e)i (op)q(eration)e(and)g(v)o(erify)h(that)e(the)i(data)e(has)i(b)q(een)g(receiv) o(ed)g(in)o(to)g(the)f(receiv)o(e)75 614 y(bu\013er.)k(With)14 b(suitable)h(hardw)o(are,)f(the)g(transfer)f(of)h(data)g(in)o(to)g(the)g (receiv)o(er)h(memory)f(ma)o(y)f(pro)q(ceed)75 670 y(concurren)o(tly)f(with)g (computations)f(done)g(after)g(the)g(receiv)o(e)h(w)o(as)f(initiated)i(and)e (b)q(efore)h(it)f(completed.)75 727 y(The)17 b(use)h(of)f(non)o(blo)q(c)o (king)h(receiv)o(es)g(ma)o(y)f(also)g(a)o(v)o(oid)g(system)f(bu\013ering)i (and)f(memory)g(to)g(memory)75 783 y(cop)o(ying,)e(as)g(information)g(is)h (pro)o(vided)g(early)g(on)f(the)g(lo)q(cation)h(of)f(the)g(receiv)o(e)h (bu\013er.)166 840 y(Non)o(blo)q(c)o(king)g(sends)g(can)g(use)f(the)h(same)f (three)g(mo)q(des)h(as)f(blo)q(c)o(king)h(sends:)21 b Fp(standard)p FA(,)14 b Fp(ready)75 896 y FA(and)21 b Fp(synchronous)p FA(.)35 b(These)21 b(carry)g(the)g(same)f(meaning.)37 b(The)21 b(initiation)i(and)e (completion)h(of)e(a)75 953 y(standard)e(send)i(do)f(not)f(dep)q(end)j(on)d (the)h(status)f(of)h(a)f(matc)o(hing)h(receiv)o(e.)32 b(A)19 b(ready)g(send)g(can)g(b)q(e)75 1009 y(initiated)12 b(only)f(if)g(a)g(matc)o (hing)f(receiv)o(e)i(has)e(already)h(b)q(een)h(initiated,)g(otherwise)f(the)g (call)g(is)g(erroneous;)75 1065 y(its)18 b(completion)g(do)q(es)g(not)f(dep)q (end)i(on)f(the)f(status)g(of)g(a)g(matc)o(hing)h(receiv)o(e.)27 b(A)18 b(sync)o(hronous)f(send)75 1122 y(can)f(b)q(e)h(initiated)h(b)q(efore) f(a)f(matc)o(hing)g(receiv)o(e)h(has)f(b)q(een)i(initiated,)g(but)e(will)i (complete)f(only)g(after)75 1178 y(a)f(matc)o(hing)f(receiv)o(e)i(has)f(b)q (een)h(initiated,)g(and)f(has)g(started)f(receiving)j(the)e(message)f (generated)h(b)o(y)75 1235 y(the)f(send)h(op)q(eration.)166 1291 y(In)f(all)h(cases)e(the)h(op)q(eration)g(that)f(initiates)i(the)f(comm) o(unication)g(\(send)g(or)f(receiv)o(e\))i(is)f(a)f(lo)q(cal,)75 1348 y(non)o(blo)q(c)o(king)19 b(call)f(that)f(returns)g(as)g(so)q(on)g(as)g (p)q(ossible.)29 b(Suc)o(h)18 b(call)g(ma)o(y)f(fail)h(b)q(ecause)g(b)q (ecause)h(the)75 1404 y(system)f(has)h(exhausted)g(a)o(v)m(ailable)h (resources)f(\(e.g.,)f(exceeded)i(the)f(limit)h(on)f(n)o(um)o(b)q(er)g(of)f (p)q(ending)75 1461 y(comm)o(unications)c(p)q(er)g(no)q(de\).)19 b(Go)q(o)q(d)14 b(qualit)o(y)g(implemen)o(tations)g(of)f(MPI)h(will)h(set)e (these)g(limits)i(high)75 1517 y(enough)h(so)e(that)h(a)g(non)o(blo)q(c)o (king)h(comm)o(unication)g(initiation)h(call)g(will)g(fail)f(only)g(in)g (\\pathological")75 1574 y(cases.)166 1630 y(Non)o(blo)q(c)o(king)g(sends)g (can)f(b)q(e)h(matc)o(hed)f(with)h(blo)q(c)o(king)g(receiv)o(es,)g(and)f (vice-v)o(ersa.)75 1752 y Fr(1.8.1)49 b(Comm)o(unication)14 b(Objects)75 1838 y FA(Non)o(blo)q(c)o(king)f(comm)o(unications)g(use)f (opaque)g(comm)o(unication)h(ob)s(jects)f(to)f(iden)o(tify)i(comm)o (unication)75 1894 y(op)q(erations)18 b(and)h(matc)o(h)e(the)i(op)q(eration)f (that)f(initiates)j(the)e(comm)o(unication)h(with)f(the)h(op)q(eration)75 1950 y(that)e(terminates)i(it.)29 b(These)18 b(are)g(system)g(ob)s(jects)g (that)f(are)h(accessed)h(via)f(a)g(handle.)30 b(An)19 b(opaque)75 2007 y(comm)o(unication)f(ob)s(ject)e(iden)o(ti\014es)j(v)m(arious)f(prop)q (erties)f(of)g(a)g(comm)o(unication)h(op)q(eration,)f(suc)o(h)g(as)75 2063 y(the)g(\(send)g(or)f(receiv)o(e\))h(bu\013er)g(that)f(is)h(asso)q (ciated)g(with)g(it,)g(its)g(con)o(text,)f(the)g(tag)g(and)h(destination)75 2120 y(parameters)c(to)f(b)q(e)j(used)f(for)e(a)i(send,)g(or)f(the)g(tag)g (and)g(source)h(parameters)e(to)h(b)q(e)h(used)g(for)f(a)g(receiv)o(e.)75 2176 y(In)j(addition,)g(this)g(ob)s(ject)f(stores)f(information)i(ab)q(out)f (the)g(status)g(of)g(the)g(p)q(ending)i(comm)o(unication)75 2233 y(op)q(eration)e(that)g(is)h(p)q(erformed)f(with)h(this)f(ob)s(ject.)75 2354 y Fr(1.8.2)49 b(Comm)o(unication)14 b(initiation)75 2440 y FA(W)l(e)k(use)g(the)g(same)g(naming)g(con)o(v)o(en)o(tions)g(as)f(for)g (blo)q(c)o(king)j(comm)o(unication:)26 b(a)17 b(pre\014x)h(of)g Fp(R)f FA(\()p Fp(S)p FA(\))g(is)75 2497 y(used)h(for)f Fp(READY)g FA(\()p Fp(SYNCHRONOUS)p FA(\))f(mo)q(de.)27 b(In)19 b(addition)f(a)g (pre\014x)g(of)f Fp(I)h FA(\(for)e Fp(IMMEDIATE)p FA(\))h(indicates)75 2553 y(that)d(the)i(call)g(is)g(non)o(blo)q(c)o(king.)166 2610 y(Initiate)g(a)f(standard)g(mo)q(de)g(non)o(blo)q(c)o(king)i(comm)o (unication.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 15 17 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(15)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(ISEND\(handle,)16 b(sta)o(rt,)f(count,)h(datat)o(yp)q(e,)g(dest,)h(tag,)d(comm)n(\))117 126 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j (\(handle\))117 202 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 277 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h (bu\013er)f(\(in)o(teger\))117 352 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 427 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 502 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 577 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 702 y FA(Initiate)16 b(a)f(ready)g(mo)q(de)h(non)o(blo)q(c)o(king)g(comm)o (unication.)75 805 y Ft(MPI)p 160 805 V 16 w(IRSEND\(handle,)g(sta)o(rt,)f (count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(\))117 883 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j (\(handle\))117 958 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 1033 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h (bu\013er)f(\(in)o(teger\))117 1108 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 1183 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 1258 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 1333 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 1458 y FA(Initiate)16 b(a)f(sync)o(hronous)g(mo)q(de)h(non)o(blo)q(c)o(king)g (comm)o(unication.)75 1562 y Ft(MPI)p 160 1562 V 16 w(ISSEND\(handle,)g(sta)o (rt,)f(count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(\))117 1639 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))117 1714 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 1789 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in) h(send)h(bu\013er)f(\(in)o(teger\))117 1864 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)117 1939 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 2014 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2090 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 2214 y FA(Initiate)16 b(a)f(non)o(blo)q(c)o(king)i(receiv)o(e.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 18 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(16)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(IRECV)15 b(\(handle,)h(sta)o(rt,)f(count,)h(datat)o(yp)q(e,)h(source,)e(tag,)g(comm)m (\))117 126 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))117 202 y(OUT)108 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 277 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h (receiv)o(e)h(bu\013er)g(\(in)o(teger\))117 352 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h (bu\013er)g(elemen)o(t)117 427 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o(teger\))117 502 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 577 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 702 y FA(These)19 b(calls)h(allo)q(cate)f(a)f(comm)o(unication)i(ob)s(ject)e(and)h(asso)q (ciates)f(it)h(with)g(the)g(handle.)32 b(The)75 758 y(handle)20 b(can)e(b)q(e)i(used)f(later)f(to)g(query)h(ab)q(out)f(the)h(status)f(of)g (the)g(comm)o(unication)i(or)e(w)o(ait)g(for)g(its)75 815 y(completion.)166 871 y(A)f(non)o(blo)q(c)o(king)i(send)f(call)g(indicates)h(that)e(the)g (system)g(ma)o(y)g(start)f(cop)o(ying)h(data)g(out)g(of)g(the)75 928 y(send)f(bu\013er.)21 b(The)15 b(sender)h(should)h(not)e(up)q(date)h(an)o (y)f(part)g(of)g(the)g(send)h(bu\013er)g(after)f(a)g(non)o(blo)q(c)o(king)75 984 y(send)h(op)q(eration)f(returns,)g(un)o(til)h(the)g(send)f(completes.)166 1040 y(A)k(non)o(blo)q(c)o(king)i(receiv)o(e)f(call)g(indicates)h(that)e(the) g(sytem)g(ma)o(y)f(start)g(writing)i(data)f(in)o(to)g(the)75 1097 y(receiv)o(e)c(bu\013er.)20 b(The)15 b(receiv)o(er)g(should)g(not)f (access)h(or)f(up)q(date)h(an)o(y)f(part)f(of)h(the)h(receiv)o(e)g(bu\013er)g (after)75 1153 y(a)g(non)o(blo)q(c)o(king)h(receiv)o(e)h(op)q(eration)e (returns,)g(un)o(til)h(the)f(receiv)o(e)h(completes.)75 1275 y Fr(1.8.3)49 b(Comm)o(unication)14 b(Completion)75 1361 y FA(The)j(functions)g Ft(MPI)p 452 1361 V 16 w(W)l(AIT)f FA(and)g Ft(MPI)p 768 1361 V 16 w(TEST)h FA(are)f(used)h(to)f(complete)h(a)f(non)o (blo)q(c)o(king)h(comm)o(unica-)75 1417 y(tion.)27 b(The)17 b(completion)i(of)e(a)g(send)h(op)q(eration)f(indicates)i(that)e(the)g (sender)h(is)g(no)o(w)f(free)h(to)e(up)q(date)75 1474 y(the)h(lo)q(cations)h (in)g(the)f(send)g(bu\013er,)h(or)e(an)o(y)h(other)f(lo)q(cation)i(that)e (can)i(b)q(e)f(referenced)h(b)o(y)f(the)g(send)75 1530 y(op)q(eration)i (\(the)g(send)g(op)q(eration)g(itself)h(lea)o(v)o(es)e(the)h(con)o(ten)o(t)f (of)h(the)g(send)g(bu\013er)g(unc)o(hanged\).)31 b(It)75 1587 y(do)q(es)19 b(not)f(indicate)j(that)d(the)h(message)f(has)g(b)q(een)i (receiv)o(ed;)i(rather,)c(it)h(ma)o(y)f(ha)o(v)o(e)h(b)q(een)h(bu\013ered)75 1643 y(b)o(y)f(the)g(comm)o(unication)h(subsystem.)32 b(Ho)o(w)o(ev)o(er,)19 b(if)h(a)f Fp(synchronous)f FA(mo)q(de)h(send)h(w)o(as)e(used,)j(the)75 1700 y(completion)c(of)e(the)g(send)h(op)q(eration)g(indicates)h(that)e(a)g (matc)o(hing)g(receiv)o(e)i(w)o(as)d(initiated,)j(and)f(that)75 1756 y(the)f(message)g(will)i(ev)o(en)o(tually)f(b)q(e)g(receiv)o(ed)g(b)o(y) f(this)h(matc)o(hing)f(receiv)o(e.)166 1813 y(The)20 b(completion)h(of)e(a)g (receiv)o(e)i(op)q(eration)f(indicates)h(that)e(the)h(receiv)o(e)g(bu\013er)g (con)o(tains)g(the)75 1869 y(receiv)o(ed)f(message,)f(and)g(that)f(the)h (status)f(ob)s(ject)h(is)g(set;)h(the)f(receiv)o(er)h(is)f(no)o(w)f(free)h (to)g(access)g(the)75 1925 y(receiv)o(e)c(bu\013er,)f(or)f(an)o(y)g(other)h (lo)q(cation)h(that)e(can)h(b)q(e)g(referenced)h(b)o(y)f(the)g(receiv)o(e)g (op)q(eration.)20 b(It)13 b(do)q(es)75 1982 y(not)g(indicate)i(that)d(the)i (matc)o(hing)f(send)h(op)q(eration)g(has)f(completed)h(\(but)g(indicates,)g (of)f(course,)h(that)75 2038 y(the)h(send)h(w)o(as)e(initiated\).)75 2142 y Ft(MPI)p 160 2142 V 16 w(W)l(AIT)h(\(handle,)h(status\))117 2219 y Fs(IN)155 b Ft(handle)465 b Fs(handle)14 b(to)g(comm)o(unicati)o(on)d (ob)r(ject)117 2294 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 2419 y FA(A)g(call)h(to)f Ft(MPI)p 439 2419 V 16 w(W)l(AIT)g FA(returns)g(when)g(the)g(send)h(op)q(eration)f(iden)o (ti\014ed)j(b)o(y)d Ft(handle)h FA(is)g(complete.)75 2475 y(If)e(the)f(comm)o (unication)i(ob)s(ject)d(asso)q(ciated)i(with)g(this)g(handle)h(w)o(as)d (created)i(b)o(y)f(a)h(non)o(blo)q(c)o(king)g(send)75 2532 y(or)19 b(receiv)o(e)i(call,)h(then)e(the)g(ob)s(ject)f(is)i(deallo)q(cated)g (b)o(y)f(the)g(call)h(to)39 b Ft(MPI)p 1422 2532 V 16 w(W)l(AIT)20 b FA(and)g(the)g(handle)75 2588 y(b)q(ecomes)c(n)o(ull.)166 2645 y(The)h(call)g(returns)g(in)g Ft(status)h FA(information)f(on)f(the)h (completed)g(op)q(eration.)24 b(The)17 b(status)f(ob)s(ject)75 2701 y(for)g(a)g(receiv)o(e)i(op)q(eration)f(can)g(b)q(e)g(queried)h(using)f (the)g(functions)g(describ)q(ed)i(in)f(Section)f(1.3.1.)23 b(The)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 17 19 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(17)75 49 y(status)15 b(parameter)g(is)i(not)e(used)i(or)e(up)q (dated)i(for)e(a)h(send)g(op)q(eration,)g(and)g(an)g(arbitrary)g(v)m(alue)h (ma)o(y)75 106 y(b)q(e)f(passed)f(for)g(this)h(parameter)e(in)i(the)f(call.) 75 209 y Ft(MPI)p 160 209 14 2 v 16 w(TEST)g(\(handle,)h(\015ag,)f(status\)) 117 287 y Fs(IN)155 b Ft(handle)465 b Fs(handle)14 b(to)g(comm)o(unicati)o (on)d(ob)r(ject)117 362 y(OUT)124 b Ft(\015ag)502 b Fs(logical)117 437 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 561 y FA(A)d(call)h(to)f Ft(MPI)p 430 561 V 15 w(TEST)h FA(returns)f Ft(\015ag)g(=)g(true)h FA(if)f(the)g(op)q(eration)h(iden)o(ti\014ed)h(b)o(y)e Ft(handle)h FA(is)g(complete.)75 618 y(In)j(suc)o(h)g(a)f(case,)h(the)f (status)g(ob)s(ject)g(is)h(set)f(to)g(con)o(tain)h(information)f(on)h(the)g (completed)g(op)q(eration;)75 674 y(if)e(the)g(comm)o(unication)g(ob)s(ject)g (w)o(as)e(created)i(b)o(y)g(a)f(non)o(blo)q(c)o(king)i(send)g(or)e(receiv)o (e,)h(then)g(it)g(is)h(deallo-)75 731 y(cated)h(and)g(the)h(handle)g(b)q (ecomes)g(n)o(ull.)24 b(The)16 b(call)h(returns)f Ft(\015ag=false)p FA(,)h(otherwise.)22 b(In)17 b(suc)o(h)g(a)e(case,)75 787 y(the)g(v)m(alue)i (of)d(the)i(status)e(ob)s(ject)h(is)g(unde\014ned.)166 844 y Ft(MPI)p 251 844 V 16 w(TEST)g FA(is)h(a)f(lo)q(cal,)h(non)o(blo)q(c)o (king)g(op)q(eration.)166 900 y(The)11 b(return)g(status)f(ob)s(ject)h(for)f (a)h(receiv)o(e)h(op)q(eration)f(carries)g(information)g(that)g(can)g(b)q(e)g (accessed)75 957 y(using)k(the)f(functions)h(describ)q(ed)h(in)g(Section)f (1.3.1.)j(The)c(status)g(argumen)o(t)f(is)i(not)f(used)g(or)g(up)q(dated)75 1013 y(for)h(a)f(send)i(op)q(eration)g(and)f(an)g(arbitrary)g(v)m(alue)h(can) f(b)q(e)h(passed)g(for)e(this)i(parameter)e(in)i(the)g(call.)166 1069 y(In)i(m)o(ulti-threaded)h(en)o(vironmen)o(t,)f(the)g(use)g(of)f(a)h (blo)q(c)o(king)h(receiv)o(e)f(op)q(eration)g(\()g Ft(MPI)p 1732 1069 V 15 w(W)l(AIT)p FA(\))75 1126 y(ma)o(y)g(allo)o(w)i(the)f(op)q (erating)g(system)g(to)g(de-sc)o(hedule)i(the)e(blo)q(c)o(k)o(ed)h(thread)f (and)h(sc)o(hedule)g(another)75 1182 y(thread)j(for)g(execution,)j(if)e(suc)o (h)g(is)f(a)o(v)m(ailable.)47 b(The)23 b(use)h(of)f(a)g(non)o(blo)q(c)o(king) h(receiv)o(e)h(op)q(eration)75 1239 y(\()p Ft(MPI)p 178 1239 V 15 w(TEST)p FA(\))e(allo)o(ws)f(the)g(user)g(to)g(sc)o(hedule)i(alternativ) o(e)e(activities)i(within)f(a)f(single)h(thread)f(of)75 1295 y(execution.)166 1352 y(Example:)75 1446 y Fp(CALL)h(MPI_COMM_RANK\(comm,)f (rank,)h(ierr\))75 1502 y(IF\(rank.EQ.0\))123 1558 y(THEN)170 1615 y(CALL)h(MPI_ISEND\(handle,)d(a\(1\),)i(10,)h(MPI_REAL,)f(1,)g(tag,)g (comm,)h(ierr\))170 1671 y(****)g(do)f(some)h(computation)e(to)i(mask)f (latency)g(****)170 1728 y(CALL)h(MPI_WAIT\(handle,)d(status,)i(ierr\))123 1784 y(ELSE)170 1841 y(CALL)h(MPI_IRECV\(handle,)d(a\(1\),)i(15,)h(MPI_REAL,) f(0,)g(tag,)g(comm,)h(ierr\))170 1897 y(****)g(do)f(some)h(computation)e(to)i (mask)f(latency)g(****)170 1954 y(CALL)h(MPI_WAIT\(handle,)d(status,)i (ierr\))75 2010 y(END)g(IF)166 2104 y FA(The)f(functions)h Ft(MPI)p 554 2104 V 15 w(W)l(AIT)f FA(and)g Ft(MPI)p 881 2104 V 16 w(TEST)g FA(can)g(b)q(e)g(used)h(to)e(complete)h(b)q(oth)g(sends)h(and) 75 2160 y(receiv)o(es;)j(they)c(will)i(b)q(e)e(also)g(used)h(to)f(complete)g (an)o(y)g(other)g(non)o(blo)q(c)o(king)h(comm)o(unication)g(call)75 2217 y(pro)o(vided)16 b(b)o(y)f(MPI.)75 2339 y Fr(1.8.4)49 b(Semantics)13 b(of)k(Nonblo)q(cking)h(Comm)o(unications)1875 2407 y Fo(C)n(H)-6 b(A)h(N)f(G)h(E)75 2424 y FA(The)11 b(seman)o(tics)f(of)h (non)o(blo)q(c)o(king)g(comm)o(unication)h(is)f(de\014ned)h(b)o(y)e(suitably) i(extending)g(the)e(de\014nitions)75 2481 y(in)16 b(Section)g(1.4)e(for)h (order)g(and)g(progress.)166 2537 y(Non)o(blo)q(c)o(king)k(comm)o(unication)f (op)q(erations)g(are)g(ordered)g(according)g(to)f(the)h(execution)h(order)75 2594 y(of)h(the)g(calls)h(that)e(initiate)j(the)e(comm)o(unication;)i(this)f (de\014nes)g(the)f(meaning)h(of)e(non-o)o(v)o(ertaking)75 2650 y(comm)o(unication.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 20 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(18)166 49 y(A)22 b(call)g(to)f Ft(MPI)p 458 49 14 2 v 16 w(W)l(AIT)h FA(that)f(completes)h(a)f(receiv)o(e)i(will)g(ev)o (en)o(tually)g(terminate)e(and)h(return)75 106 y(if)f(a)f(matc)o(hing)g(send) h(has)f(b)q(een)h(started,)g(unless)g(the)g(send)g(is)f(satis\014ed)h(b)o(y)f (another)g(receiv)o(e.)36 b(In)75 162 y(particular,)20 b(if)f(the)g(matc)o (hing)g(send)g(is)g(non)o(blo)q(c)o(king,)i(then)e(the)g(receiv)o(e)g(should) h(complete)f(ev)o(en)g(if)75 219 y(no)c(call)i(is)e(executed)i(b)o(y)e(the)g (sender)h(to)f(complete)h(the)f(send.)20 b(Similarly)l(,)e(a)d(call)h(to)f Ft(MPI)p 1651 219 V 15 w(W)l(AIT)h FA(that)75 275 y(completes)i(a)e(send)i (will)h(ev)o(en)o(tually)f(return)f(if)g(a)g(matc)o(hing)g(receiv)o(e)h(has)f (b)q(een)h(started,)f(unless)h(the)75 332 y(receiv)o(e)e(is)g(satis\014ed)f (b)o(y)g(another)g(send,)g(and)g(ev)o(en)h(if)f(no)g(call)h(is)g(executed)g (to)e(complete)i(the)f(receiv)o(e.)166 388 y(If)i Ft(MPI)p 298 388 V 15 w(TEST)g FA(that)f(completes)h(a)f(receiv)o(e)i(is)f(rep)q (eatedly)g(called)i(with)d(the)h(same)f(parameters,)75 444 y(and)e(a)g(matc)o(hing)g(send)g(has)g(b)q(een)h(started,)e(then)h(the)g (call)h(will)g(ev)o(en)o(tually)g(return)f Ft(\015ag=true)p FA(,)h(unless)75 501 y(the)i(send)g(is)g(satis\014ed)h(b)o(y)f(another)f (receiv)o(e.)25 b(If)17 b Ft(MPI)p 1020 501 V 16 w(TEST)g FA(that)f (completes)i(a)e(send)i(is)f(rep)q(eatedly)75 557 y(called)j(with)f(the)g (same)f(parameters,)g(and)h(a)f(matc)o(hing)h(receiv)o(e)g(has)f(b)q(een)i (started,)e(then)h(the)g(call)75 614 y(will)e(ev)o(en)o(tually)f(return)f Ft(\015ag=true)p FA(,)h(unless)g(the)g(receiv)o(e)g(is)f(satis\014ed)h(b)o(y) f(another)g(send.)166 670 y(Non)o(blo)q(c)o(king)g(comm)o(unications)g (cannot)f(b)q(e)g(replaced)i(b)o(y)e(blo)q(c)o(king)h(comm)o(unications,)g (ev)o(en)f(in)75 727 y(the)h Fp(synchronous)f FA(comm)o(unication)i(mo)q(de.) k(Consider)c(the)f(follo)o(wing)h(example:)75 821 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 877 y(IF\(rank.EQ.0\))123 933 y(THEN)170 990 y(CALL)h(MPI_ISEND\(handle1,)d(a\(1\),)i(count,)g (MPI_REAL,)g(1,)h(tag1,)f(comm,)g(ierr\))170 1046 y(CALL)h (MPI_ISEND\(handle2,)d(a\(2\),)i(count,)g(MPI_REAL,)g(1,)h(tag2,)f(comm,)g (ierr\))170 1103 y(CALL)h(MPI_WAIT\(handle1,)d(status,)i(ierr\))170 1159 y(CALL)h(MPI_WAIT\(handle2,)d(status,)i(ierr\))123 1216 y(ELSE)170 1272 y(CALL)h(MPI_IRECV\(handle2,)d(a\(2\),)i(count,)g(MPI_REAL,)g (0,)h(tag2,)f(comm,)g(ierr\))170 1329 y(CALL)h(MPI_IRECV\(handle1,)d(a\(1\),) i(count,)g(MPI_REAL,)g(0,)h(tag1,)f(comm,)g(ierr\))170 1385 y(CALL)h(MPI_WAIT\(handle2,)d(status,)i(ierr\))170 1442 y(CALL)h (MPI_WAIT\(handle1,)d(status,)i(ierr\))75 1498 y(END)g(IF)166 1592 y FA(The)16 b(co)q(de)g(is)g(guaran)o(teed)f(to)g(execute)h(correctly)l (,)g(ev)o(en)g(in)g(an)g(implemen)o(tation)h(that)d(do)q(es)i(not)75 1648 y(bu\013er)d(messages)f(\(the)h(only)h(resource)f(requiremen)o(t)g(is)h (the)f(abilit)o(y)h(to)e(ha)o(v)o(e)h(t)o(w)o(o)f(p)q(ending)i(comm)o(uni-)75 1705 y(cations\).)20 b(If)15 b(one)h(replaces)g(the)f(non)o(blo)q(c)o(king)i (sends)f(\(receiv)o(es\))f(b)o(y)g(blo)q(c)o(king)i(sends)f(\(receiv)o(es\),) f(and)75 1761 y(deletes)i(the)f(calls)g(to)f Ft(MPI)p 546 1761 V 16 w(W)l(AIT)p FA(,)h(then)g(the)f(co)q(de)i(will)g(deadlo)q(c)o(k.)22 b(The)16 b(co)q(de)h(will)g(also)f(deadlo)q(c)o(k)g(if)75 1818 y(one)i(deletes)i(the)e(calls)i(that)d(initiate)j(comm)o(unication)f(and)f (replaces)h(eac)o(h)g(call)g(to)f Ft(MPI)p 1683 1818 V 16 w(W)l(AIT)g FA(b)o(y)75 1874 y(the)d(appropriate)g(blo)q(c)o(king)i(send)f(or)f(receiv)o (e.)75 1996 y Fr(1.8.5)49 b(Multiple)17 b(Completions)75 2082 y FA(It)23 b(is)h(con)o(v)o(enien)o(t)g(to)f(b)q(e)h(able)g(to)e(w)o(ait)h (for)g(the)g(completion)i(of)d(an)o(y)h(or)g(all)h(the)g(op)q(erations)f(in) 75 2138 y(a)g(set,)h(rather)e(than)h(ha)o(ving)g(to)f(w)o(ait)h(for)f(a)g(sp) q(eci\014c)j(message.)43 b(A)23 b(call)h(to)e Ft(MPI)p 1597 2138 V 16 w(W)l(AIT)l(ANY)h FA(or)75 2195 y Ft(MPI)p 160 2195 V 16 w(TEST)l(ANY)e FA(can)f(b)q(e)g(used)g(to)f(w)o(ait)h(for)f(the)h (completion)g(of)g(one)g(out)f(of)g(sev)o(eral)h(op)q(erations;)75 2251 y(a)14 b(call)h(to)27 b Ft(MPI)p 347 2251 V 15 w(W)l(AIT)l(ALL)14 b FA(or)f Ft(MPI)p 701 2251 V 16 w(TEST)l(ALL)h FA(can)g(b)q(e)g(used)h(to)e (w)o(ait)g(for)h(all)g(p)q(ending)i(op)q(erations)e(in)75 2307 y(a)h(list.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 19 21 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(19)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(W)l(AIT)l(ANY)16 b(\(count,)g(a)o(rra)o(y)p 640 49 V 14 w(of)p 691 49 V 16 w(handles,)h (index,)f(status\))117 126 y Fs(IN)155 b Ft(count)482 b Fs(list)14 b(length)f(\(in)o(teger\))117 202 y(IN)155 b Ft(a)o(rra)o(y)p 416 202 V 15 w(of)p 468 202 V 16 w(handles)285 b Fs(arra)o(y)14 b(of)f(handles)h(to)g(comm)o(unication)c(ob)r(jects)117 277 y(OUT)108 b Ft(index)489 b Fs(index)12 b(of)f(handle)g(for)g(op)q(eration)h (that)f(completed)g(\(in)o(teger\))117 352 y(OUT)108 b Ft(status)476 b Fs(return)15 b(status)g(ob)r(ject)166 476 y FA(Blo)q(c)o(ks)j(un)o(til)g (one)g(of)f(the)g(op)q(erations)h(asso)q(ciated)g(with)f(the)h(comm)o (unication)g(handles)h(in)f(the)75 533 y(arra)o(y)13 b(has)i(completed.)21 b(Returns)15 b(the)g(index)h(of)e(that)g(handle)i(in)f(the)g(arra)o(y)l(,)f (and)g(returns)h(the)g(status)75 589 y(of)g(that)f(op)q(eration)i(in)g(the)f (ob)s(ject)g(asso)q(ciated)g(with)h(the)f(status.)166 646 y(The)20 b(execution)h(of)f Ft(MPI)p 614 646 V 16 w(W)l(AIT)l(ANY\(count,)j(a)o(rra)o (y)p 1085 646 V 14 w(of)p 1136 646 V 16 w(handles,)g(index,)f(status\))f FA(has)f(the)g(same)75 702 y(e\013ect)11 b(as)g(the)h(execution)g(of)23 b Ft(MPI)p 662 702 V 15 w(W)l(AIT\(a)o(rra)o(y)p 903 702 V 15 w(of)p 955 702 V 16 w(handles[i],)13 b(status\))p FA(,)h(where)d Ft(i)g FA(is)h(the)g(v)m(alue)h(returned)75 759 y(b)o(y)21 b Ft(index)p FA(.)38 b(In)21 b(particular,)i(the)e(asso)q(ciated)g(comm)o (unication)g(ob)s(ject)g(is)g(is)g(deallo)q(cated,)j(and)d(the)75 815 y(handle)15 b(to)f(it)g(in)h Ft(a)o(rra)o(y)p 469 815 V 15 w(of)p 521 815 V 16 w(handles)h FA(is)e(set)g(to)g(n)o(ull.)21 b(If)14 b(more)g(then)h(one)f(op)q(eration)g(is)h(enabled)g(and)g(can)75 872 y(terminate,)k(one)g(is)g(arbitrarily)g(c)o(hosen.)30 b(There)19 b(is)g(no)g(requiremen)o(t)g(that)f(the)g(c)o(hoice)i(satisfy)e(an)o(y)75 928 y(fairness)e(criterion.)75 1032 y Ft(MPI)p 160 1032 V 16 w(TEST)l(ANY\()g(count,)g(a)o(rra)o(y)p 641 1032 V 15 w(of)p 693 1032 V 16 w(handles,)g(index,)g(status\))117 1109 y Fs(IN)155 b Ft(count)482 b Fs(list)14 b(length)f(\(in)o(teger\))117 1184 y(IN)155 b Ft(a)o(rra)o(y)p 416 1184 V 15 w(of)p 468 1184 V 16 w(handles)285 b Fs(arra)o(y)14 b(of)f(handles)h(to)g(comm)o(unication)c (ob)r(jects)117 1259 y(OUT)108 b Ft(index)489 b Fs(index)9 b(of)g(handle)g(for)g(op)q(eration)g(that)h(completed,)f(or)g Fm(MPI)p 1839 1259 13 2 v 14 w(UNDEFINED)905 1316 y Fs(if)k(none)h(completed) g(\(in)o(teger\))117 1391 y(OUT)108 b Ft(status)476 b Fs(return)15 b(status)g(ob)r(ject)166 1515 y FA(Causes)d(either)h(one)f(or)g(none)h(of)f (the)g(op)q(erations)g(asso)q(ciated)h(with)f(the)h(comm)o(unication)g (handles)75 1572 y(to)e(return.)19 b(In)13 b(the)f(former)f(case,)h(it)g (returns)g(in)h Fp(index)e FA(the)h(index)h(of)f(its)g(handle)h(in)g(the)f (arra)o(y)l(,)f(returns)75 1628 y(in)19 b Fp(status)e FA(the)h(status)f(of)h (that)f(op)q(eration,)i(deallo)q(cates)g(the)f(asso)q(ciated)g(comm)o (unication)h(ob)s(ject,)75 1685 y(and)13 b(sets)g(the)h(handle)g(to)f(n)o (ull.)21 b(In)14 b(the)f(latter)g(case,)g(it)h(returns)f(a)g(v)m(alue)h(of)f Fm(MPI)p 1467 1685 V 15 w(UNDEFINED)f FA(in)i Ft(index)75 1741 y FA(and)h Ft(status)j FA(is)d(unde\014ned.)1875 1770 y Fo(C)n(H)-6 b(A)h(N)f(G)h(E)166 1797 y FA(The)20 b(execution)h(of)f Ft(MPI)p 614 1797 14 2 v 15 w(TEST)l(ANY\(count,)j(a)o(rra)o(y)p 1085 1797 V 15 w(of)p 1137 1797 V 16 w(handles,)f(index,)g(status\))f FA(has)f(the)g(same)75 1854 y(e\013ect)d(as)f(the)h(execution)h(of)f Ft(MPI)p 678 1854 V 16 w(TEST\(a)o(rra)o(y)p 921 1854 V 14 w(of)p 972 1854 V 16 w(handles[i],)i(\015ag,)e(status\))p FA(,)h(for)f(all)h Fp(i=0,)23 b(1)h(,)f(...)75 1910 y(,)h(count-1)p FA(,)18 b(in)h(some)g (arbitrary)f(order,)h(un)o(til)h(one)e(call)i(returns)f Fp(flag=true)p FA(,)f(or)g(all)h(fail.)32 b(In)19 b(the)75 1967 y(former)12 b(case,)h Fp(index)f FA(is)h(set)g(to)f(the)h(last)f(v)m(alue)i(of)f Fp(i)p FA(;)g(in)g(the)g(later)g(case,)g(it)g(is)g(set)g(to)f Fm(MPI)p 1618 1967 13 2 v 14 w(UNDEFINED)p FA(.)75 2071 y Ft(MPI)p 160 2071 14 2 v 16 w(W)l(AIT)l(ALL\()i(count,)i(a)o(rra)o(y)p 626 2071 V 15 w(of)p 678 2071 V 16 w(handles,)h(a)o(rra)o(y)p 958 2071 V 14 w(of)p 1009 2071 V 16 w(status\))117 2148 y Fs(IN)155 b Ft(count)482 b Fs(lists)14 b(length)g(\(in)o(teger\))117 2223 y(IN)155 b Ft(a)o(rra)o(y)p 416 2223 V 15 w(of)p 468 2223 V 16 w(handles)285 b Fs(arra)o(y)14 b(of)f(handles)h(to)g(comm)o(unication)c (ob)r(jects)117 2298 y(OUT)108 b Ft(a)o(rra)o(y)p 416 2298 V 15 w(of)p 468 2298 V 16 w(status)313 b Fs(arra)o(y)14 b(of)f(status)i(ob)r (jects)166 2422 y FA(Blo)q(c)o(ks)e(un)o(til)h(all)g(comm)o(unication)g(op)q (erations)f(asso)q(ciated)g(with)g(handles)h(in)g(the)f(list)h(complete,)75 2479 y(and)j(return)g(the)g(status)f(of)h(all)h(these)f(op)q(erations.)26 b(Both)16 b(arra)o(ys)g(ha)o(v)o(e)h(the)g(same)f(n)o(um)o(b)q(er)i(of)e(v)m (alid)75 2535 y(en)o(tries.)k(The)14 b Ft(i)p FA(-th)f(en)o(try)h(in)g Ft(a)o(rra)o(y)p 679 2535 V 15 w(of)p 731 2535 V 16 w(status)h FA(is)g(set)e(to)g(the)h(return)g(status)e(of)i(the)f Ft(i)p FA(-th)h(op)q(eration.)20 b(All)75 2592 y(comm)o(unication)c(ob)s(jects)e (are)h(deallo)q(cated)i(and)e(all)i(handles)f(are)f(set)g(to)f(n)o(ull.)1875 2621 y Fo(C)n(H)-6 b(A)h(N)f(G)h(E)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 22 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(20)166 49 y(The)20 b(execution)i(of)d Ft(MPI)p 614 49 14 2 v 16 w(W)l(AIT)l(ALL\(count,)j(a)o(rra)o(y)p 1072 49 V 14 w(of)p 1123 49 V 16 w(handles,)h(a)o(rra)o(y)p 1409 49 V 15 w(of)p 1461 49 V 16 w(status)f FA(has)e(the)g(same)75 106 y(e\013ect)d(as)h(the)g(execution)g(of)g Ft(MPI)p 682 106 V 16 w(W)l(AIT\(a)o(rra)o(y)p 924 106 V 14 w(of)p 975 106 V 16 w(handle[i],)h(a)o(rra)o(y)p 1277 106 V 15 w(of)p 1329 106 V 16 w(status[i]\))p FA(,)g(for)f Fp(i=0,)23 b(...)47 b(,)75 162 y(count-1)p FA(,)14 b(in)i(some)f(arbitrary)f(order.)75 266 y Ft(MPI)p 160 266 V 16 w(TEST)l(ALL\(count,)i(a)o(rra)o(y)p 613 266 V 14 w(of)p 664 266 V 16 w(handles,)h(\015ag,)d(a)o(rra)o(y)p 1040 266 V 15 w(of)p 1092 266 V 16 w(status\))117 343 y Fs(IN)155 b Ft(count)482 b Fs(lists)14 b(length)g(\(in)o(teger\))117 415 y(IN)155 b Ft(a)o(rra)o(y)p 416 415 V 15 w(of)p 468 415 V 16 w(handles)285 b Fs(arra)o(y)14 b(of)f(handles)h(to)g(comm)o(unication)c (ob)r(jects)117 487 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))117 559 y(OUT)108 b Ft(a)o(rra)o(y)p 416 559 V 15 w(of)p 468 559 V 16 w(status)313 b Fs(arra)o(y)14 b(of)f(status)i(ob)r(jects)166 684 y FA(Causes)f(either)g(all)h(or)f(none)g(of)f(the)i(op)q(erations)f(asso) q(ciated)g(with)g(the)g(comm)o(unication)h(handles)75 740 y(to)c(complete.)19 b(It)11 b(returns)g Ft(\015ag)h(=)f(true)h FA(if)g(all)g(comm)o(unications)g (asso)q(ciated)g(with)f(handles)i(in)f(the)f(arra)o(y)75 797 y(ha)o(v)o(e)18 b(completed.)31 b(In)19 b(this)h(case,)f(eac)o(h)f(status)g (en)o(try)g(is)h(set)g(to)f(the)g(status)g(of)g(the)h(corresp)q(onding)75 853 y(comm)o(unication.)h(All)15 b(comm)o(unication)g(ob)s(jects)e(are)h (deallo)q(cated,)h(and)f(all)h(handles)g(are)f(set)g(to)f(n)o(ull.)75 910 y(Otherwise,)g Ft(\015ag)g(=)g(false)f FA(is)h(returned,)g(no)g(handle)h (is)f(mo)q(di\014ed)g(and)g(the)g(v)m(alues)g(of)f(the)h(status)f(en)o(tries) 75 966 y(are)j(unde\014ned.)22 b(This)15 b(is)h(a)f(non)o(blo)q(c)o(king)i (op)q(eration)e(that)f(returns)h(immediately)l(.)166 1022 y(Example:)75 1157 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1214 y(IF\(rank)g(<)h(2\))123 1270 y(THEN)214 b(!)24 b(rank=0)f(or)g(rank=1) 170 1327 y(CALL)h(MPI_ISEND\(handle,)d(a,)j(n,)f(MPI_REAL,)g(2,)h(tag,)f (comm,)g(ierr\))170 1383 y(****)h(do)f(some)h(computation)e(to)i(mask)f (latency)g(****)170 1440 y(CALL)h(MPI_WAIT\(handle,)d(status,)i(ierr\))123 1496 y(ELSE)214 b(!)24 b(rank=2)170 1553 y(CALL)g (MPI_IRECV\(handle_list\(0\))o(,)d(a,)j(n,)f(MPI_REAL,)g(0,)h(tag,)f(comm,)g (ierr\))170 1609 y(CALL)h(MPI_IRECV\(handle_list\(1\))o(,)d(b,)j(n,)f (MPI_REAL,)g(1,)h(tag,)f(comm,)g(ierr\))170 1665 y(****)h(do)f(some)h (computation)e(to)i(mask)f(latency)g(****)170 1722 y(CALL)h(MPI_WAITANY\(2,)e (handle_list,)g(index,)h(status,)g(ierr\))170 1778 y(IF\(index.EQ.0\))218 1835 y(THEN)266 1891 y(****)g(handle)g(message)g(from)g(process)g(0)h(****) 266 1948 y(CALL)f(MPI_WAIT\(handle_list\(1\),)e(status,)i(ierr\))266 2004 y(****)g(handle)g(message)g(from)g(process)g(1)h(****)218 2061 y(ELSE)266 2117 y(****)f(handle)g(message)g(from)g(process)g(1)h(****) 266 2174 y(CALL)f(MPI_WAIT\(handle_list\(0\),)e(status,)i(ierr\))266 2230 y(****)g(handle)g(message)g(from)g(process)g(0)h(****)170 2286 y(END)g(IF)75 2343 y(END)f(IF)166 2421 y FA(The)16 b(calls)h(in)o(tro)q (duced)g(in)g(this)g(subsection)f(can)h(b)q(e)f(used)h(to)e(w)o(ait)h(or)f (test)g(for)h(the)g(completion)75 2478 y(of)d(an)h(arbitrary)f(mix)h(of)f (non)o(blo)q(c)o(king)i(comm)o(unication)g(calls,)f(including)j(a)c(mix)h (sends)g(and)g(receiv)o(es)75 2534 y(and)h(of)g(an)o(y)g(additional)i(non)o (blo)q(c)o(king)f(comm)o(unication)g(calls)g(pro)o(vided)g(b)o(y)f(MPI.)166 2591 y(The)d Ft(status)h FA(argumen)o(t)e(in)i Ft(MPI)p 715 2591 V 16 w(W)l(AIT)l(ANY)f FA(and)g Ft(MPI)p 1111 2591 V 16 w(W)l(AIT)l(ALL)f FA(and)h(the)g Ft(a)o(rra)o(y)p 1582 2591 V 15 w(of)p 1634 2591 V 16 w(status)h FA(argu-)75 2647 y(men)o(t)d(in)h Ft(MPI)p 318 2647 V 16 w(W)l(AIT)l(ALL)f FA(and)g Ft(MPI)p 698 2647 V 16 w(TEST)l(ALL)g FA(are)g(not)g(used)g(or)g(up)q(dated)h(if)g (the)f(completing)h(comm)o(u-)75 2704 y(nication)j(is)g(a)f(send.)20 b(If)13 b(all)i(op)q(erations)e(in)h(the)g(list)g(pro)o(vided)g(b)o(y)f(the)g (argumen)o(t)g Ft(a)o(rra)o(y)p 1596 2704 V 14 w(of)p 1647 2704 V 16 w(handles)i FA(are)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 21 23 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(21)75 49 y(send)15 b(op)q(erations,)g(then)g(an)f(arbitrary)g (v)m(alue)i(can)f(b)q(e)g(passed)g(to)f(the)h Ft(status)i FA(\(resp.)34 b Ft(a)o(rra)o(y)p 1680 49 14 2 v 14 w(of)p 1731 49 V 16 w(status)p FA(\))75 106 y(argumen)o(t.)1875 135 y Fo(C)n(H)-6 b(A)h(N)f(G)h(E)166 162 y FA(An)23 b Ft(a)o(rra)o(y)p 346 162 V 14 w(of)p 397 162 V 16 w(handles)h FA(argumen)o(t)e(that)f(is)i(passed)g(to)e(an)o(y)h(of)g (the)g(last)h(four)f(functions)h(ma)o(y)75 219 y(con)o(tain)18 b(n)o(ull)h(en)o(tries.)29 b(These)18 b(en)o(tries)h(are)e(ignored)i(and)f (the)g(call)h(executes)f(as)g(if)g(it)g(w)o(as)g(supplied)75 275 y(an)e(arra)o(y)e(of)h(handles)i(con)o(taining)g(only)f(the)g(non-n)o (ull)h(en)o(tries.)22 b(As)16 b(a)f(handle)i(in)f Ft(a)o(rra)o(y)p 1623 275 V 15 w(of)p 1675 275 V 16 w(handles)h FA(is)75 332 y(set)c(to)g(n)o(ull)i(when)f(the)g(corresp)q(onding)g(op)q(eration)g (completes,)g(this)g(allo)o(ws)g(to)f(reuse)g(the)h(same)f(arra)o(y)75 388 y(for)j(completing)h(in)g(successiv)o(e)g(calls)h(the)e(op)q(erations)g (asso)q(ciated)h(with)f(this)h(arra)o(y)l(.)22 b(Th)o(us,)16 b(the)g(last)75 444 y(example)g(can)f(b)q(e)h(rewritten)f(as)g(follo)o(ws:)75 590 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 647 y(IF\(rank)g(<)h(2\))123 703 y(THEN)214 b(!)24 b(rank=0)f(or)g(rank=1)170 760 y(CALL)h(MPI_ISEND\(handle,)d(a,)j(n,)f(MPI_REAL,)g(2,)h(tag,)f(comm,)g (ierr\))170 816 y(CALL)h(MPI_WAIT\(handle,)d(status,)i(ierr\))123 872 y(ELSE)214 b(!)24 b(rank=2)170 929 y(CALL)g(MPI_IRECV\(handle_list\(0\))o (,)d(a,)j(n,)f(MPI_REAL,)g(0,)h(tag,)f(comm,)g(ierr\))170 985 y(CALL)h(MPI_IRECV\(handle_list\(1\))o(,)d(b,)j(n,)f(MPI_REAL,)g(1,)h(tag,)f (comm,)g(ierr\))170 1042 y(CALL)h(MPI_WAITANY\(2,)e(handle_list,)g(index,)h (status,)g(ierr\))266 1098 y(****)g(handle)g(first)g(message)47 b(****)170 1155 y(CALL)24 b(MPI_WAITANY\(2,)e(handle_list,)g(index,)h (status,)g(ierr\))266 1211 y(****)g(handle)g(second)g(message)47 b(****)166 1300 y FA(If)11 b Ft(MPI)p 292 1300 V 16 w(W)l(AIT)l(ANY)h FA(or)f Ft(MPI)p 655 1300 V 16 w(TEST)l(ANY)h FA(are)f(called)i(with)e(an)g (empt)o(y)g Ft(a)o(rra)o(y)p 1470 1300 V 15 w(of)p 1522 1300 V 16 w(handles)i FA(argumen)o(t)75 1357 y(\(i.e.,)k(an)g(arra)o(y)f(of)g (length)i(zero,)f(or)f(an)h(arra)o(y)f(with)i(all)g(en)o(tries)f(n)o(ull\))h (then)g(they)f(return)g(the)g(v)m(alue)75 1413 y Fm(MPI)p 152 1413 13 2 v 14 w(UNDEFINED)g FA(in)h Ft(index)p FA(;)i Ft(MPI)p 685 1413 14 2 v 16 w(W)l(AIT)l(ALL)d FA(with)h(an)f(empt)o(y)h Ft(a)o(rra)o(y)p 1315 1413 V 14 w(of)p 1366 1413 V 16 w(handles)i FA(argumen)o(t)d(has)g(no)75 1470 y(e\013ect;)e Ft(MPI)p 293 1470 V 15 w(TEST)l(ALL)h FA(with)f(an)g(empt)o(y)g Ft(a)o(rra)o(y)p 914 1470 V 15 w(of)p 966 1470 V 16 w(handles)i FA(argumen)o(t)d(returns)h Ft(\015ag=true)p FA(.)75 1611 y Fx(1.9)59 b(Prob)r(e)20 b(and)f(Cancel)75 1712 y FA(The)d Ft(MPI)p 254 1712 V 16 w(PROBE)g FA(and)g Ft(MPI)p 605 1712 V 16 w(IPROBE)g FA(op)q(erations)g(allo)o(w)g(incoming)h(messages)e (to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)75 1769 y(without)g(actually)h (receiving)h(them.)k(The)17 b(user)f(can)g(then)h(decide)g(ho)o(w)f(to)g (receiv)o(e)h(them,)f(based)g(on)75 1825 y(the)i(information)f(returned)h(b)o (y)f(the)h(prob)q(e)g(\(basically)l(,)h(the)e(information)h(returned)g(b)o(y) f Ft(status)p FA(\).)28 b(In)75 1882 y(particular,)16 b(the)g(user)g(ma)o(y)f (allo)q(cate)h(memory)f(for)g(the)h(receiv)o(e)h(bu\013er,)e(according)h(to)f (the)h(length)g(of)75 1938 y(the)f(prob)q(ed)h(message.)166 1994 y(The)h Ft(MPI)p 346 1994 V 16 w(CANCEL)g FA(op)q(eration)g(allo)o(ws)g (p)q(ending)i(comm)o(unications)e(to)f(b)q(e)i(cancelled.)27 b(This)17 b(is)75 2051 y(required)j(for)f(clean)o(up.)33 b(P)o(osting)19 b(a)g(send)h(or)f(a)g(receiv)o(e)h(ties)f(up)h(user)f(resources)h(\(send)f (or)g(receiv)o(e)75 2107 y(bu\013ers\),)14 b(and)i(a)f(cancel)h(ma)o(y)f(b)q (e)g(needed)i(to)d(free)i(these)f(resources)g(gracefully)l(.)75 2211 y Ft(MPI)p 160 2211 V 16 w(IPROBE\(source,)h(tag,)f(comm)m(,)c(\015ag,)k (status\))117 2288 y Fs(IN)155 b Ft(source)468 b Fs(source)15 b(rank,)f(or)27 b Fm(MPI)p 1281 2288 13 2 v 15 w(ANY)p 1381 2288 V 14 w(SOURCE)13 b Fs(\(in)o(teger\))117 2361 y(IN)155 b Ft(tag)526 b Fs(tag)14 b(v)n(alue)f(or)28 b Fm(MPI)p 1226 2361 V 14 w(ANY)p 1325 2361 V 15 w(T)m(A)o(G)12 b Fs(\(in)o(teger\))117 2434 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2506 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))117 2579 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 2704 y Ft(MPI)p 251 2704 14 2 v 16 w(IPROBE)k FA(returns)f Ft(\015ag)g(=)h(true)f FA(if)h(there)g(is)f(a)g(message)g(that)g(can)g(b)q(e)h(receiv)o(ed)g(and)g (that)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 22 24 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(22)75 49 y(matc)o(hes)16 b(the)h(pattern)f(sp)q(eci\014ed)j(b) o(y)e(the)g(parameters)f Ft(source)p FA(,)h Ft(tag)p FA(,)g(and)g Ft(comm)m FA(.)22 b(It)16 b(returns)h Ft(\015ag)g(=)75 106 y(false)p FA(,)d(otherwise.)19 b(If)14 b Ft(MPI)p 534 106 14 2 v 16 w(IPROBE)g FA(returns)f Ft(\015ag)g(=)h(true)p FA(,)g(then)g(the)f (source)h(and)f(tag)g(of)g(the)g(message)75 162 y(matc)o(hed)g(are)g (returned)h(in)g(the)f(status)g(ob)s(ject.)18 b(These)c(are)f(the)g(same)g(v) m(alues)i(that)d(w)o(ould)i(ha)o(v)o(e)f(b)q(een)75 219 y(returned)j(b)o(y)f (a)g(call)h(to)e Ft(MPI)p 584 219 V 16 w(RECV)i FA(executed)g(at)f(the)g (same)g(p)q(oin)o(t)h(in)g(the)f(program.)166 275 y(These)22 b(v)m(alues)g(can)g(b)q(e)g(deco)q(ded)h(from)e(the)g(return)h(status)e(ob)s (ject)h(using)i(the)e(status)g(query)75 332 y(functions)16 b(describ)q(ed)h(in)f(Section)g(1.3.1.)i(The)e(return)f(status)f(ob)s(ject)h (is)h(unde\014ned)h(if)e Ft(\015ag)h(=)f(false)p FA(.)166 388 y Ft(MPI)p 251 388 V 16 w(GET)p 355 388 V 16 w(COUNT)f FA(cannot)e(b)q(e)h (used)g(after)f(a)g(call)i(to)e Ft(MPI)p 1203 388 V 15 w(IPROBE)i FA(to)e(query)g(on)h(the)f(length)h(of)75 444 y(the)g(prob)q(ed)h(message.)19 b(\(The)13 b(reason)f(is)i(that)e Ft(MPI)p 967 444 V 16 w(IPROBE)i FA(do)q(es)g(not)e(carry)h(a)g(datat)o(yp)q(e)f(argumen)o(t,)75 501 y(and)f(the)g(message)g(t)o(yp)q(e)g(ma)o(y)f(not)h(b)q(e)h(kno)o(wn)e (when)i(the)f(prob)q(e)h(is)f(executed.)20 b(On)11 b(the)g(other)g(hand,)h (the)75 557 y(datat)o(yp)q(e)h(information)i(is)f(needed)h(to)f(compute)g (the)g(n)o(um)o(b)q(er)g(of)g(elemen)o(ts)h(in)g(a)e(message.\))19 b(Instead,)75 614 y(a)14 b(di\013eren)o(t)g(status)g(deco)q(ding)h(function)h (that)d(pro)o(vides)i(the)f(datat)o(yp)q(e)g(as)g(argumen)o(t)f(can)h(b)q(e)h (used)g(in)75 670 y(this)h(case.)75 774 y Ft(MPI)p 160 774 V 16 w(PROBE)p 324 774 V 17 w(COUNT\(status,)i(datat)o(yp)q(e,)e(count\))117 851 y Fs(IN)155 b Ft(status)476 b Fs(status)15 b(ob)r(ject)117 926 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(message)14 b(datat)o(yp)q(e)117 1001 y(OUT)108 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(en)o(tries)h(in) e(message)h(\(in)o(teger\))166 1126 y FA(Returns)h(the)f(n)o(um)o(b)q(er)h (of)f(elemen)o(ts)h(in)h(the)e(message)g(that)g(w)o(as)f(last)i(prob)q(ed)g (with)g Ft(status)h FA(as)e(an)75 1182 y(argumen)o(t.)23 b(The)17 b(call)g(is)g(erroneous)g(if)g Ft(datat)o(yp)q(e)h FA(do)q(es)f(not)f(matc)o (h)g(the)g(t)o(yp)q(e)h(of)f(the)h(message.)23 b(It)16 b(is)75 1239 y(erroneous)f(to)f(in)o(v)o(ok)o(e)g Ft(MPI)p 559 1239 V 16 w(PROBE)p 723 1239 V 18 w(COUNT)h FA(with)g(a)g Ft(status)h FA(argumen)o(t)e(that)g(w)o(as)g(set)h(b)o(y)f(a)h(regular)75 1295 y(receiv)o(e)h(op)q(eration.)166 1352 y(A)d(subsequen)o(t)i(receiv)o(e)f (executed)g(with)g(the)g(same)f(con)o(text,)g(and)g(the)h(source)f(and)h(tag) f(returned)75 1408 y(b)o(y)j(the)g(call)i(to)d Ft(MPI)p 444 1408 V 16 w(IPROBE)i FA(will)h(receiv)o(e)f(the)g(message)e(that)h(w)o(as)f (matc)o(hed)h(b)o(y)g(the)h(prob)q(e,)f(if)h(no)75 1465 y(other)11 b(in)o(terv)o(ening)h(receiv)o(e)g(o)q(ccurred)g(after)e(the)h(prob)q(e.)19 b(If)11 b(the)g(receiving)i(pro)q(cess)e(is)h(m)o(ulti-threaded,)75 1521 y(it)j(is)h(the)f(user)h(resp)q(onsibilit)o(y)h(to)e(ensure)h(that)e (the)i(last)f(condition)h(holds.)166 1654 y Fn(Discussion:)166 1703 y Fs(MPI)f(guaran)o(tees)g(that)g(successiv)o(e)i(messages)e(sen)o(t)h (from)d(a)h(source)i(to)f(a)f(destination)h(within)e(the)j(same)75 1753 y(con)o(text)21 b(are)g(receiv)o(ed)h(in)e(the)h(order)g(they)g(are)g (sen)o(t.)39 b(Th)o(us,)22 b(MPI)e(m)o(ust)g(supp)q(ort,)i(either)g (explicitly)d(or)75 1803 y(implicitly)l(,)13 b(a)i(FIF)o(O)g(structure)j(to)d (manage)e(messages)j(b)q(et)o(w)o(een)h(eac)o(h)e(pair)g(of)g(pro)q(cesses,)j (for)d(eac)o(h)h(con)o(text.)75 1853 y Fm(MPI)p 152 1853 13 2 v 14 w(PROBE)e Fs(returns)j(information)c(on)i(the)h(\014rst)h(matc)o(hing) c(message)j(in)f(this)g(FIF)o(O;)h(this)f(will)f(also)h(b)q(e)h(the)75 1903 y(message)10 b(receiv)o(ed)h(b)o(y)f(the)h(\014rst)g(subsequen)o(t)h (receiv)o(e)f(with)f(the)h(same)e(source,)j(tag)d(and)h(con)o(text)h(as)g (the)f(message)75 1952 y(matc)o(hed)j(b)o(y)h Fm(MPI)p 377 1952 V 14 w(PROBE)p Fs(.)75 2139 y Ft(MPI)p 160 2139 14 2 v 16 w(PROBE\()i(source,)f(tag,)g(comm)m(,)d(status\))117 2216 y Fs(IN)155 b Ft(source)468 b Fs(source)15 b(rank,)f(or)27 b Fm(MPI)p 1281 2216 13 2 v 15 w(ANY)p 1381 2216 V 14 w(SOURCE)13 b Fs(\(in)o(teger\))117 2291 y(IN)155 b Ft(tag)526 b Fs(tag)14 b(v)n(alue,)e(or)28 b Fm(MPI)p 1237 2291 V 14 w(ANY)p 1336 2291 V 15 w(T)m(A)o(G)13 b Fs(\(in)o(teger\))117 2366 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2441 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 2566 y Ft(MPI)p 251 2566 14 2 v 16 w(PROBE)i FA(b)q(eha)o(v)o(es)g(lik)o(e)g Ft(MPI)p 769 2566 V 16 w(IPROBE)g FA(except)g(that)f(it)g(is)h(a)f(blo)q(c)o (king)i(call)f(whic)o(h)g(returns)75 2622 y(only)f(after)e(a)h(matc)o(hing)g (message)g(has)g(b)q(een)h(found.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 23 25 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(23)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(CANCEL\(handle\))117 126 y Fs(IN)155 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j (\(handle\))166 251 y FA(A)j(call)h(to)e Ft(MPI)p 444 251 V 15 w(CANCEL)i FA(marks)e(for)g(cancellation)i(a)f(p)q(ending)h(non)o(blo)q(c) o(king)g(comm)o(unication)75 307 y(op)q(eration)e(\(send)g(or)g(receiv)o (e\).)23 b(The)16 b(cancel)h(call)g(is)g(non-blo)q(c)o(king,)g(and)f(lo)q (cal.)24 b(It)16 b(returns)f(immedi-)75 364 y(ately)l(,)g(p)q(ossibly)i(b)q (efore)e(the)h(comm)o(unication)g(is)f(actually)h(cancelled.)166 420 y(It)c(is)g(still)h(necessary)f(to)g(complete)g(a)g(comm)o(unication)g (that)f(has)h(b)q(een)h(mark)o(ed)e(for)h(cancellation,)75 477 y(using)i(a)g(call)h(to)e Ft(MPI)p 450 477 V 15 w(W)l(AIT)h FA(or)f Ft(MPI)p 728 477 V 16 w(TEST)h FA(\(or)f(an)o(y)g(of)g(the)h(deriv)o (ed)h(op)q(erations\).)k(If)14 b(the)g(op)q(eration)75 533 y(has)20 b(b)q(een)h(cancelled,)i(then)d(information)g(ab)q(out)g(to)f(that)h (e\013ect)f(will)j(b)q(e)f(returned)f(in)h(the)f(status)75 590 y(argumen)o(t)10 b(of)g(the)h(op)q(eration)g(that)f(completes)h(the)g (comm)o(unication.)19 b(If)11 b(a)g(comm)o(unication)g(is)g(mark)o(ed)75 646 y(for)20 b(cancellation,)i(then)f(a)f Ft(MPI)p 655 646 V 15 w(W)l(AIT)h FA(call)g(for)e(that)h(comm)o(unication)h(is)f(guaran)o (teed)g(to)f(return,)75 703 y(irresp)q(ectiv)o(e)f(of)e(the)h(activities)h (of)e(other)h(pro)q(cesses;)g(similarly)i(if)e Ft(MPI)p 1347 703 V 15 w(TEST)g FA(is)h(rep)q(eatedly)f(called)75 759 y(in)23 b(a)f(busy)g(w)o(ait)f(lo)q(op)i(for)e(a)h(cancelled)i(comm)o(unication,)g (then)f Ft(MPI)p 1363 759 V 15 w(TEST)g FA(will)g(ev)o(en)o(tually)g(b)q(e)75 815 y(successful.)166 872 y(Either)c(the)f(cancellation)i(succeeds,)f(or)f (the)g(comm)o(unication)h(succeeds,)h(but)e(not)g(b)q(oth.)29 b(If)18 b(a)75 928 y(send)h(is)h(mark)o(ed)e(for)g(cancellation,)j(then)e(it) g(m)o(ust)f(b)q(e)i(the)f(case)g(that)f(either)h(the)g(send)g(completes)75 985 y(normally)l(,)f(in)h(whic)o(h)f(case)f(the)h(message)f(sen)o(t)g(w)o(as) g(receiv)o(ed)h(at)f(the)h(destination)g(pro)q(cess,)g(or)f(that)75 1041 y(the)e(send)h(is)f(successfully)i(cancelled,)g(in)e(whic)o(h)h(case)f (no)g(part)f(of)h(the)g(message)f(w)o(as)g(receiv)o(ed)j(at)d(the)75 1098 y(destination.)21 b(Then,)15 b(an)o(y)f(matc)o(hing)h(receiv)o(e)h(has)e (to)h(b)q(e)g(satis\014ed)h(b)o(y)e(another)h(send.)20 b(If)15 b(a)g(receiv)o(e)g(is)75 1154 y(mark)o(ed)d(for)f(cancellation,)k(then)d(it)h (m)o(ust)e(b)q(e)i(the)g(case)f(that)f(either)i(the)g(receiv)o(e)g(completes) g(normally)l(,)75 1211 y(or)k(that)g(the)h(receiv)o(e)h(is)f(successfully)i (cancelled,)g(in)f(whic)o(h)f(case)g(no)g(part)f(of)g(the)h(receiv)o(e)h (bu\013er)e(is)75 1267 y(altered.)j(Then,)c(an)o(y)e(matc)o(hing)i(send)g (has)f(to)f(b)q(e)i(satis\014ed)g(b)o(y)f(another)g(receiv)o(e.)75 1371 y Ft(MPI)p 160 1371 V 16 w(TEST)p 290 1371 V 16 w(CANCELLED\(status,)i (\015ag\))117 1448 y Fs(IN)155 b Ft(status)476 b Fs(status)15 b(ob)r(ject)117 1523 y(IN)155 b Ft(\015ag)518 b Fs(\(logical\))166 1648 y FA(Returns)11 b Ft(\015ag)f(=)h(true)g FA(if)g(the)f(comm)o(unication) h(asso)q(ciated)g(with)f(the)h(status)e(ob)s(ject)h(w)o(as)g(cancelled)75 1704 y(successfully)l(.)27 b(In)18 b(suc)o(h)f(a)g(case,)g(all)g(other)g (\014elds)h(of)e Ft(status)j FA(\(suc)o(h)e(as)g Ft(count)h FA(or)e Ft(tag)p FA(\))h(are)f(unde\014ned.)75 1761 y(Returns)h Ft(\015ag)g(=)f(false)p FA(,)h(otherwise.)24 b(If)17 b(a)g(receiv)o(e)g(op)q (eration)g(migh)o(t)f(b)q(e)h(cancelled)i(then)e(one)g(should)75 1817 y(call)39 b Ft(MPI)p 266 1817 V 16 w(TEST)p 396 1817 V 17 w(CANCELLED)19 b FA(\014rst,)g(to)f(c)o(hec)o(k)i(whether)f(the)g(op)q (eration)g(w)o(as)g(cancelled,)j(b)q(efore)75 1873 y(c)o(hec)o(king)16 b(on)f(the)g(other)g(\014elds)i(of)d(the)i(return)f(status.)75 2017 y Fx(1.10)60 b(P)n(ersistent)21 b(comm)o(unication)c(objects)75 2118 y FA(Often)k(a)g(comm)o(unication)g(with)g(the)g(same)g(parameter)f (list)i(is)f(rep)q(eatedly)h(executed)g(within)g(the)75 2175 y(inner)d(lo)q(op)g(of)e(a)h(parallel)i(computation.)28 b(In)18 b(suc)o(h)h(a)f(situation,)g(it)h(ma)o(y)e(b)q(e)i(p)q(ossible)g(to)f (optimize)75 2231 y(the)c(comm)o(unication)g(b)o(y)g(binding)i(the)d(list)i (of)e(comm)o(unication)i(parameters)d(to)h(the)h(comm)o(unication)75 2287 y(ob)s(ject)g(once)g(and,)h(then,)f(rep)q(eatedly)i(using)f(the)f(comm)o (unication)h(ob)s(ject)f(to)g(initiate)i(and)e(complete)75 2344 y(messages.)24 b(The)16 b(comm)o(unication)i(ob)s(ject)e(th)o(us)g (created)h(can)f(b)q(e)i(though)o(t)e(of)g(as)g(a)g(comm)o(unication)75 2400 y(p)q(ort)11 b(or)f(a)h(\\half-c)o(hannel")h(.)18 b(It)12 b(do)q(es)f(not)f(pro)o(vide)i(the)f(full)h(functionalit)o(y)h(of)d(a)h(con)o (v)o(en)o(tional)g(c)o(hannel,)75 2457 y(since)i(there)g(is)g(no)f(binding)i (of)e(the)g(send)h(p)q(ort)f(to)f(the)i(receiv)o(e)g(p)q(ort:)18 b(this)13 b(construct)f(allo)o(ws)g(reduction)75 2513 y(of)18 b(the)g(o)o(v)o(erhead)f(for)h(comm)o(unication)g(b)q(et)o(w)o(een)h(pro)q (cessor)e(and)h(comm)o(unication)h(con)o(troller,)g(but)75 2570 y(not)c(the)g(o)o(v)o(erhead)g(for)f(comm)o(unication)i(b)q(et)o(w)o (een)g(one)f(comm)o(unication)h(con)o(troller)g(and)f(another.)166 2626 y(A)j(comm)o(unication)h(ob)s(ject)f(is)g(created)h(using)f(one)h(of)e (the)i(four)f(follo)o(wing)g(calls.)30 b(These)19 b(calls)75 2683 y(in)o(v)o(olv)o(e)d(no)f(comm)o(unication.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 24 26 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(24)166 49 y(The)13 b(function)g Ft(MPI)p 517 49 14 2 v 16 w(CREA)l(TE)p 702 49 V 17 w(SEND)g FA(creates)f(a)g(comm)o (unication)h(ob)s(ject)f(for)g(a)g(standard)g(mo)q(de)75 106 y(send)k(op)q(eration,)f(and)g(binds)i(to)d(it)i(all)g(the)f(parameters)f(of) h(a)g(send)h(op)q(eration.)75 209 y Ft(MPI)p 160 209 V 16 w(CREA)l(TE)p 345 209 V 17 w(SEND\(handle,)h(sta)o(rt,)e(count,)h(datat)o(yp)q(e,)g(dest,)h (tag,)d(comm)m(\))117 287 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication) 11 b(ob)r(ject)j(\(handle\))117 362 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 437 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen) o(t)i(\(in)o(teger\))117 512 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)117 587 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 662 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 737 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 862 y FA(The)i(call)i(allo)q(cates)f(a)f(new)h(comm)o(unication)g(ob)s(ject)e (and)i(asso)q(ciates)f(the)h(comm)o(unication)g(han-)75 918 y(dle)i(with)g(it.)166 975 y(The)g(function)h Ft(MPI)p 524 975 V 16 w(CREA)l(TE)p 709 975 V 17 w(RSEND)g FA(creates)e(a)h(comm)o (unication)h(ob)s(ject)e(for)g(a)h(ready)g(mo)q(de)75 1031 y(send)g(op)q(eration.)75 1135 y Ft(MPI)p 160 1135 V 16 w(CREA)l(TE)p 345 1135 V 17 w(RSEND\(handle,)h(sta)o(rt,)e(count,)h(datat)o(yp)q(e,)h (dest,)f(tag,)f(comm)m(\))117 1212 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))117 1287 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h (\(c)o(hoice\))117 1362 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117 1438 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)117 1513 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 1588 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 1663 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 1787 y FA(The)k(function)i Ft(MPI)p 523 1787 V 15 w(CREA)l(TE)p 707 1787 V 18 w(SSEND)f FA(creates)f(a)g(comm)o(unication)h(ob)s(ject)e(for)h (a)g(sync)o(hronous)75 1844 y(mo)q(de)g(send)h(op)q(eration.)75 1948 y Ft(MPI)p 160 1948 V 16 w(CREA)l(TE)p 345 1948 V 17 w(SSEND\(handle,)h (sta)o(rt,)e(count,)h(datat)o(yp)q(e,)g(dest,)h(tag,)e(comm)m(\))117 2025 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))117 2100 y(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 2175 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (sen)o(t)i(\(in)o(teger\))117 2250 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)117 2325 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 2400 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2476 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 2600 y FA(The)k(function)h Ft(MPI)p 522 2600 V 15 w(CREA)l(TE)p 706 2600 V 18 w(RECV)g FA(creates)e(a)h(comm)o(unication)g(ob)s(ject)f(for)h (a)f(receiv)o(e)i(op)q(era-)75 2656 y(tion.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 25 27 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(25)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(CREA)l(TE)p 345 49 V 17 w(RECV\(handle,)17 b(sta)o(rt,)e(count,)h(datat)o(yp)q(e,)h (source,)e(tag,)g(comm)m(\))117 126 y Fs(OUT)108 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))117 202 y(OUT)108 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g (\(c)o(hoice\))117 277 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117 352 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)117 427 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(source)j(or)d(MPI)p 1309 427 13 2 v 16 w(ANY)p 1418 427 V 15 w(SOUR)o(CE)h(\(in)o(teger\))117 502 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(or)h(MPI)p 1270 502 V 15 w(ANY)p 1378 502 V 16 w(T)m(A)o(G)f(\(in)o(teger\))117 577 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))166 702 y FA(A)21 b(comm)o(unication)g(\(send)g(or)f(receiv)o(e\))i(that)e(uses)h (a)f(prede\014ned)j(comm)o(unication)e(ob)s(ject)f(is)75 758 y(initiated)d(b)o(y)e(the)g(function)h Ft(MPI)p 661 758 14 2 v 16 w(ST)l(ART)p FA(.)75 862 y Ft(MPI)p 160 862 V 16 w(ST)l(ART\(handle\)) 117 939 y Fs(INOUT)62 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))166 1064 y FA(The)k(comm)o(unication)h(ob)s(ject)f (asso)q(ciated)h(with)f Ft(handle)i FA(should)f(b)q(e)g(an)f(ob)s(ject)g (that)f(w)o(as)h(cre-)75 1120 y(ated)f(b)o(y)g(one)h(of)f(the)g(previous)h (four)f(functions,)h(so)f(that)f(all)j(the)e(comm)o(unication)h(parameters)e (are)75 1176 y(already)k(de\014ned.)34 b(A)19 b(send)h(can)g(b)q(e)g(started) f(pro)o(vided)h(that)f(the)g(previous)h(send)h(using)f(the)f(same)75 1233 y(ob)s(ject)d(has)h(completed,)h(or)f(as)f(so)q(on)h(as)g(the)g(ob)s (ject)f(is)h(created,)g(if)h(it)f(has)g(not)f(y)o(et)h(b)q(een)h(used)f(in)h (a)75 1289 y(comm)o(unication.)j(In)16 b(addition,)h(if)f(the)f(comm)o (unication)h(mo)q(de)g(is)g Fp(ready)f FA(then)g(a)h(matc)o(hing)f(receiv)o (e)75 1346 y(should)k(b)q(e)f(p)q(osted.)28 b(The)18 b(send)g(bu\013er)g (should)h(not)e(b)q(e)i(up)q(dated)f(after)f(the)h(send)g(is)h(started,)e(un) o(til)75 1402 y(the)e(op)q(eration)h(completes.)166 1459 y(A)e(receiv)o(e)g (can)g(b)q(e)g(started)f(pro)o(vided)i(that)e(the)h(preceding)h(receiv)o(e)f (using)h(the)f(same)f(ob)s(ject)g(has)75 1515 y(completed,)f(or)f(as)f(so)q (on)h(as)f(the)h(ob)s(ject)f(is)i(created,)f(if)g(it)g(has)g(not)g(y)o(et)f (b)q(een)i(used)f(in)h(a)f(comm)o(unication.)75 1572 y(The)18 b(receiv)o(e)h(bu\013er)f(should)h(not)e(b)q(e)i(accessed)g(after)e(the)h (receiv)o(e)h(is)f(started,)g(un)o(til)h(the)f(op)q(eration)75 1628 y(completes.)166 1685 y(The)23 b(call)g(is)g(non)o(blo)q(c)o(king,)i (with)e(similar)g(seman)o(tics)g(as)f(the)h(non)o(blo)q(c)o(king)g(comm)o (unication)75 1741 y(op)q(erations)15 b(describ)q(ed)i(in)g(Section)f(1.8.) 166 1797 y(A)10 b(comm)o(unication)h(started)e(with)i(a)f(call)h(to)e Ft(MPI)p 1016 1797 V 16 w(ST)l(ART)i FA(is)g(completed)g(b)o(y)f(a)g(call)h (to)f Ft(MPI)p 1740 1797 V 16 w(W)l(AIT)p FA(,)75 1854 y Ft(MPI)p 160 1854 V 16 w(TEST)p FA(,)i(or)f(one)h(of)g(the)g(deriv)o(ed)h(functions)f (describ)q(ed)i(in)f(Section)g(1.8.5.)k(These)12 b(comm)o(unication)75 1910 y(completion)18 b(functions)g(do)e(not)h(deallo)q(cate)h(the)f(comm)o (unication)g(ob)s(ject,)g(and)g(this)g(can)g(b)q(e)g(reused)75 1967 y(anew)f(b)o(y)g(a)g Ft(MPI)p 380 1967 V 15 w(ST)l(ART)h FA(call.)24 b(The)16 b(ob)s(ject)g(needs)g(to)g(b)q(e)g(explicitly)j(deallo)q (cated)f(b)o(y)e(a)f(call)i(to)f(the)75 2023 y(function)f Ft(MPI)p 337 2023 V 15 w(COMMOBJ)p 578 2023 V 18 w(FREE)p FA(,)f(b)q(elo)o(w.)20 b(W)l(e)13 b(th)o(us)h(ha)o(v)o(e)f(t)o(w)o(o)g(t)o(yp)q(es)g(of)h(comm)o (unication)g(ob)s(jects:)75 2080 y Fw(p)q(ersisten)o(t)k FA(ob)s(jects,)f (whic)o(h)i(are)e(allo)q(cated)i(b)o(y)f(a)f(call)i(to)e Ft(MPI)p 1230 2080 V 16 w(CREA)l(TE)p 1415 2080 V 18 w(xxx)p FA(,)h(and)g(are)f (explicitly)75 2136 y(deallo)q(cated)j(b)o(y)e Ft(MPI)p 468 2136 V 16 w(COMMOBJ)p 710 2136 V 18 w(FREE)p FA(,)g(and)h Fw(ephemeral)f FA(ob)s(jects)g(that)g(p)q(ersist)h(for)e(one)i(com-)75 2193 y(m)o(unication)14 b(only:)19 b(they)13 b(are)g(created)g(b)o(y)g(a)g(non)o (blo)q(c)o(king)h(comm)o(unication)g(initiation)h(function,)f(and)75 2249 y(are)h(freed)g(b)o(y)h(the)f(comm)o(unication)h(completion)g(call.)75 2353 y Ft(MPI)p 160 2353 V 16 w(COMMOBJ)p 402 2353 V 17 w(FREE\(handle\))117 2430 y Fs(INOUT)62 b Ft(handle)465 b Fs(comm)o(unication)11 b(ob)r(ject)j(\(handle\))166 2554 y FA(Marks)e(the)h(comm)o(unication)g(ob)s (ject)f(for)g(deallo)q(cation.)21 b(The)13 b(ob)s(ject)f(will)j(b)q(e)e (deallo)q(cated)h(when)75 2611 y(there)23 b(are)f(no)g(p)q(ending)j(comm)o (unications)e(in)o(v)o(olving)g(this)g(ob)s(ject,)h(at)e(whic)o(h)h(p)q(oin)o (t)g(the)g(handle)75 2667 y(b)q(ecomes)16 b(n)o(ull.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 26 28 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(26)166 49 y(The)18 b(call)g(is)h(non)o(blo)q(c)o(king.)28 b(It)18 b(is)g(p)q(ermissible)i(to)d(call)h Ft(MPI)p 1240 49 14 2 v 16 w(COMMOBJ)p 1482 49 V 18 w(FREE\(handle\))h FA(after)75 106 y(a)f(comm)o(unication)h(that)f(uses)38 b Ft(handle)19 b FA(has)g(b)q(een)h(initiated,)g(but)f(b)q(efore)g(it)f(has)h(completed.)31 b(The)75 162 y(ob)s(ject)15 b(will)j(b)q(e)f(deallo)q(cated)g(after)e(the)h (comm)o(unication)h(completes.)22 b(It)16 b(is)h(erroneous)f(to)f(initiate)i (a)75 219 y(comm)o(unication)f(with)f(a)g(handle)i(that)d(has)h(b)q(een)i (freed.)166 275 y(A)e(correct)f(in)o(v)o(o)q(cation)i(of)e(the)h(functions)h (describ)q(ed)g(in)g(this)f(section)h(will)g(o)q(ccur)g(in)f(a)g(sequence)75 332 y(of)g(the)g(form)613 388 y Fw(Create)j FA(\()p Fw(Start)f(Complete)p FA(\))1172 369 y Fj(\003)1209 388 y Fw(F)l(ree)f Fi(;)75 471 y FA(where)j Fl(\003)f FA(indicates)i(zero)e(or)g(more)g(rep)q(etitions.)31 b(If)18 b(the)h(same)f(comm)o(unication)h(ob)s(ject)f(is)h(used)g(in)75 528 y(sev)o(eral)12 b(concurren)o(t)g(threads,)g(it)g(is)h(the)f(user)g(resp) q(onsibilit)o(y)i(to)d(co)q(ordinate)i(calls)g(so)e(that)g(the)h(correct)75 584 y(sequence)k(is)g(ob)q(ey)o(ed.)166 641 y(A)c(send)g(op)q(eration)f (initiated)j(with)d Ft(MPI)p 874 641 V 16 w(ST)l(ART)i FA(can)e(b)q(e)i(matc) o(hed)e(with)h(an)o(y)f(receiv)o(e)i(op)q(eration)75 697 y(and,)g(lik)o (ewise,)h(a)e(receiv)o(e)i(op)q(eration)e(initiated)i(with)f Ft(MPI)p 1098 697 V 16 w(ST)l(ART)g FA(can)g(receiv)o(e)g(messages)f (generated)75 754 y(b)o(y)j(an)o(y)g(send)h(op)q(eration.)75 897 y Fx(1.11)60 b(Send-receive)75 998 y FA(The)17 b Fw(send-receiv)o(e)f FA(op)q(erations)h(com)o(bines)h(in)f(one)g(call)h(the)f(sending)h(of)e(a)h (message)f(to)g(one)h(desti-)75 1055 y(nation)f(and)g(the)g(receiving)i(of)d (another)h(message,)f(from)g(another)h(destination,)g(p)q(ossibly)i(the)e (same.)75 1111 y(The)j Fw(exc)o(hange)g FA(op)q(erations)g(are)f(the)h(same)f (as)h(send-receiv)o(e,)i(except)e(that)f(the)h(send)g(bu\013er)g(and)75 1168 y(the)14 b(receiv)o(e)h(bu\013er)f(are)g(iden)o(tical.)22 b(A)14 b(send-receiv)o(e)i(op)q(eration)e(is)h(v)o(ery)f(useful)h(for)f (executing)h(a)f(shift)75 1224 y(op)q(eration)g(across)g(a)f(c)o(hain)i(of)f (pro)q(cesses.)20 b(If)14 b(blo)q(c)o(king)h(sends)g(and)f(receiv)o(es)h(are) f(used)g(for)g(suc)o(h)g(shift,)75 1281 y(then)19 b(one)g(needs)h(to)e(order) h(correctly)g(the)g(sends)g(and)g(receiv)o(es)h(\(e.g.,)e(ev)o(en)i(pro)q (cesses)f(send,)h(next)75 1337 y(receiv)o(e,)14 b(o)q(dd)f(pro)q(cesses)g (receiv)o(e)g(\014rst,)f(next)h(send\))g(so)f(as)g(to)g(prev)o(en)o(t)g (cyclic)j(dep)q(endencies)g(that)d(lead)75 1394 y(to)k(deadlo)q(c)o(k.)25 b(When)17 b(a)g(send-receiv)o(e)h(or)e(exc)o(hange)h(op)q(eration)g(is)g (used,)g(the)g(comm)o(unication)g(sub-)75 1450 y(system)e(tak)o(es)f(care)h (of)g(these)g(issues.)21 b(Also,)15 b(a)g(send-receiv)o(e)i(op)q(eration)e (is)h(useful)g(for)f(implemen)o(ting)75 1506 y(remote)g(pro)q(cedure)h (calls.)166 1563 y(A)h(message)f(sen)o(t)h(b)o(y)g(a)f(send-receiv)o(e)j(or)d (exc)o(hange)h(op)q(eration)g(can)g(b)q(e)h(receiv)o(ed)g(b)o(y)e(a)h (regular)75 1619 y(receiv)o(e)f(op)q(eration,)f(and)h(vice)g(v)o(ersa.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 27 29 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(27)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(SENDRECV\(send)p 509 49 V 19 w(sta)o(rt,)11 b(send)p 721 49 V 18 w(count,)h(send)p 952 49 V 18 w(t)o(yp)q(e,)g(dest,)g(send)p 1259 49 V 18 w(tag,)f(recv)p 1439 49 V 16 w(sta)o(rt,)g(recv)p 1642 49 V 16 w(count,)h(recv)p 1865 49 V 16 w(t)o(yp)q(e,)75 106 y(source,)j(recv)p 302 106 V 17 w(tag,)f(comm)n(,)d(status\))117 183 y Fs(IN)155 b Ft(send)p 404 183 V 18 w(sta)o(rt)400 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er) h(\(c)o(hoice\))117 258 y(IN)155 b Ft(send)p 404 258 V 18 w(count)381 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 333 y(IN)155 b Ft(send)p 404 333 V 18 w(t)o(yp)q(e)406 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(send)i(bu\013er)117 407 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 482 y(IN)155 b Ft(send)p 404 482 V 18 w(tag)425 b Fs(send)15 b(tag)e(\(in)o(teger\))117 557 y(OUT)108 b Ft(recv)p 398 557 V 17 w(sta)o(rt)407 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g (\(c)o(hoice\))117 632 y(IN)155 b Ft(recv)p 398 632 V 17 w(count)388 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g (\(in)o(teger\))117 707 y(IN)155 b Ft(recv)p 398 707 V 17 w(t)o(yp)q(e)413 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(receiv)o(e)i(bu\013er)117 782 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o(teger\))117 857 y(IN)155 b Ft(recv)p 398 857 V 17 w(tag)432 b Fs(receiv)o(e)15 b(tag)f(\(in)o(teger\))117 931 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 1006 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 1131 y FA(Execute)e(a)f(blo)q(c)o(king)i(send)g(and)f(receiv) o(e)g(op)q(eration.)19 b(Both)13 b(send)g(and)g(receiv)o(e)h(use)f(the)f (same)h(tag)75 1187 y(v)m(alue)h(and)g(the)f(same)g(comm)o(unicator.)19 b(Ho)o(w)o(ev)o(er,)12 b(the)i(send)g(bu\013er)f(and)g(receiv)o(e)h(bu\013er) g(are)f(disjoin)o(t,)75 1244 y(and)i(ma)o(y)g(ha)o(v)o(e)g(di\013eren)o(t)g (length)h(and)f(di\013eren)o(t)h(datat)o(yp)q(es.)75 1347 y Ft(MPI)p 160 1347 V 16 w(EX)o(CHANGE\(sta)o(rt,)g(count,)g(datat)o(yp)q(e,)h (dest,)f(send)p 1060 1347 V 18 w(tag)f(source,)g(recv)p 1378 1347 V 17 w(tag,)f(comm)m(,)e(status\))117 1424 y Fs(IN)155 b Ft(sta)o(rt)501 b Fs(initial)12 b(address)j(of)f(send)g(and)g(receiv)o(e)h (bu\013er)g(\(c)o(hoice\))117 1499 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)9 b(of)g(elemen)o(ts)h(in)f(send)i(and)e(receiv)o(e)i (bu\013er)g(\(in)o(teger\))117 1574 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(send)i(and)f(receiv)o(e)h (bu\013er)117 1649 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h (\(in)o(teger\))117 1724 y(IN)155 b Ft(send)p 404 1724 V 18 w(tag)425 b Fs(send)15 b(message)f(tag)f(\(in)o(teger\))117 1799 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o(teger\))117 1874 y(IN)155 b Ft(recv)p 398 1874 V 17 w(tag)432 b Fs(receiv)o(e)15 b(message)f(tag)f(\(in)o(teger\))117 1948 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2023 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)166 2148 y FA(Execute)g(a)f(blo)q(c)o(king)i(send) f(and)g(receiv)o(e;)g(the)g(same)f(bu\013er)g(is)h(used)h(b)q(oth)e(for)g (the)h(send)g(and)g(for)75 2204 y(the)g(receiv)o(e.)166 2261 y(The)20 b(seman)o(tics)h(of)f(a)g(send-receiv)o(e)i(op)q(eration)e(is)h (what)f(w)o(ould)g(obtain)h(if)g(the)f(caller)h(fork)o(ed)75 2317 y(t)o(w)o(o)16 b(concurren)o(t)h(threads,)g(one)g(to)f(execute)i(the)f (send,)g(and)h(one)f(to)f(execute)i(the)f(receiv)o(e,)h(follo)o(w)o(ed)75 2374 y(b)o(y)f(a)f(join)h(of)f(these)h(t)o(w)o(o)e(threads.)24 b(A)16 b(send-receiv)o(e)j(cannot)d(b)q(e)h(implemen)o(ted)i(b)o(y)d(a)g(blo) q(c)o(king)i(send)75 2430 y(follo)o(w)o(ed)e(b)o(y)f(a)g(blo)q(c)o(king)i (receiv)o(e)g(or)e(a)g(blo)q(c)o(king)i(receiv)o(e,)f(follo)o(w)o(ed)g(b)o(y) f(a)g(blo)q(c)o(king)i(send.)k(Consider)75 2486 y(the)15 b(follo)o(wing)h(co) q(de,)g(where)f(t)o(w)o(o)f(pro)q(cesses)i(exc)o(hange)f(messages:)75 2591 y Fp(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2647 y(IF)h(\(rank.EQ.0\))123 2704 y(THEN)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 28 30 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(28)170 49 y Fp(CALL)24 b(MPI_SENDRECV\(send_buff,)c (send_count,)j(MPI_REAL,)f(0,)i(send_tag,)242 106 y(recv_buff,)f(recv_count,) f(MPI_REAL,)h(1,)g(recv_tag,)g(comm,)g(status,)g(ierr\))123 162 y(ELSE)170 219 y(CALL)h(MPI_SENDRECV\(send_buff,)c(send_count,)j (MPI_REAL,)f(1,)i(send_tag,)242 275 y(recv_buff,)f(recv_count,)f(MPI_REAL,)h (0,)g(tag,)g(comm,)h(status,)f(ierr\))75 332 y(END)g(IF)166 438 y FA(If)12 b(the)f(send)h(receiv)o(es)g(are)g(replaced)g(either)g(b)o(y)g (blo)q(c)o(king)g(send,)h(follo)o(w)o(ed)f(b)o(y)f(blo)q(c)o(king)i(receiv)o (e,)f(or)75 494 y(blo)q(c)o(king)g(receiv)o(e,)h(follo)o(w)o(ed)e(b)o(y)g (blo)q(c)o(king)i(send,)f(then)g(the)f(co)q(de)g(ma)o(y)g(deadlo)q(c)o(k.)19 b(On)12 b(the)f(other)g(hand,)75 551 y(send-receiv)o(e)21 b(can)f(b)q(e)h (implemen)o(ted)g(using)g(non)o(blo)q(c)o(king)g(sends)f(and)g(receiv)o(es.) 35 b(Note)19 b(that)g(some)75 607 y(system)g(bu\013ering)h(is)g(required)h (for)e(a)g(correct)g(implemen)o(tation)i(of)e Ft(MPI)p 1400 607 14 2 v 16 w(EX)o(CHANGE)p FA(.)i(\(Consider)75 664 y(the)15 b(last)g(example,)h(with)g(send-receiv)o(e)g(replaced)h(b)o(y)e(exc)o (hange.\))75 807 y Fx(1.12)60 b(Null)19 b(p)n(ro)r(cesses)75 908 y FA(In)c(man)o(y)e(instances,)i(it)f(is)h(con)o(v)o(enien)o(t)g(to)e(sp) q(ecify)j(a)d(\\dumm)o(y")h(source)g(or)g(destination)h(for)e(comm)o(u-)75 965 y(nication.)29 b(This)18 b(simpli\014es)i(the)e(co)q(de)g(that)f(is)h (needed)i(for)d(dealing)i(with)f(b)q(oundaries,)h(e.g.,)e(in)i(the)75 1021 y(case)c(of)g(a)g(non-circular)i(shift)e(done)h(with)f(calls)i(to)d (send-receiv)o(e.)166 1078 y(The)f(sp)q(ecial)j(v)m(alue)26 b Fm(MPI)p 608 1078 13 2 v 15 w(PROCNULL)12 b FA(can)i(b)q(e)g(used)g (instead)g(of)f(a)g(rank)g(wherev)o(er)g(a)g(source)g(or)g(a)75 1134 y(destination)i(parameter)e(is)i(required)g(in)g(a)e(call.)21 b(A)14 b(comm)o(unication)h(with)f(pro)q(cess)27 b Fm(MPI)p 1649 1134 V 15 w(PROCNULL)75 1191 y FA(has)13 b(no)h(e\013ect:)k(a)13 b(send)h(to)25 b Fm(MPI)p 633 1191 V 14 w(PROCNULL)13 b FA(succeeds)i(and)e (returns)g(as)g(so)q(on)h(as)f(p)q(ossible.)21 b(A)13 b(receiv)o(e)75 1247 y(from)26 b Fm(MPI)p 271 1247 V 14 w(PROCNULL)13 b FA(succeeds)i(and)f (returns)g(as)f(so)q(on)h(as)f(p)q(ossible)j(with)e(no)g(mo)q(di\014cations)h (to)e(the)75 1303 y(receiv)o(e)18 b(bu\013er.)25 b(When)17 b(a)g(receiv)o(e)h(with)f Ft(source)h FA(=)32 b Fm(MPI)p 1090 1303 V 15 w(PROCNULL)16 b FA(is)i(executed)f(then)h(the)f(status)75 1360 y(ob)s(ject)e(returns)30 b Ft(source)16 b FA(=)29 b Fm(MPI)p 661 1360 V 14 w(PROCNULL)p FA(,)14 b Ft(tag)h FA(=)30 b Fm(MPI)p 1132 1360 V 14 w(ANY)p 1231 1360 V 15 w(T)m(A)o(G)14 b FA(and)h Ft(count=0)p FA(.)75 1503 y Fx(1.13)60 b(Derived)19 b(datat)n(yp)r(es)75 1605 y FA(Up)c(to)f(no)o(w,)g(all)i(p)q(oin)o(t)f(to)f(p)q(oin)o(t)h(comm)o (unication)h(in)o(v)o(olv)o(ed)f(only)g(con)o(tiguous)g(bu\013ers)g(con)o (taining)g(a)75 1661 y(sequence)g(of)e(elemen)o(ts)h(of)f(the)g(same)g(t)o (yp)q(e.)19 b(This)14 b(is)g(to)q(o)f(constraining)h(on)f(t)o(w)o(o)f(accoun) o(ts:)19 b(One)14 b(often)75 1718 y(w)o(an)o(ts)f(to)h(pass)g(messages)g (that)f(con)o(tain)i(v)m(alues)g(with)g(di\013eren)o(t)f(datat)o(yp)q(es)g (\(e.g.,)f(an)h(in)o(teger)h(coun)o(t,)75 1774 y(follo)o(w)o(ed)i(b)o(y)f(a)g (sequence)i(of)e(real)h(n)o(um)o(b)q(ers\);)f(and)h(one)g(often)f(w)o(an)o (ts)f(to)h(send)h(noncon)o(tiguous)g(data)75 1830 y(\(e.g.)h(a)12 b(sub-blo)q(c)o(k)i(of)e(a)g(matrix\).)18 b(One)13 b(solution)g(is)g(to)f (pro)o(vide)h(functions)g(that)e(pac)o(k)i(noncon)o(tiguous)75 1887 y(data)f(in)o(to)h(a)g(con)o(tiguous)g(bu\013er)g(at)f(the)h(sender)h (site)f(and)g(unpac)o(k)h(it)f(bac)o(k)g(at)f(the)h(receiv)o(er)h(site.)19 b(This)75 1943 y(has)f(the)g(disadv)m(an)o(tage)g(of)g(requiring)h (additional)h(memory-to-memory)c(cop)o(y)i(op)q(erations)g(at)g(b)q(oth)75 2000 y(sites,)12 b(ev)o(en)f(when)g(the)g(comm)o(unication)g(subsystem)g(has) f(scatter-gather)g(capabilities.)20 b(Instead,)12 b(MPI)75 2056 y(pro)o(vides)22 b(mec)o(hanisms)f(to)g(sp)q(ecify)h(more)f(general,)i (mixed)f(and)f(noncon)o(tiguous)g(comm)o(unication)75 2113 y(bu\013ers.)i(It)16 b(is)g(up)h(to)f(the)g(implemen)o(tation)h(to)f(decide)i (whether)e(data)f(should)j(b)q(e)e(\014rst)g(pac)o(k)o(ed)g(in)h(a)75 2169 y(con)o(tiguous)h(bu\013er)h(b)q(efore)f(b)q(eeing)i(transmitted,)f(or)e (whether)i(it)g(can)f(b)q(e)h(collected)h(directly)g(from)75 2226 y(where)15 b(it)h(resides.)166 2282 y(The)d(general)g(mec)o(hanisms)g (pro)o(vided)g(here)g(allo)o(ws)g(one)f(to)g(transfer)g(directly)l(,)i (without)f(cop)o(ying,)75 2339 y(ob)s(jects)k(of)g(v)m(arious)h(shap)q(e)h (and)f(size.)28 b(It)17 b(is)i(not)e(assumed)h(that)f(the)g(MPI)h(library)g (is)h(cognizan)o(t)e(of)75 2395 y(the)f(ob)s(jects)g(declared)h(in)g(the)f (host)g(language;)g(th)o(us,)g(if)g(one)g(w)o(an)o(ts)f(to)h(transfer)f(a)h (structure,)f(or)h(an)75 2451 y(arra)o(y)d(section,)i(it)f(will)i(b)q(e)f (necessary)f(to)g(pro)o(vide)h(in)g(MPI)f(a)g(de\014nition)i(of)e(a)g(comm)o (unication)h(bu\013er)75 2508 y(that)e(mimics)j(the)e(de\014nition)i(of)e (the)g(structure)g(or)f(arra)o(y)g(section)i(in)g(question.)20 b(These)15 b(facilities)h(can)75 2564 y(b)q(e)j(used)g(b)o(y)f(library)h (designers)h(to)d(de\014ne)j(comm)o(unication)f(functions)g(that)f(can)g (transfer)g(ob)s(jects)75 2621 y(de\014ned)g(in)f(the)f(host)g(language)g({)g (b)o(y)h(deco)q(ding)g(their)g(de\014nitions)h(as)e(a)o(v)m(ailable)i(in)f(a) f(sym)o(b)q(ol)h(table)75 2677 y(or)e(a)g(dop)q(e)g(v)o(ector.)20 b(Suc)o(h)15 b(higher-lev)o(el)j(comm)o(unication)e(functions)g(are)f(not)g (part)f(of)h(MPI.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 29 31 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(29)166 49 y(More)12 b(general)g(comm)o(unication)i(bu\013ers)e (are)g(sp)q(eci\014ed)i(b)o(y)f(replacing)g(the)g(basic)g(datat)o(yp)q(es)f (that)75 106 y(ha)o(v)o(e)18 b(used)h(so)f(far)g(with)g(deriv)o(ed)i(datat)o (yp)q(es)e(that)f(are)h(constructed)h(from)e(basic)i(datat)o(yp)q(es)f(using) 75 162 y(the)13 b(constructors)e(describ)q(ed)j(in)g(this)f(section.)19 b(These)13 b(metho)q(ds)f(of)g(constructing)h(deriv)o(ed)h(datat)o(yp)q(es)75 219 y(can)h(b)q(e)h(applied)h(recursiv)o(ely)l(.)166 275 y(A)e Fw(general)j(datat)o(yp)q(e)e FA(is)g(an)f(opaque)h(ob)s(ject)e(that)h(sp)q (eci\014es)i(t)o(w)o(o)d(things:)143 369 y Fl(\017)23 b FA(A)15 b(sequence)h(of)f(basic)h(datat)o(yp)q(es)143 463 y Fl(\017)23 b FA(A)15 b(sequence)h(of)f(in)o(teger)g(\(b)o(yte\))g(displacemen)o(ts)166 557 y(The)22 b(displacemen)o(ts)g(are)g(not)f(required)h(to)f(b)q(e)h(p)q (ositiv)o(e,)i(distinct,)g(or)d(in)h(increasing)h(order;)75 613 y(therefore)17 b(the)g(order)g(of)f(items)i(need)g(not)f(coincide)i(with) e(their)h(order)f(in)g(store,)g(and)g(an)g(item)h(ma)o(y)75 669 y(app)q(ear)c(more)g(than)g(once.)20 b(W)l(e)14 b(call)h(suc)o(h)g(pair)f (of)g(sequences)h(\(or)e(sequence)j(of)d(pairs\))h(a)g Fw(t)o(yp)q(e)i(map)p FA(.)75 726 y(Let)476 782 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 789 y Fh(0)867 782 y Fi(;)8 b(disp)972 789 y Fh(0)990 782 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 789 y Fg(n)p Fj(\000)p Fh(1)1253 782 y Fi(;)h(disp)1358 789 y Fg(n)p Fj(\000)p Fh(1)1421 782 y FA(\))p Fl(g)p Fi(;)75 866 y FA(b)q(e)18 b(suc)o(h)g(a)f(t)o(yp)q(e)h(map,)f(where)h Fi(ty)r(pe)723 873 y Fg(i)754 866 y FA(are)f(basic)i(t)o(yp)q(es,)e(and)h Fi(disp)1259 873 y Fg(i)1290 866 y FA(are)f(displacemen)o(ts.)28 b(This)18 b(t)o(yp)q(e)75 922 y(map,)j(together)e(with)i(a)f(base)g(address)h Fq(start)p FA(,)g(sp)q(eci\014es)h(a)e(comm)o(unication)h(bu\013er:)30 b(the)20 b(comm)o(u-)75 979 y(nication)f(bu\013er)f(that)f(consists)i(of)e Fi(n)h FA(en)o(tries,)h(where)f(the)g Fi(i)p FA(-th)g(en)o(try)g(is)g(at)f (address)h Fi(star)q(t)13 b FA(+)g Fi(disp)1862 986 y Fg(i)75 1035 y FA(and)18 b(has)f(t)o(yp)q(e)g Fi(ty)r(pe)437 1042 y Fg(i)452 1035 y FA(.)26 b(A)18 b(message)f(assem)o(bled)h(from)e(suc)o(h)i (comm)o(unication)g(bu\013er)g(will)h(con)o(tain)e(a)75 1092 y(sequence)d(of)f Fi(n)h FA(v)m(alues,)g(where)g(v)m(alue)g Fi(i)f FA(has)g(t)o(yp)q(e)h Fi(ty)r(pe)1036 1099 y Fg(i)1050 1092 y FA(.)19 b(The)14 b(message)e(will)j(carry)e Fi(n)h FA(v)m(alues)g(of)f (t)o(yp)q(es)75 1148 y(\()p Fi(ty)r(pe)177 1155 y Fh(0)196 1148 y Fi(;)8 b(:::;)g(ty)r(pe)361 1155 y Fg(n)p Fj(\000)p Fh(1)423 1148 y FA(\).)166 1205 y(W)l(e)j(can)f(use)h(a)f(handle)i(to)e(a)g (general)h(datat)o(yp)q(e)f(as)h(an)f(argumen)o(t)g(in)h(a)f(send)i(or)e (receiv)o(e)h(op)q(eration,)75 1261 y(in)g(replacemen)o(t)g(of)f(a)g(basic)h (datat)o(yp)q(e)e(argumen)o(t.)18 b(The)10 b(op)q(eration)h Ft(MPI)p 1338 1261 14 2 v 15 w(SEND\(sta)o(rt,)h(1,)e(datat)o(yp)q(e,...\))75 1317 y FA(will)j(use)f(the)g(send)g(bu\013er)f(de\014ned)i(b)o(y)f(the)f (base)h(address)g Ft(sta)o(rt)f FA(and)h(the)g(general)g(datat)o(yp)q(e)f (asso)q(ciated)75 1374 y(with)h Ft(datat)o(yp)q(e)p FA(;)i(it)e(will)i (generate)d(a)h(message)f(with)h(the)g(t)o(yp)q(e)g(signature)g(determined)h (b)o(y)e(the)h Ft(datat)o(yp)q(e)75 1430 y FA(argumen)o(t.)23 b Ft(MPI)p 385 1430 V 16 w(RECV\(sta)o(rt,)18 b(1,)e(datat)o(yp)q(e,...\))24 b FA(will)19 b(use)e(the)f(receiv)o(e)i(bu\013er)f(de\014ned)h(b)o(y)e(the)h (base)75 1487 y(address)e Ft(sta)o(rt)h FA(and)f(the)g(general)h(datat)o(yp)q (e)f(asso)q(ciated)g(with)h Ft(datat)o(yp)q(e)p FA(.)166 1543 y(General)g(datat)o(yp)q(es)e(can)i(b)q(e)g(used)f(in)h(all)h(send)e(and)h (receiv)o(e)g(op)q(erations.)k(W)l(e)c(address)f(later)g(in)75 1600 y(Section)h(1.13.5)d(the)j(case)f(where)g(the)h(second)f(argumen)o(t)g Ft(count)h FA(has)f(v)m(alue)i Fi(>)c FA(1.)166 1656 y(The)d(basic)h(datat)o (yp)q(es)f(presen)o(ted)g(in)h(Section)g(1.2.1)e(are)h(particular)h(cases)f (of)f(a)h(general)h(datat)o(yp)q(e,)75 1713 y(and)19 b(are)f(prede\014ned.)32 b(Th)o(us,)19 b Ft(MPI)p 711 1713 V 16 w(INT)f FA(is)h(a)g(prede\014ned)h (handle)g(to)e(a)g(datat)o(yp)q(e)g(with)h(t)o(yp)q(e)g(map)75 1769 y Fl(f)p FA(\()p Fp(int)o Fi(;)8 b FA(0\))p Fl(g)p FA(,)16 b(with)h(one)h(en)o(try)f(of)g(t)o(yp)q(e)g Fp(int)g FA(and)g(displacemen)o (t)i(zero.)26 b(And)18 b(similarly)h(for)e(all)h(other)75 1826 y(basic)e(datat)o(yp)q(es.)166 1882 y(The)f Fw(exten)o(t)f FA(of)g(a)g(datat)o(yp)q(e)g(is)h(de\014ned)g(to)f(b)q(e)h(the)g(span)f(from) g(the)g(\014rst)g(b)o(yte)h(to)e(the)i(last)f(b)o(yte)75 1938 y(o)q(ccupied)19 b(b)o(y)d(en)o(tries)h(in)h(this)f(datat)o(yp)q(e,)f (rounded)h(up)g(to)f(satisfy)h(alignmen)o(t)g(requiremen)o(ts.)25 b(That)75 1995 y(is,)15 b(if)476 2051 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 2058 y Fh(0)867 2051 y Fi(;)8 b(disp)972 2058 y Fh(0)990 2051 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 2058 y Fg(n)p Fj(\000)p Fh(1)1253 2051 y Fi(;)h(disp)1358 2058 y Fg(n)p Fj(\000)p Fh(1)1421 2051 y FA(\))p Fl(g)p Fi(;)75 2135 y FA(then)501 2237 y Fi(l)q(b)p FA(\()p Fi(T)e(y)r(pemap)p FA(\))39 b(=)j(min)906 2263 y Fg(j)959 2237 y Fi(disp)1043 2244 y Fg(j)1060 2237 y Fi(;)489 2324 y(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))40 b(=)i(max)911 2350 y Fg(j)968 2324 y Fi(disp)1052 2331 y Fg(j)1079 2324 y FA(+)11 b Fi(siz)r(eof)5 b FA(\()p Fi(ty)r(pe)1357 2331 y Fg(j)1375 2324 y FA(\))p Fi(;)22 b FA(and)406 2410 y Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))42 b(=)g Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))j Fl(\000)h Fi(l)q(b)p FA(\()p Fi(T)c(y)r(pemap)p FA(\))i(+)j Fi(\017:)238 b FA(\(1.1\))75 2512 y(If)17 b Fi(ty)r(pe)206 2519 y Fg(i)237 2512 y FA(requires)g(alignmen)o(t)g(to)f(a)h(b)o(yte)f (address)h(that)f(is)h(is)g(a)f(m)o(ultiple)j(of)d Fi(k)1474 2519 y Fg(i)1487 2512 y FA(,)h(then)g Fi(\017)g FA(is)g(the)g(least)75 2569 y(nonnegativ)o(e)f(incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))16 b(to)e(the)h(next)h(m)o (ultiple)h(of)d(max)1794 2576 y Fg(i)1815 2569 y Fi(k)1839 2576 y Fg(i)1853 2569 y FA(.)166 2625 y(Example:)k(Assume)10 b(that)g Fi(T)c(y)r(pe)13 b FA(=)g Fl(f)p FA(\()p Fp(double)n Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)h FA(\(a)g Fp(double)h FA(at)g(displacemen)o(t)h(zero,)75 2682 y(follo)o(w)o(ed)17 b(b)o(y)g(a)f Fp(char)g FA(at)g(displacemen)o(t)i (eigh)o(t\).)24 b(Assume,)17 b(furthermore,)f(that)g(doubles)i(ha)o(v)o(e)e (to)g(b)q(e)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 30 32 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(30)75 49 y(strictly)15 b(aligned)g(at)f(addresses)h(that)e (are)h(m)o(ultiple)i(of)e(eigh)o(t.)20 b(Then,)15 b(the)f(exten)o(t)g(of)g (this)h(datat)o(yp)q(e)e(is)75 106 y(16)d(\(9)g(rounded)h(to)f(the)h(next)g (m)o(ultiple)h(of)f(8\).)17 b(A)11 b(datat)o(yp)q(e)f(that)g(consists)h(of)f (a)g(c)o(haracter)g(immediately)75 162 y(follo)o(w)o(ed)16 b(b)o(y)f(a)g(double)h(will)h(also)e(ha)o(v)o(e)g(an)g(exten)o(t)g(of)g(16.) 166 295 y Fn(Discussion:)166 344 y Fs(The)f(de\014nition)g(of)f(exten)o(t)i (is)f(motiv)n(ated)e(b)o(y)i(the)h(assumption)d(that)i(the)h(amoun)o(t)d(of)i (padding)f(added)h(at)75 394 y(the)f(end)g(of)f(eac)o(h)h(structure)i(in)d (an)h(arra)o(y)f(of)g(structures)j(is)e(the)g(least)g(needed)h(to)e(ful\014l) f(alignmen)o(t)g(constrain)o(ts.)75 444 y(This)k(condition)g(w)o(ould)f(b)q (e)i(violated)f(b)o(y)g(a)g(compiler)f(that)h(alw)o(a)o(ys)f(align)g (structures)k(in)d(an)g(arra)o(y)g(to)g(double)75 494 y(w)o(ord)i(b)q (oundaries,)g(ev)o(en)g(if)f(they)h(do)g(not)g(con)o(tain)f(double)g(w)o(ord) h(comp)q(onen)o(ts.)26 b(On)17 b(a)f(mac)o(hine)g(with)g(suc)o(h)75 544 y(h)o(yp)q(othetical)h(compiler)e(some)h(of)g(the)h(functions)g(in)o(tro) q(duced)h(b)q(elo)o(w)e(b)q(ecome)h(less)g(useful.)27 b(This)16 b(is)h(further)75 594 y(discussed)f(in)d(Section)h(1.13.6.)75 798 y Fr(1.13.1)49 b(Datat)o(yp)q(e)17 b(constructo)o(rs)75 884 y Ft(Contiguous)47 b FA(The)16 b(simplest)i(datat)o(yp)q(e)e(constructor) f(is)i Ft(MPI)p 1171 884 14 2 v 16 w(TYPE)p 1304 884 V 17 w(CONTIGUOUS)h FA(whic)o(h)f(allo)o(ws)75 940 y(replication)g(of)e(a)g(datat)o(yp)q(e)f(in)o (to)h(con)o(tiguous)h(lo)q(cations.)75 1044 y Ft(MPI)p 160 1044 V 16 w(TYPE)p 293 1044 V 17 w(CONTIGUOUS\(count,)h(oldt)o(yp)q(e,)f (newt)o(yp)q(e\))117 1121 y Fs(IN)155 b Ft(count)482 b Fs(replication)14 b(coun)o(t)g(\(nonnegativ)o(e)g(in)o(teger\))117 1196 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 1271 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))166 1396 y Ft(newt)o(yp)q(e)19 b FA(is)f(the)f(datat)o(yp)q(e)g (obtained)h(b)o(y)f(concatenating)h Ft(count)h FA(copies)f(of)f Ft(oldt)o(yp)q(e)p FA(.)27 b(Concate-)75 1452 y(nation)15 b(is)h(de\014ned)h (using)f Fq(extent)e FA(as)h(the)h(size)g(of)e(the)i(concatenated)f(copies.) 166 1509 y(Example:)26 b(let)18 b Ft(oldt)o(yp)q(e)h FA(ha)o(v)o(e)e(t)o(yp)q (e)h(map)g Fl(f)p FA(\()p Fp(double)n Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)p Fi(;)16 b FA(with)i(exten)o(t)g(16,) f(and)h(let)75 1565 y Ft(count)c FA(=)f(3.)19 b(The)d(t)o(yp)q(e)f(map)g(of)g (the)g(datat)o(yp)q(e)g(returned)g(b)o(y)h Ft(newt)o(yp)q(e)h FA(is)75 1622 y Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fi(;)g FA(\()p Fp(doub)o(le)m Fi(;)g FA(16\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(24\))p Fi(;)g FA(\()p Fp(doubl)o(e)m Fi(;)g FA(32\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(40\))p Fl(g)p FA(;)75 1678 y(i.e.,)15 b(alternating)g Fp(double)g FA(and)g Fp(char)g FA(elemen)o(ts,)g(with)h(displacemen)o(ts)g(0) p Fi(;)8 b FA(8)p Fi(;)g FA(16)p Fi(;)g FA(24)o Fi(;)f FA(32)p Fi(;)g FA(40.)166 1735 y(In)16 b(general,)f(assume)g(that)g(the)g(t)o(yp)q(e) g(map)g(of)g Ft(oldt)o(yp)q(e)h FA(is)75 1791 y Fl(f)p FA(\()p Fi(ty)r(pe)200 1798 y Fh(0)218 1791 y Fi(;)8 b(disp)323 1798 y Fh(0)341 1791 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 1798 y Fg(n)p Fj(\000)p Fh(1)604 1791 y Fi(;)g(disp)709 1798 y Fg(n)p Fj(\000)p Fh(1)773 1791 y FA(\))p Fl(g)p Fi(;)75 1847 y FA(with)16 b(exten)o(t)e Fi(extent)p FA(.)22 b(Then)15 b Ft(newt)o(yp)q(e)i FA(has)e(a)g(t)o(yp)q(e)h(map)f(with)g Fp(count)9 b Fl(\001)h Fp(n)15 b FA(en)o(tries)h(de\014ned)g(b)o(y:)75 1904 y Fl(f)p FA(\()p Fi(ty)r(pe)200 1911 y Fh(0)218 1904 y Fi(;)8 b(disp)323 1911 y Fh(0)341 1904 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 1911 y Fg(n)p Fj(\000)p Fh(1)604 1904 y Fi(;)g(disp)709 1911 y Fg(n)p Fj(\000)p Fh(1)773 1904 y FA(\))p Fi(;)g FA(\()p Fi(ty)r(pe)914 1911 y Fh(0)931 1904 y Fi(;)g(disp)1036 1911 y Fh(0)1053 1904 y FA(+)p Fi(extent)p FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1416 1911 y Fg(n)p Fj(\000)p Fh(1)1480 1904 y Fi(;)g(disp)1585 1911 y Fg(n)p Fj(\000)p Fh(1)1649 1904 y FA(+)p Fi(extent)p FA(\))p Fi(;)g(:::;)75 1960 y FA(\()p Fi(ty)r(pe)177 1967 y Fh(0)196 1960 y Fi(;)g(disp)301 1967 y Fh(0)328 1960 y FA(+)j Fi(extent)g Fl(\001)f FA(\()p Fp(count)f Fl(\000)h FA(1\)\))p Fi(;)e(:::;)g FA(\()o Fi(ty)r(pe)968 1967 y Fg(n)p Fj(\000)p Fh(1)1031 1960 y Fi(;)g(disp)1136 1967 y Fg(n)p Fj(\000)p Fh(1)1210 1960 y FA(+)i Fi(extent)i Fl(\001)d FA(\()p Fp(count)h Fl(\000)g FA(1\)\))p Fl(g)p Fi(:)75 2080 y Ft(V)o(ecto)o(r)45 b FA(The)16 b(function)g Ft(MPI)p 600 2080 V 16 w(TYPE)p 733 2080 V 17 w(VECTOR)g FA(is)g(a)f(more)g(general)h(constructor)e(that)h(allo)o (ws)g(repli-)75 2137 y(cation)20 b(of)g(a)g(datat)o(yp)q(e)f(in)o(to)h(lo)q (cations)h(that)e(consist)i(of)e(equally)j(spaced)f(blo)q(c)o(ks.)35 b(Eac)o(h)20 b(blo)q(c)o(k)g(is)75 2193 y(obtained)g(b)o(y)f(concatenating)h (the)f(same)g(n)o(um)o(b)q(er)h(of)f(copies)h(of)f(the)g(old)h(datat)o(yp)q (e.)32 b(The)19 b(spacing)75 2250 y(b)q(et)o(w)o(een)d(blo)q(c)o(ks)f(is)h(a) f(m)o(ultiple)i(of)e(the)g(exten)o(t)g(of)g(the)g(old)h(datat)o(yp)q(e.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 31 33 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(31)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(TYPE)p 293 49 V 17 w(VECTOR\()16 b(count,)g(oldt)o(yp)q(e,)g(stride,)f(blo)q (cklength,)i(newt)o(yp)q(e\))117 126 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(nonnegativ)o(e)f(in)o (teger\))117 202 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 277 y(IN)155 b Ft(stride)484 b Fs(n)o(um)o(b)q(er)13 b(of)f(elemen)o(ts)h(b)q(et)o(w)o(een)h(start)g(of)f (eac)o(h)g(blo)q(c)o(k)g(\(in)o(te-)905 333 y(ger\))117 408 y(IN)155 b Ft(blo)q(cklength)371 b Fs(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g (in)f(eac)o(h)i(blo)q(c)o(k)e(\(nonnegativ)o(e)h(in)o(te-)905 465 y(ger\))117 540 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))166 664 y FA(Example:)20 b(Assume,)15 b(again,)g(that)g Ft(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)g Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)p Fi(;)13 b FA(with)j(ex-)75 721 y(ten)o(t)g(16.)21 b(A)c(call)g(to)e Ft(MPI)p 527 721 V 16 w(TYPE)p 660 721 V 17 w(VECTOR\()i(2,)e(oldt)o(yp)q(e,)i(4,)f(3,)f(newt)o (yp)q(e\))j FA(will)g(create)d(the)h(datat)o(yp)q(e)75 777 y(with)g(t)o(yp)q(e)f(map)75 834 y Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fi(;)g FA(\()p Fp(doub)o(le)m Fi(;)g FA(16\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(24\))p Fi(;)g FA(\()p Fp(doubl)o(e)m Fi(;)g FA(32\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(40\))p Fi(;)75 890 y FA(\()p Fp(double)o Fi(;)g FA(64\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(72\))p Fi(;)g FA(\()p Fp(do)o(uble)m Fi(;)g FA(80\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(88\))p Fi(;)g FA(\()p Fp(dou)o(ble)m Fi(;)g FA(96\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(104\))p Fl(g)i FA(:)g(t)o(w)o(o)f(blo)q(c)o(ks)h(with)75 947 y(three)15 b(copies)h(eac)o(h)g(of)e(the)i(old)f(t)o(yp)q(e,)g(starting)g(4*16)f(apart.) 166 1003 y(A)h(call)h(to)f Ft(MPI)p 439 1003 V 16 w(TYPE)p 572 1003 V 17 w(VECTOR\(3,)g(oldt)o(yp)q(e,)h(-2,)f(1,)f(newt)o(yp)q(e\))j FA(will)g(create)e(the)g(datat)o(yp)q(e)75 1060 y Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fi(;)g FA(\()p Fp(doub)o(le)m Fi(;)g Fl(\000)p FA(32\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g Fl(\000)p FA(24\))p Fi(;)g FA(\()p Fp(double)m Fi(;)g Fl(\000)p FA(64\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g Fl(\000)p FA(56\))p Fl(g)p Fi(:)166 1116 y FA(In)16 b(general,)f(assume)g(that)g Ft(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e) g(map)75 1172 y Fl(f)p FA(\()p Fi(ty)r(pe)200 1179 y Fh(0)218 1172 y Fi(;)8 b(disp)323 1179 y Fh(0)341 1172 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 1179 y Fg(n)p Fj(\000)p Fh(1)604 1172 y Fi(;)g(disp)709 1179 y Fg(n)p Fj(\000)p Fh(1)773 1172 y FA(\))p Fl(g)p Fi(;)75 1229 y FA(with)k(exten)o(t)e Fi(extent)p FA(.)20 b(The)12 b(newly)g(created)f(datat)o(yp)q(e)g(has)g(a)g(t)o(yp)q(e)g(map)g (with)g Fp(count)r Fl(\001)r Fp(blocklength)p Fl(\001)r Fi(n)75 1285 y FA(en)o(tries:)75 1342 y Fl(f)p FA(\()p Fi(ty)r(pe)200 1349 y Fh(0)218 1342 y Fi(;)d(disp)323 1349 y Fh(0)341 1342 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 1349 y Fg(n)p Fj(\000)p Fh(1)604 1342 y Fi(;)g(disp)709 1349 y Fg(n)p Fj(\000)p Fh(1)773 1342 y FA(\))p Fi(;)75 1398 y FA(\()p Fi(ty)r(pe)177 1405 y Fh(0)196 1398 y Fi(;)g(disp)301 1405 y Fh(0)328 1398 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)702 1405 y Fg(n)p Fj(\000)p Fh(1)765 1398 y Fi(;)g(disp)870 1405 y Fg(n)p Fj(\000)p Fh(1)944 1398 y FA(+)i Fi(extent)p FA(\))p Fi(;)e(:::;)75 1455 y FA(\()p Fi(ty)r(pe)177 1462 y Fh(0)196 1455 y Fi(;)g(disp)301 1462 y Fh(0)320 1455 y FA(+)s(\()p Fp(blocklength)q Fl(\000)s FA(1\))s Fl(\001)s Fi(extent)p FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1066 1462 y Fg(n)p Fj(\000)p Fh(1)1128 1455 y Fi(;)h(disp)1233 1462 y Fg(n)p Fj(\000)p Fh(1)1300 1455 y FA(+)s(\()p Fp(blocklength)p Fl(\000)s FA(1\))s Fl(\001)s Fi(extent)p FA(\))p Fi(;)75 1511 y FA(\()p Fi(ty)r(pe)177 1518 y Fh(0)196 1511 y Fi(;)g(disp)301 1518 y Fh(0)328 1511 y FA(+)j Fp(stride)e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)878 1518 y Fg(n)p Fj(\000)p Fh(1)941 1511 y Fi(;)g(disp)1046 1518 y Fg(n)p Fj(\000)p Fh(1)1120 1511 y FA(+)i Fp(stride)g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 1568 y FA(\()p Fi(ty)r(pe)177 1575 y Fh(0)196 1568 y Fi(;)g(disp)301 1575 y Fh(0)328 1568 y FA(+)j(\()p Fp(stride)d FA(+)j Fp(blocklength)e Fl(\000)h FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 1624 y FA(\()p Fi(ty)r(pe)177 1631 y Fg(n)p Fj(\000)p Fh(1)242 1624 y Fi(;)g(disp)347 1631 y Fg(n)p Fj(\000)p Fh(1)421 1624 y FA(+)i(\()p Fp(stride)f FA(+)i Fp(blocklength)d Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(::::;)75 1681 y FA(\()p Fi(ty)r(pe)177 1688 y Fh(0)196 1681 y Fi(;)g(disp)301 1688 y Fh(0)328 1681 y FA(+)j Fp(stride)e Fl(\001)h FA(\()p Fp(count)f Fl(\000)h FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 1737 y FA(\()p Fi(ty)r(pe)177 1744 y Fg(n)p Fj(\000)p Fh(1)242 1737 y Fi(;)g(disp)347 1744 y Fg(n)p Fj(\000)p Fh(1)421 1737 y FA(+)i Fp(stride)g Fl(\001)f FA(\()p Fp(count)h Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 1793 y FA(\()p Fi(ty)r(pe)177 1800 y Fh(0)196 1793 y Fi(;)g(disp)301 1800 y Fh(0)328 1793 y FA(+)j(\()p Fp(stride)d Fl(\001)i FA(\()p Fp(count)f Fl(\000)i FA(1\))e(+)i Fp(blocklength)d Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)75 1850 y FA(\()p Fi(ty)r(pe)177 1857 y Fg(n)p Fj(\000)p Fh(1)242 1850 y Fi(;)g(disp)347 1857 y Fg(n)p Fj(\000)p Fh(1)421 1850 y FA(+)i(\()p Fp(stride)f Fl(\001)h FA(\()p Fp(count)f Fl(\000)i FA(1\))e(+)i Fp(blocklength)d Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fl(g)166 1906 y FA(A)16 b(call)i(to)e Ft(MPI)p 443 1906 V 16 w(TYPE)p 576 1906 V 16 w(CONTIGUOUS\(count,)k(oldt)o(yp)q(e,)d(newt)o(yp)q (e\))h FA(is)f(equiv)m(alen)o(t)h(to)e(a)g(call)i(to)75 1963 y Ft(MPI)p 160 1963 V 16 w(TYPE)p 293 1963 V 17 w(VECTOR\(count,)g(oldt)o(yp) q(e,)g(1,)e(1,)g(newt)o(yp)q(e\))p FA(,)i(or)e(to)g(a)h(call)g(to)f Ft(MPI)p 1495 1963 V 16 w(TYPE)p 1628 1963 V 17 w(VECTOR\(1,)75 2019 y(oldt)o(yp)q(e,)g(n,)f(count,)h(newt)o(yp)q(e\))p FA(,)h Fp(n)e FA(arbitrary)l(.)75 2139 y Ft(Hvecto)o(r)45 b FA(The)19 b(function)h Ft(MPI)p 631 2139 V 16 w(TYPE)p 764 2139 V 17 w(HVECTOR)g FA(is)f(iden)o(tical)i(to)d Ft(MPI)p 1389 2139 V 16 w(TYPE)p 1522 2139 V 17 w(VECTOR)p FA(,)h(except)75 2196 y(that)h Ft(stride)h FA(is)g(giv)o(en)g(in)g(b)o(ytes,)g(rather)f(then)g(in)i (elemen)o(ts.)36 b(The)20 b(use)h(for)f(b)q(oth)g(t)o(yp)q(es)h(of)f(v)o (ector)75 2252 y(constructors)14 b(is)i(illustrated)h(in)f(Section)g(1.13.6.) i(\()p Fp(H)c FA(is)i(used)g(for)e(\\heterogeneous"\).)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 32 34 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(32)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(TYPE)p 293 49 V 17 w(HVECTOR\()16 b(count,)g(oldt)o(yp)q(e,)g(stride,)g(blo)q (cklength,)g(newt)o(yp)q(e\))117 126 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g(\(nonnegativ)o(e)f(in)o (teger\))117 202 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 277 y(IN)155 b Ft(stride)484 b Fs(n)o(um)o(b)q(er)13 b(of)g(b)o(ytes)h(b)q(et)o(w)o(een)h(start)f(of)f (eac)o(h)h(blo)q(c)o(k)f(\(in)o(teger\))117 352 y(IN)155 b Ft(blo)q(cklength)371 b Fs(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o (h)i(blo)q(c)o(k)e(\(nonnegativ)o(e)h(in)o(te-)905 408 y(ger\))117 483 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))166 608 y FA(Assume)h(that)g Ft(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)75 664 y Fl(f)p FA(\()p Fi(ty)r(pe)200 671 y Fh(0)218 664 y Fi(;)8 b(disp)323 671 y Fh(0)341 664 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 671 y Fg(n)p Fj(\000)p Fh(1)604 664 y Fi(;)g(disp)709 671 y Fg(n)p Fj(\000)p Fh(1)773 664 y FA(\))p Fl(g)p Fi(;)75 721 y FA(with)k(exten)o(t)e Fi(extent)p FA(.)20 b(The)12 b(newly)g(created)f(datat)o(yp)q(e)g(has)g(a)g(t)o(yp)q(e)g (map)g(with)g Fp(count)r Fl(\001)r Fp(blocklength)p Fl(\001)r Fi(n)75 777 y FA(en)o(tries:)75 834 y Fl(f)p FA(\()p Fi(ty)r(pe)200 841 y Fh(0)218 834 y Fi(;)d(disp)323 841 y Fh(0)341 834 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 841 y Fg(n)p Fj(\000)p Fh(1)604 834 y Fi(;)g(disp)709 841 y Fg(n)p Fj(\000)p Fh(1)773 834 y FA(\))p Fi(;)75 890 y FA(\()p Fi(ty)r(pe)177 897 y Fh(0)196 890 y Fi(;)g(disp)301 897 y Fh(0)328 890 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)702 897 y Fg(n)p Fj(\000)p Fh(1)765 890 y Fi(;)g(disp)870 897 y Fg(n)p Fj(\000)p Fh(1)944 890 y FA(+)i Fi(extent)p FA(\))p Fi(;)e(:::;)75 947 y FA(\()p Fi(ty)r(pe)177 954 y Fh(0)196 947 y Fi(;)g(disp)301 954 y Fh(0)320 947 y FA(+)s(\()p Fp(blocklength)q Fl(\000)s FA(1\))s Fl(\001)s Fi(extent)p FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1066 954 y Fg(n)p Fj(\000)p Fh(1)1128 947 y Fi(;)h(disp)1233 954 y Fg(n)p Fj(\000)p Fh(1)1300 947 y FA(+)s(\()p Fp(blocklength)p Fl(\000)s FA(1\))s Fl(\001)s Fi(extent)p FA(\))p Fi(;)75 1003 y FA(\()p Fi(ty)r(pe)177 1010 y Fh(0)196 1003 y Fi(;)g(disp)301 1010 y Fh(0)328 1003 y FA(+)j Fp(stride)o FA(\))p Fi(;)d(:::;)g FA(\()o Fi(ty)r(pe)717 1010 y Fg(n)p Fj(\000)p Fh(1)780 1003 y Fi(;)g(disp)885 1010 y Fg(n)p Fj(\000)p Fh(1)959 1003 y FA(+)i Fp(stride)o FA(\))p Fi(;)e(:::;)75 1060 y FA(\()p Fi(ty)r(pe)177 1067 y Fh(0)196 1060 y Fi(;)g(disp)301 1067 y Fh(0)328 1060 y FA(+)j Fp(stride)e FA(+)h(\()p Fp(blocklength)f Fl(\000)h FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 1116 y FA(\()p Fi(ty)r(pe)177 1123 y Fg(n)p Fj(\000)p Fh(1)242 1116 y Fi(;)g(disp)347 1123 y Fg(n)p Fj(\000)p Fh(1)421 1116 y FA(+)i Fp(stride)g FA(+)g(\()p Fp(blocklength)e Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(::::;)75 1172 y FA(\()p Fi(ty)r(pe)177 1179 y Fh(0)196 1172 y Fi(;)g(disp)301 1179 y Fh(0)328 1172 y FA(+)j Fp(stride)e Fl(\001)h FA(\()p Fp(count)f Fl(\000)h FA(1\)\))p Fi(;)e(:::;)75 1229 y FA(\()p Fi(ty)r(pe)177 1236 y Fg(n)p Fj(\000)p Fh(1)242 1229 y Fi(;)g(disp)347 1236 y Fg(n)p Fj(\000)p Fh(1)421 1229 y FA(+)i Fp(stride)g Fl(\001)f FA(\()p Fp(count)h Fl(\000)g FA(1\)\))p Fi(;)e(:::;)75 1285 y FA(\()p Fi(ty)r(pe)177 1292 y Fh(0)196 1285 y Fi(;)g(disp)301 1292 y Fh(0)328 1285 y FA(+)j Fp(stride)e Fl(\001)h FA(\()p Fp(count)f Fl(\000)h FA(1\))g(+)g(\()p Fp(blocklength)e Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)75 1342 y FA(\()p Fi(ty)r(pe)177 1349 y Fg(n)p Fj(\000)p Fh(1)242 1342 y Fi(;)g(disp)347 1349 y Fg(n)p Fj(\000)p Fh(1)421 1342 y FA(+)i Fp(stride)g Fl(\001)f FA(\()p Fp(count)h Fl(\000)g FA(1\))g(+)g(\()p Fp(blocklength)e Fl(\000)j FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fl(g)75 1462 y Ft(Indexed)47 b FA(The)19 b(function)h Ft(MPI)p 629 1462 V 16 w(TYPE)p 762 1462 V 17 w(INDEXED)f FA(allo)o(ws)h(replication)h (of)d(an)i(old)f(datat)o(yp)q(e)g(in)o(to)g(a)75 1518 y(sequence)c(of)e(con)o (tiguous)h(blo)q(c)o(ks,)g(where)g(eac)o(h)g(blo)q(c)o(k)h(can)f(con)o(tain)g (di\013eren)o(t)g(n)o(um)o(b)q(er)g(of)f(copies)i(and)75 1575 y(b)q(e)21 b(ha)o(v)o(e)f(a)g(di\013eren)o(t)h(displacemen)o(t.)37 b(All)21 b(blo)q(c)o(k)g(displacemen)o(ts)h(are)e(m)o(ultiples)j(of)c(the)i (old)g(t)o(yp)q(e)75 1631 y(exten)o(t.)75 1735 y Ft(MPI)p 160 1735 V 16 w(TYPE)p 293 1735 V 17 w(INDEXED\()12 b(count,)i(oldt)o(yp)q(e,)g (a)o(rra)o(y)p 926 1735 V 14 w(of)p 977 1735 V 16 w(displacements,)f(a)o(rra) o(y)p 1376 1735 V 14 w(of)p 1427 1735 V 16 w(blo)q(cklengths,)j(newt)o(yp)q (e\))117 1869 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)9 b(of)f(blo)q(c)o(ks)i({)f(also)f(n)o(um)o(b)q(er)h(of)g(en)o(tries)h(in)f Fm(a)o(rra)o(y)p 1773 1869 13 2 v 15 w(of)p 1822 1869 V 14 w(displacements)905 1925 y Fs(and)14 b Fm(a)o(rra)o(y)p 1073 1925 V 15 w(of)p 1122 1925 V 14 w(blo)q(cklengths)i Fs(\(in)o(teger\))117 2000 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i (\(handle\))117 2075 y(IN)155 b Ft(a)o(rra)o(y)p 416 2075 14 2 v 15 w(of)p 468 2075 V 16 w(displacements)162 b Fs(displacemen)o(t)19 b(for)h(eac)o(h)g(blo)q(c)o(k,)g(in)g(m)o(ultiples)e(of)h Fm(oldt)o(yp)q(e) 905 2132 y Fs(exten)o(t)c(\(arra)o(y)f(of)f(in)o(teger\))117 2207 y(IN)155 b Ft(a)o(rra)o(y)p 416 2207 V 15 w(of)p 468 2207 V 16 w(blo)q(cklengths)191 b Fs(n)o(um)o(b)q(er)18 b(of)f(elemen)o(ts)h(p)q (er)i(blo)q(c)o(k)d(\(arra)o(y)i(of)e(nonnegativ)o(e)905 2263 y(in)o(tegers\))117 2339 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))166 2463 y FA(Example:)29 b(let)20 b Ft(oldt)o(yp)q(e)h FA(ha)o(v)o(e)e(t)o(yp)q(e)g(map)h Fl(f)p FA(\()p Fp(double)n Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)p Fi(;)18 b FA(with)i(exten)o(t)f(16.)32 b(Let)20 b Fp(D)75 2519 y(=)k(\(4,)f(0\))17 b FA(and)g(let)h Fp(B)24 b(=)g(\(3,)f(1\))p FA(.)j(A)17 b(call)h(to)f Ft(MPI)p 998 2519 V 16 w(TYPE)p 1131 2519 V 17 w(INDEXED\(2,)f(oldt)o(yp)q(e,)j(D,)d (B,)i(newt)o(yp)q(e\))75 2576 y FA(returns)d(a)g(data)o(yp)q(e)g(with)g(t)o (yp)q(e)h(map)75 2632 y Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(64\))p Fi(;)g FA(\()p Fp(cha)o(r)m Fi(;)g FA(72\))p Fi(;)g FA(\()p Fp(double)m Fi(;)g FA(80\))p Fi(;)g FA(\()p Fp(cha)o(r)m Fi(;)g FA(88\))p Fi(;)g FA(\()p Fp(double)m Fi(;)g FA(96\))p Fi(;)g FA(\()p Fp(char)l Fi(;)g FA(104\))-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 33 35 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(33)75 49 y(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)15 b FA(:)j(three)f(copies)i (of)e(the)h(old)h(t)o(yp)q(e)f(starting)f(at)g(displacemen)o(t)i(64,)f(and)g (one)75 106 y(cop)o(y)d(starting)g(at)f(displacemen)o(t)j(0.)166 162 y(In)f(general,)f(assume)g(that)g Ft(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e) g(map)75 219 y Fl(f)p FA(\()p Fi(ty)r(pe)200 226 y Fh(0)218 219 y Fi(;)8 b(disp)323 226 y Fh(0)341 219 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 226 y Fg(n)p Fj(\000)p Fh(1)604 219 y Fi(;)g(disp)709 226 y Fg(n)p Fj(\000)p Fh(1)773 219 y FA(\))p Fl(g)p Fi(;)75 275 y FA(with)i(exten)o(t)g Fq(extent)p FA(.)18 b(Let)11 b Ft(B)f FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 802 275 14 2 v 15 w(of)p 854 275 V 15 w(blo)q(cklength)j FA(parameter)c(and)h Ft(D)g FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1661 275 V 15 w(of)p 1713 275 V 16 w(displacements)75 332 y FA(parameter.)19 b(The)d(newly)g (created)f(datat)o(yp)q(e)f(has)i Fi(n)10 b Fl(\001)1025 299 y Ff(P)1068 310 y Fe(count)s Fj(\000)p Fh(1)1068 343 y Fg(i)p Fh(=0)1198 332 y Ft(B)p FA([)p Ft(i)p FA(])k(en)o(tries:)75 388 y Fl(f)p FA(\()p Fi(ty)r(pe)200 395 y Fh(0)218 388 y Fi(;)8 b(disp)323 395 y Fh(0)351 388 y FA(+)i Ft(D)p FA([)p Ft(0)p FA(])f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)838 395 y Fg(n)p Fj(\000)p Fh(1)901 388 y Fi(;)g(disp)1006 395 y Fg(n)p Fj(\000)p Fh(1)1080 388 y FA(+)j Ft(D)p FA([)p Ft(0)p FA(])e Fl(\001)g Fi(extent)p FA(\))p Fi(;)f(:::;)75 444 y FA(\()p Fi(ty)r(pe)177 451 y Fh(0)196 444 y Fi(;)g(disp)301 451 y Fh(0)325 444 y FA(+)f(\()p Ft(D)p FA([)p Ft(0)p FA(])e(+)i Ft(B)p FA([)p Ft(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1037 451 y Fg(n)p Fj(\000)p Fh(1)1099 444 y Fi(;)g(disp)1204 451 y Fg(n)p Fj(\000)p Fh(1)1275 444 y FA(+)f(\()p Ft(D)p FA([)p Ft(0)p FA(])e(+)i Ft(B)p FA([)p Ft(0)p FA(])g Fl(\000)g FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)75 501 y FA(\()p Fi(ty)r(pe)177 508 y Fh(0)196 501 y Fi(;)g(disp)301 508 y Fh(0)328 501 y FA(+)j Ft(D)p FA([)p Ft(count)f Fl(\000)h Ft(1)p FA(])e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)977 508 y Fg(n)p Fj(\000)p Fh(1)1041 501 y Fi(;)g(disp)1146 508 y Fg(n)p Fj(\000)p Fh(1)1219 501 y FA(+)j Ft(D)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 557 y FA(\()p Fi(ty)r(pe)177 564 y Fh(0)196 557 y Fi(;)g(disp)301 564 y Fh(0)328 557 y FA(+)j(\()p Ft(D)p FA([)p Ft(count)f Fl(\000)h Ft(1)p FA(])e(+)h Ft(B)p FA([)p Ft(count)i Fl(\000)e Ft(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 614 y FA(\()p Fi(ty)r(pe)177 621 y Fg(n)p Fj(\000)p Fh(1)242 614 y Fi(;)g(disp)347 621 y Fg(n)p Fj(\000)p Fh(1)421 614 y FA(+)i(\()p Ft(D)p FA([)p Ft(count)h Fl(\000)f Ft(1)p FA(])g(+)g Ft(B)p FA([)p Ft(count)i Fl(\000)e Ft(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fl(g)p Fi(:)166 670 y FA(A)k(call)g(to)f Ft(MPI)p 431 670 V 16 w(TYPE)p 564 670 V 17 w(VECTOR\(count,)j(oldt)o(yp)q(e,)f(stride,)f(blo)q(cklength,)i(newt)o (yp)q(e\))f FA(is)f(equiv)m(alen)o(t)75 727 y(to)i(a)f(call)j(to)d Ft(MPI)p 392 727 V 16 w(TYPE)p 525 727 V 17 w(INDEX\(count,)i(oldt)o(yp)q(e,) g(D,)e(B,)h(newt)o(yp)q(e\))i FA(where)616 829 y Ft(D)p FA([)p Ft(j)p FA(])12 b(=)h Fi(j)g Fl(\001)c Ft(stride)16 b Fi(;)8 b(j)14 b FA(=)f(0)p Fi(;)8 b(:::;)g Fp(count)f Fl(\000)j FA(1)p Fi(;)75 931 y FA(and)588 987 y Ft(B)p FA([)p Ft(j)p FA(])i(=)h Ft(blo)q(cklength)k Fi(;)8 b(j)14 b FA(=)f(0)p Fi(;)8 b(:::;)g Fp(count)f Fl(\000)j FA(1)p Fi(:)75 1107 y Ft(Hindexed)47 b FA(The)11 b(function)h Ft(MPI)p 643 1107 V 15 w(TYPE)p 775 1107 V 17 w(HINDEXED)f FA(is)g(iden)o(tical)i(to)d Ft(MPI)p 1383 1107 V 15 w(TYPE)p 1515 1107 V 17 w(INDEXED)p FA(,)g(except)75 1164 y(that)19 b(blo)q(c)o(k)i(displacemen)o(ts)h(in)41 b Ft(a)o(rra)o(y)p 774 1164 V 14 w(of)p 825 1164 V 16 w(displacements)20 b FA(are)g(sp)q (eci\014ed)i(in)f(b)o(ytes,)g(rather)e(than)h(in)75 1220 y(m)o(ultiples)d(of) e(the)g Ft(oldt)o(yp)q(e)h FA(exten)o(t.)75 1324 y Ft(MPI)p 160 1324 V 16 w(TYPE)p 293 1324 V 17 w(HINDEXED\()10 b(count,)i(oldt)o(yp)q (e,)g(a)o(rra)o(y)p 952 1324 V 14 w(of)p 1003 1324 V 16 w(displacements,)f(a) o(rra)o(y)p 1400 1324 V 14 w(of)p 1451 1324 V 16 w(blo)q(cklength,)i(newt)o (yp)q(e\))117 1458 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)9 b(of)f(blo)q(c)o(ks)i({)f(also)f(n)o(um)o(b)q(er)h(of)g(en)o(tries)h(in)f Fm(a)o(rra)o(y)p 1773 1458 13 2 v 15 w(of)p 1822 1458 V 14 w(displacements)905 1514 y Fs(and)14 b Fm(a)o(rra)o(y)p 1073 1514 V 15 w(of)p 1122 1514 V 14 w(blo)q(cklengths)p Fs(\(in)o(teger\))117 1589 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i (\(handle\))117 1664 y(IN)155 b Ft(a)o(rra)o(y)p 416 1664 14 2 v 15 w(of)p 468 1664 V 16 w(displacements)162 b Fs(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o (teger\))117 1740 y(IN)155 b Ft(a)o(rra)o(y)p 416 1740 V 15 w(of)p 468 1740 V 16 w(blo)q(cklength)208 b Fs(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)f(of)g(nonnega-) 905 1796 y(tiv)o(e)f(in)o(tegers\))117 1871 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))166 1996 y FA(Assume)h(that)g Ft(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)75 2052 y Fl(f)p FA(\()p Fi(ty)r(pe)200 2059 y Fh(0)218 2052 y Fi(;)8 b(disp)323 2059 y Fh(0)341 2052 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 2059 y Fg(n)p Fj(\000)p Fh(1)604 2052 y Fi(;)g(disp)709 2059 y Fg(n)p Fj(\000)p Fh(1)773 2052 y FA(\))p Fl(g)p Fi(;)75 2109 y FA(with)i(exten)o(t)g Fq(extent)p FA(.)18 b(Let)11 b Ft(B)f FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 802 2109 V 15 w(of)p 854 2109 V 15 w(blo)q(cklength)j FA(parameter)c(and)h Ft(D)g FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1661 2109 V 15 w(of)p 1713 2109 V 16 w(displacements)75 2165 y FA(parameter.)42 b(The)23 b(newly)g(created)g(datat)o(yp)q(e)f(has)h (a)f(t)o(yp)q(e)h(map)f(with)i Fi(n)15 b Fl(\001)1470 2133 y Ff(P)1514 2143 y Fe(count)r Fj(\000)p Fh(1)1514 2176 y Fg(i)p Fh(=0)1643 2165 y Ft(B)p FA([)p Ft(i)p FA(])22 b(en)o(tries:)75 2221 y Fl(f)p FA(\()p Fi(ty)r(pe)200 2228 y Fh(0)218 2221 y Fi(;)8 b(disp)323 2228 y Fh(0)351 2221 y FA(+)i Ft(D)p FA([)p Ft(0)p FA(])o(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)678 2228 y Fg(n)p Fj(\000)p Fh(1)740 2221 y Fi(;)g(disp)845 2228 y Fg(n)p Fj(\000)p Fh(1)919 2221 y FA(+)i Ft(D)p FA([)p Ft(0)p FA(])o(\))p Fi(;)e(:::;)75 2278 y FA(\()p Fi(ty)r(pe)177 2285 y Fh(0)196 2278 y Fi(;)g(disp)301 2285 y Fh(0)325 2278 y FA(+)f Ft(D)p FA([)p Ft(0)p FA(])e(+)i(\()p Ft(B)p FA([)p Ft(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1037 2285 y Fg(n)p Fj(\000)p Fh(1)1099 2278 y Fi(;)g(disp)1204 2285 y Fg(n)p Fj(\000)p Fh(1)1275 2278 y FA(+)f Ft(D)p FA([)p Ft(0)p FA(])f(+)h(\()p Ft(B)p FA([)p Ft(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)75 2334 y FA(\()p Fi(ty)r(pe)177 2341 y Fh(0)196 2334 y Fi(;)g(disp)301 2341 y Fh(0)328 2334 y FA(+)j Ft(D)p FA([)p Ft(count)f Fl(\000)h Ft(1)p FA(])o(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)817 2341 y Fg(n)p Fj(\000)p Fh(1)879 2334 y Fi(;)g(disp)984 2341 y Fg(n)p Fj(\000)p Fh(1)1058 2334 y FA(+)j Ft(D)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])o(\))p Fi(;)e(:::;)75 2391 y FA(\()p Fi(ty)r(pe)177 2398 y Fh(0)196 2391 y Fi(;)g(disp)301 2398 y Fh(0)328 2391 y FA(+)j Ft(D)p FA([)p Ft(count)f Fl(\000)h Ft(1)p FA(])e(+)i(\()p Ft(B)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)75 2447 y FA(\()p Fi(ty)r(pe)177 2454 y Fg(n)p Fj(\000)p Fh(1)242 2447 y Fi(;)g(disp)347 2454 y Fg(n)p Fj(\000)p Fh(1)421 2447 y FA(+)i Ft(D)p FA([)p Ft(count)h Fl(\000)g Ft(1)p FA(])e(+)i(\()p Ft(B)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fl(g)p Fi(:)75 2567 y Ft(Struct)49 b Fp(MPI)p 315 2567 15 2 v 17 w(TYPE)p 428 2567 V 16 w(STRUCT)15 b FA(is)h(the)f(most)g(general)h(constructor:)j(It)c (further)h(generalizes)h(the)e(previ-)75 2624 y(ous)g(one)g(in)h(that)f(it)g (allo)o(ws)h(eac)o(h)f(blo)q(c)o(k)h(to)e(consist)i(of)f(replications)i(of)d (a)h(di\013eren)o(t)h(datat)o(yp)q(e.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 34 36 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(34)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(TYPE)p 293 49 V 17 w(STRUCT\(count,)12 b(a)o(rra)o(y)p 729 49 V 15 w(of)p 781 49 V 16 w(t)o(yp)q(es,)g(a)o(rra)o(y)p 1014 49 V 14 w(of)p 1065 49 V 16 w(displacements,)f(a)o(rra)o(y)p 1462 49 V 15 w(of)p 1514 49 V 16 w(blo)q(cklengths,)i(new-)75 106 y(t)o(yp)q(e\))117 183 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)18 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))g({)g(also)f(n)o(um)o(b)q(er)h(of)f(en)o (tries)905 239 y(in)c(arra)o(ys)h Fm(a)o(rra)o(y)p 1167 239 13 2 v 15 w(of)p 1216 239 V 14 w(t)o(yp)q(es)p Fs(,)h Fm(a)o(rra)o(y)p 1432 239 V 15 w(of)p 1481 239 V 14 w(displacements)e Fs(and)g Fm(a)o(r-)905 296 y(ra)o(y)p 959 296 V 15 w(of)p 1008 296 V 15 w(blo)q(cklengths)117 371 y Fs(IN)155 b Ft(a)o(rra)o(y)p 416 371 14 2 v 15 w(of)p 468 371 V 16 w(t)o(yp)q(es)327 b Fs(t)o(yp)q(e)20 b(of)f(elemen)o(ts)g(in)g(eac)o(h)h(blo)q(c)o(k)f(\(arra)o(y)g(of)g(handles)g (to)905 427 y(datat)o(yp)q(e)14 b(ob)r(jects\))117 503 y(IN)155 b Ft(a)o(rra)o(y)p 416 503 V 15 w(of)p 468 503 V 16 w(displacements)162 b Fs(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g (of)f(in)o(teger\))117 578 y(IN)155 b Ft(a)o(rra)o(y)p 416 578 V 15 w(of)p 468 578 V 16 w(blo)q(cklength)208 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o (teger\))117 653 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))166 777 y FA(Example:)20 b(Let)c Ft(t)o(yp)q(e)g FA(ha)o(v)o(e)f(t)o(yp)q(e)g(map)738 879 y Fl(f)p FA(\()p Fp(double)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(8\))p Fl(g)p Fi(;)75 982 y FA(with)24 b(exten)o(t)g(16.) 45 b(Let)24 b Fp(D)g(=)g(\(0,)f(16,)g(26\))p FA(,)j Fp(B)d(=)h(\(2,)g(1,)f (3\))h FA(and)g Fp(T)f(=)h(\(MPI)p 1551 982 15 2 v 17 w(FLOAT,)f(type1,)75 1038 y(MPI)p 150 1038 V 17 w(CHAR\))p FA(.)13 b(Then)i(a)f(call)h(to)f Ft(MPI)p 687 1038 14 2 v 16 w(TYPE)p 820 1038 V 17 w(STRUCT\(3,)g(T,)g(D,)g (B,)g(newt)o(yp)q(e\))i FA(returns)e(datat)o(yp)q(e)g(with)75 1094 y(t)o(yp)q(e)h(map.)75 1151 y Fl(f)p FA(\()p Fp(float)o Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fp(float)m Fi(;)g FA(4\))p Fi(;)g FA(\()p Fp(doub)o(le)m Fi(;)g FA(16\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(24\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(26\))p Fi(;)g FA(\()p Fp(ch)o(ar)m Fi(;)g FA(27\))p Fi(;)g FA(\()p Fp(char)m Fi(;)g FA(28\))p Fl(g)j FA(:)75 1207 y(t)o(w)o(o)f(copies)j (of)e Ft(MPI)p 421 1207 V 15 w(FLO)o(A)l(T)h FA(starting)f(at)g(0,)g(follo)o (w)o(ed)h(b)o(y)f(one)h(cop)o(y)f(of)g Ft(t)o(yp)q(e1)i FA(starting)e(at)f (16,)i(follo)o(w)o(ed)75 1264 y(b)o(y)i(three)g(copies)h(of)e Ft(MPI)p 520 1264 V 16 w(CHAR)p FA(,)i(starting)e(at)g(26.)19 b(\(W)l(e)14 b(assume)g(that)f(a)g(\015oat)h(o)q(ccupies)h(four)f(b)o (ytes.\))166 1320 y(In)i(general,)f(let)h Ft(T)f FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 742 1320 V 14 w(of)p 793 1320 V 16 w(t)o(yp)q(es)i FA(parameter,)d(where)31 b Ft(T[i])15 b FA(is)h(a)f(handle)h(to)75 1377 y Fi(ty)r(pe)159 1384 y Fg(i)186 1377 y FA(=)d Fl(f)p FA(\()p Fi(ty)r(pe)359 1360 y Fg(i)359 1388 y Fh(0)377 1377 y Fi(;)8 b(disp)482 1360 y Fg(i)482 1388 y Fh(0)500 1377 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)700 1360 y Fg(i)700 1388 y(n)721 1392 y Fd(i)733 1377 y Fi(;)h(disp)838 1360 y Fg(i)838 1388 y(n)859 1392 y Fd(i)873 1377 y FA(\))p Fl(g)p Fi(;)75 1433 y FA(with)i(exten)o(t)g Fi(extent)434 1440 y Fg(i)450 1433 y FA(.)18 b(Let)10 b Ft(B)g FA(b)q(e)h(the)g Ft(a)o(rra)o(y)p 826 1433 V 14 w(of)p 877 1433 V 16 w(blo)q(cklength)h FA(parameter)d(and)i Ft(D)f FA(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1685 1433 V 14 w(of)p 1736 1433 V 16 w(indices)75 1490 y FA(parameter.)19 b(Then)d(the)f(newly)h (created)f(datat)o(yp)q(e)g(has)g(a)g(t)o(yp)q(e)g(map)g(with)1413 1457 y Ff(P)1457 1468 y Fe(count)r Fj(\000)p Fh(1)1457 1501 y Fg(i)p Fh(=0)1586 1490 y Ft(B)p FA([)p Ft(i)p FA(])9 b Fl(\001)h Fi(n)1712 1497 y Fg(i)1741 1490 y FA(en)o(tries)75 1546 y Fl(f)p FA(\()p Fi(ty)r(pe)200 1530 y Fh(0)200 1557 y(0)218 1546 y Fi(;)e(disp)323 1530 y Fh(0)323 1557 y(0)351 1546 y FA(+)i Ft(D)p FA([)p Ft(0)p FA(])o(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)678 1530 y Fh(0)678 1557 y Fg(n)699 1561 y Fc(0)714 1546 y Fi(;)g(disp)819 1530 y Fh(0)819 1557 y Fg(n)840 1561 y Fc(0)866 1546 y FA(+)j Ft(D)p FA([)p Ft(0)p FA(])o(\))p Fi(;)d(:::;)75 1603 y FA(\()p Fi(ty)r(pe)177 1586 y Fh(0)177 1614 y(0)196 1603 y Fi(;)g(disp)301 1586 y Fh(0)301 1614 y(0)325 1603 y FA(+)f Ft(D)p FA([)p Ft(0)p FA(])g(+)g(\()p Ft(B)p FA([)p Ft(0)p FA(])g Fl(\000)g FA(1\))g Fl(\001)g Fi(extent)839 1610 y Fh(0)860 1603 y FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1061 1586 y Fh(0)1061 1614 y Fg(n)1081 1618 y Fc(0)1097 1603 y Fi(;)g(disp)1202 1586 y Fh(0)1202 1614 y Fg(n)1223 1618 y Fc(0)1247 1603 y FA(+)f Ft(D)p FA([)p Ft(0)p FA(])g(+)g(\()p Ft(B)p FA([)p Ft(0)p FA(])i Fl(\000)i Ft(1)p FA(\))c Fl(\001)g Fi(extent)1767 1610 y Fh(0)1787 1603 y FA(\))p Fi(;)h(:::;)75 1659 y FA(\()p Fi(ty)r(pe)177 1640 y Fe(count)r Fj(\000)p Fh(1)177 1669 y(0)299 1659 y Fi(;)g(disp)404 1640 y Fe(count)q Fj(\000)p Fh(1)404 1669 y(0)534 1659 y FA(+)j Ft(D)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])o(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1023 1642 y Fe(count)p Fj(\000)p Fh(1)1023 1670 y Fg(n)1044 1674 y Fb(count)m Fa(\000)p Fc(1)1142 1659 y Fi(;)g(disp)1247 1642 y Fe(count)q Fj(\000)p Fh(1)1247 1670 y Fg(n)1268 1674 y Fb(count)o Fa(\000)p Fc(1)1378 1659 y FA(+)i Ft(D)p FA([)p Ft(count)h Fl(\000)g Ft(1)p FA(])o(\))p Fi(;)d(:::;)75 1720 y FA(\()p Fi(ty)r(pe)177 1701 y Fe(count)r Fj(\000)p Fh(1)177 1730 y(0)299 1720 y Fi(;)g(disp)404 1701 y Fe(count)q Fj(\000)p Fh(1)404 1730 y(0)534 1720 y FA(+)j Ft(D)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])g(+)g(\()p Ft(B)p FA([)p Ft(count)h Fl(\000)g Ft(1)p FA(])e Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)1392 1727 y Fe(count)s Fj(\000)p Fh(1)1515 1720 y FA(\))p Fi(;)e(:::;)75 1776 y FA(\()p Fi(ty)r(pe)177 1760 y Fe(count)r Fj(\000)p Fh(1)177 1788 y Fg(n)198 1792 y Fb(count)p Fa(\000)p Fc(1)299 1776 y Fi(;)g(disp)404 1760 y Fe(count)q Fj(\000)p Fh(1)404 1788 y Fg(n)425 1792 y Fb(count)n Fa(\000)p Fc(1)534 1776 y FA(+)j Ft(D)p FA([)p Ft(count)g Fl(\000)f Ft(1)p FA(])g(+)g(\()p Ft(B)p FA([)p Ft(count)h Fl(\000)g Ft(1)p FA(])e Fl(\000)h Ft(1)p FA(\))g Fl(\001)g Fi(extent)1392 1783 y Fe(count)s Fj(\000)p Fh(1)1515 1776 y FA(\))p Fl(g)166 1833 y FA(A)15 b(call)h(to)f Ft(MPI)p 439 1833 V 16 w(TYPE)p 572 1833 V 16 w(HINDEXED\()g(count,)h(oldt)o (yp)q(e,)g(D,)f(B,)g(newt)o(yp)q(e\))h FA(is)g(equiv)m(alen)o(t)h(to)d(a)h (call)75 1889 y(to)d Ft(MPI)p 213 1889 V 16 w(TYPE)p 346 1889 V 17 w(STRUCT\()h(count,)i(T,)d(D,)h(B,)g(newt)o(yp)q(e\))p FA(,)h(where)f(eac)o(h)g(en)o(try)g(of)g Ft(T)g FA(is)g(equal)h(to)e Ft(oldt)o(yp)q(e)p FA(.)75 2011 y Fr(1.13.2)49 b(Address)16 b(and)g(extent)f(functions)75 2097 y FA(The)21 b(displacemen)o(ts)h(in)g(a)f (general)g(datat)o(yp)q(e)f(are)h(relativ)o(e)g(to)g(some)f(start)g(address.) 37 b Fw(Absolute)75 2153 y(addresses)13 b FA(can)h(b)q(e)h(substituted)g(for) e(these)i(displacemen)o(ts:)21 b(w)o(e)13 b(treat)h(them)g(as)g(displacemen)o (ts)h(rel-)75 2210 y(ativ)o(e)f(to)e(\\address)i(zero",)f(the)h(start)e(of)h (the)h(address)f(space.)20 b(This)14 b(initial)i(address)e(zero)f(is)h (indicated)75 2266 y(b)o(y)19 b(the)g(constan)o(t)f Fm(MPI)p 488 2266 13 2 v 15 w(BOTTOM)p FA(.)g(Th)o(us,)i(a)e(datat)o(yp)q(e)h(can)g (sp)q(ecify)i(the)e(absolute)g(address)g(of)g(the)75 2323 y(en)o(tries)e(in)g (the)f(comm)o(unication)h(bu\013er,)g(in)g(whic)o(h)g(case)f(the)h Ft(sta)o(rt)f FA(parameter)g(is)h(passed)f(the)h(v)m(alue)75 2379 y Fm(MPI)p 152 2379 V 14 w(BOTTOM)p FA(.)166 2435 y(The)10 b(address)g(of)g(a)g(lo)q(cation)h(in)g(memory)f(can)g(b)q(e)h(found)f(b)o(y) g(in)o(v)o(oking)h(the)f(function)h Ft(MPI)p 1701 2435 14 2 v 16 w(ADDRESS)p FA(.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 35 37 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(35)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(ADDRESS\(lo)q(cation,) 16 b(address\))117 126 y Fs(IN)155 b Ft(lo)q(cation)437 b Fs(lo)q(cation)13 b(in)h(caller)f(memory)f(\(c)o(hoice\))117 202 y(OUT)108 b Ft(address)449 b Fs(address)15 b(of)f(lo)q(cation)f(\(in)o(teger\))166 326 y FA(Returns)j(the)f(\(b)o(yte\))f(address)i(of)e Ft(lo)q(cation)p FA(.)166 383 y(Example:)147 489 y Fp(REAL)23 b(A\(100,100\))147 545 y(INTEGER)f(I1,)i(I2,)f(DIFF)147 602 y(CALL)g(MPI_ADDRESS\(A\(1,1\),)e (I1\))147 658 y(CALL)i(MPI_ADDRESS\(A\(10,10\),)e(I2\))147 715 y(DIFF)i(=)h(I2)f(-)h(I1)75 771 y(!)g(The)f(value)g(of)h(DIFF)f(is)h (1009*sizeofreal;)e(the)h(values)g(of)h(I1)f(and)h(I2)f(are)75 828 y(!)h(implementation)e(dependent.)166 934 y FA(Another)15 b(useful)i(auxiliary)f(function)h(is)f Ft(MPI)p 974 934 V 15 w(TYPE)p 1106 934 V 17 w(EXTENT)p FA(,)g(that)e(returns)i(the)f(exten)o(t)g (of)g(a)75 990 y(datat)o(yp)q(e)g({)g(where)g(exten)o(t)g(is)h(as)e (de\014ned)j(in)f(Eq.)f(1.1.)75 1094 y Ft(MPI)p 160 1094 V 16 w(TYPE)p 293 1094 V 17 w(EXTENT\(datat)o(yp)q(e,)h(extent\))117 1171 y Fs(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(\(handle\))117 1246 y(OUT)108 b Ft(extent)471 b Fs(datat)o(yp)q(e)14 b(exten)o(t)h(\(in)o(teger\))75 1436 y Fr(1.13.3)49 b(Lo)o(w)o(er-b)q(ound)17 b(and)g(upp)q(er-b)q(ound)g(ma)o(rk)n(ers)75 1522 y FA(It)e(is)g(often)g(con) o(v)o(enien)o(t)g(to)g(de\014ne)h(explicitly)h(the)e(lo)o(w)o(er)g(b)q(ound)h (and)f(upp)q(er)h(b)q(ound)f(of)g(a)g(t)o(yp)q(e)g(map,)75 1578 y(and)d(o)o(v)o(erwrite)e(the)i(de\014nition)h(giv)o(en)f(b)o(y)f (Equation)g(1.1.)18 b(This)12 b(allo)o(ws)f(to)g(de\014ne)i(a)e(datat)o(yp)q (e)f(that)h(has)75 1635 y(\\holes")k(at)f(its)h(b)q(eginning)i(or)d(its)h (end;)h(or)e(a)h(datat)o(yp)q(e)f(with)h(en)o(tries)g(that)f(extend)i(ab)q(o) o(v)o(e)e(the)h(upp)q(er)75 1691 y(b)q(ound)h(or)e(b)q(elo)o(w)h(the)g(lo)o (w)o(er)g(b)q(ound.)20 b(Examples)c(of)e(suc)o(h)h(a)f(usage)h(are)f(pro)o (vided)i(in)g(Section)f(1.13.6.)75 1748 y(T)l(o)g(ac)o(hiev)o(e)h(this,)g(w)o (e)f(add)g(t)o(w)o(o)f(additional)j(\\pseudo-datat)o(yp)q(es")e Ft(MPI)p 1349 1748 V 16 w(LB)p FA(,)g(and)g Ft(MPI)p 1618 1748 V 16 w(UB)h FA(that)f(can)75 1804 y(b)q(e)j(used,)g(resp)q(ectiv)o(ely)l(,)h (to)e(mark)f(the)i(lo)o(w)o(er)f(b)q(ound)h(or)f(the)g(upp)q(er)h(b)q(ound)g (of)f(a)g(datat)o(yp)q(e.)26 b(These)75 1861 y(pseudo-datat)o(yp)q(es)17 b(o)q(ccup)o(y)h(no)f(space)g(\()p Fi(extent)p FA(\()p Ft(MP)p 1015 1861 V 17 w(LB)p FA(\))f(=)g Fi(extent)p FA(\()p Ft(MP)p 1386 1861 V 17 w(UB)q FA(\))f(=)h(0\).)25 b(They)17 b(do)g(not)75 1917 y(a\013ect)i(the)g(con)o(ten)o(t)g(of)g(a)g(message)f(created)i(with)f (this)h(datat)o(yp)q(e.)32 b(Ho)o(w)o(ev)o(er,)19 b(they)g(do)g(a\013ect)g (the)75 1973 y(de\014nition)f(of)d(the)h(exten)o(t)g(of)f(a)h(datat)o(yp)q(e) f(and,)h(therefore,)f(a\013ect)g(the)h(outcome)g(of)f(a)h(replication)h(of)75 2030 y(this)f(datat)o(yp)q(e)e(b)o(y)h(a)g(datat)o(yp)q(e)g(constructor.)166 2086 y(Example:)30 b(Let)21 b Fp(D)j(=)f(\(-3,)h(0,)f(6\))p FA(;)f Fp(T)i(=)g(\(MPI)p 1033 2086 15 2 v 16 w(LB,)g(MPI)p 1217 2086 V 17 w(INT,)f(MPI)p 1425 2086 V 17 w(UB\))p FA(,)c(and)h Fp(B)k(=)g(\(1,)f(1,)75 2143 y(1\))p FA(.)c(Then)13 b(a)f(call)i(to)e Ft(MPI)p 525 2143 14 2 v 16 w(TYPE)p 658 2143 V 17 w(STRUC\(3,)h(T,)f(D,)g (B,)h(t)o(yp)q(e1\))g FA(creates)g(a)f(new)h(datat)o(yp)q(e)f(that)g(has)h (an)75 2199 y(exten)o(t)i(of)g(9)g(\(from)f(-3)i(to)e(5\),)h(and)g(con)o (tains)h(an)f(in)o(teger)h(at)e(displacemen)o(t)j(0.)j(This)c(is)g(the)f (datat)o(yp)q(e)75 2256 y(de\014ned)f(b)o(y)e(the)g(sequence)i Fl(f)p Fp(\(lb,)23 b(-3\),)g(\(int,)g(0\),)h(\(ub,)f(6\))p Fl(g)g FA(.)c(If)13 b(this)g(t)o(yp)q(e)f(is)h(replicated)h(t)o(wice)75 2312 y(b)o(y)h(a)g(call)h(to)f Ft(MPI)p 400 2312 V 15 w(TYPE)p 532 2312 V 17 w(CONTIGUOUS\(2,)h(t)o(yp)q(e1,)g(t)o(yp)q(e2\))f FA(then)h(the)f(newly)h(created)f(t)o(yp)q(e)g(can)g(b)q(e)75 2369 y(describ)q(ed)e(b)o(y)d(the)h(sequence)h Fl(f)p Fp(\(lb,)23 b(-3\),)g(\(int,)g(0\),)h(\(int,9\),)e(\(ub,)i(15\))p Fl(g)f FA(.)18 b(\(En)o(tries)11 b(of)f(t)o(yp)q(e)75 2425 y Fp(lb)15 b FA(or)g Fp(ub)f FA(can)i(b)q(e)g(deleted)g(if)g(they)f(are)g(not)g(at)f (the)i(endp)q(oin)o(ts)g(of)f(the)g(datat)o(yp)q(e.\))166 2482 y(In)h(general,)f(if)476 2584 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 2591 y Fh(0)867 2584 y Fi(;)8 b(disp)972 2591 y Fh(0)990 2584 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 2591 y Fg(n)p Fj(\000)p Fh(1)1253 2584 y Fi(;)h(disp)1358 2591 y Fg(n)p Fj(\000)p Fh(1)1421 2584 y FA(\))p Fl(g)p Fi(;)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 36 38 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(36)75 49 y(then)16 b(the)f Fw(lo)o(w)o(er)h(b)q(ound)g FA(of)f Fi(T)6 b(y)r(pemap)15 b FA(is)g(de\014ned)i(to)e(b)q(e)213 163 y Fi(l)q(b)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))11 b(=)531 91 y Ff(\()585 134 y FA(min)661 141 y Fg(j)686 134 y Fi(disp)770 141 y Fg(j)1142 134 y FA(if)16 b(no)f(en)o(try)g(has)g(basic)h(t)o(yp)q(e)f Fp(lb)585 191 y FA(min)q Fl(f)p Fi(disp)768 198 y Fg(j)812 191 y FA(:)28 b Fi(ty)r(pe)937 198 y Fg(j)967 191 y FA(=)13 b Fp(lb)p Fl(g)41 b FA(otherwise)166 276 y(Similarly)l(,)17 b(the)e Fw(upp)q(er)j(b)q(ound)e FA(of)e Fi(T)6 b(y)r(pemap)15 b FA(is)h(de\014ned)h(to)d(b)q(e)191 390 y Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))11 b(=)520 318 y Ff(\()574 361 y FA(max)659 368 y Fg(j)684 361 y Fi(disp)768 368 y Fg(j)795 361 y FA(+)f Fi(siz)r(eof)5 b FA(\()p Fi(ty)r(pe)1072 368 y Fg(j)1090 361 y FA(\))57 b(if)15 b(no)h(en)o(try)e(has)h(basic)h(t)o(yp)q(e)g Fp(ub)574 418 y FA(max)p Fl(f)p Fi(disp)766 425 y Fg(j)810 418 y FA(:)28 b Fi(ty)r(pe)935 425 y Fg(j)965 418 y FA(=)13 b Fp(ub)p Fl(g)66 b FA(otherwise)166 503 y(Then)441 560 y Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))14 b(=)e Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))j Fl(\000)i Fi(l)q(b)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))i(+)i Fi(\017)75 634 y FA(If)17 b Fi(ty)r(pe)206 641 y Fg(i)237 634 y FA(requires)g(alignmen)o(t)g(to)f(a)h(b)o(yte)f(address) h(that)f(is)h(is)g(a)f(m)o(ultiple)j(of)d Fi(k)1474 641 y Fg(i)1487 634 y FA(,)h(then)g Fi(\017)g FA(is)g(the)g(least)75 690 y(nonnegativ)o(e)f (incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))16 b(to)e(the)h(next)h(m)o(ultiple)h(of)d(max)1794 697 y Fg(i)1815 690 y Fi(k)1839 697 y Fg(i)1853 690 y FA(.)166 747 y(The)g(formal)f(de\014nitions)j(giv)o(en)e(for)f(the)h(v)m(arious)g (datat)o(yp)q(e)f(constructors)g(apply)i(no)o(w,)e(with)h(the)75 803 y(amended)i(de\014nition)h(of)e Fw(exten)o(t)p FA(.)75 922 y Fr(1.13.4)49 b(Comm)o(it)13 b(and)k(free)1875 980 y Fo(C)n(H)-6 b(A)h(N)f(G)h(E)75 1008 y FA(A)17 b(datat)o(yp)q(e)f(ob)s(ject)h(has)g(to)f (b)q(e)h Fw(committed)h FA(b)q(efore)f(it)g(can)g(b)q(e)h(used)g(in)f(a)g (comm)o(unication.)26 b(The)75 1064 y(system)20 b(ma)o(y)g(\\compile")i(at)e (commit)g(time)h(an)g(in)o(ternal)g(represen)o(tation)g(for)f(the)h(datat)o (yp)q(e)f(that)75 1121 y(facilitates)c(comm)o(unication,)g(e.g.)k(c)o(hange) 15 b(from)g(a)g(compacted)g(represen)o(tation)h(to)e(a)h(\015at)g(represen-) 75 1177 y(tation)j(of)g(the)g(datat)o(yp)q(e,)g(and)g(select)h(the)f(most)g (con)o(v)o(enien)o(t)g(transfer)g(mec)o(hanism.)29 b(A)19 b(commited)75 1233 y(datat)o(yp)q(e)c(can)g(still)i(b)q(e)e(used)h(as)f(a)g(parameter)f(in) i(datat)o(yp)q(e)f(constructors.)166 1290 y(Note:)28 b(the)20 b(commit)f(op)q(eration)h(commits)f(the)h(datat)o(yp)q(e,)f(i.e.,)i(the)e (formal)g(description)i(of)e(a)75 1346 y(comm)o(unication)i(bu\013er,)h(not)e (the)h(con)o(ten)o(t)f(of)g(that)g(bu\013er.)36 b(Th)o(us,)22 b(after)e(a)g(datat)o(yp)q(e)g(has)h(b)q(een)75 1403 y(committed,)d(it)h(can) f(b)q(e)h(rep)q(eatedly)g(reused)g(to)e(comm)o(unicate)h(the)h(c)o(hanging)f (con)o(ten)o(t)g(of)f(a)h(bu\013er)75 1459 y(or,)c(indeed,)j(the)e(con)o(ten) o(t)g(of)g(di\013eren)o(t)g(bu\013ers,)g(with)g(di\013eren)o(t)h(starting)e (addresses.)75 1563 y Ft(MPI)p 160 1563 14 2 v 16 w(TYPE)p 293 1563 V 17 w(COMMIT\(datat)o(yp)q(e\))117 1640 y Fs(INOUT)62 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(that)g(is)g(committed)e (\(handle\))166 1765 y FA(There)j(is)h(no)f(need)h(to)f(commit)g(basic)h (datat)o(yp)q(es;)e(they)h(are)g(\\pre-committed".)75 1868 y Ft(MPI)p 160 1868 V 16 w(TYPE)p 293 1868 V 17 w(FREE\(datat)o(yp)q(e\))117 1946 y Fs(INOUT)62 b Ft(datat)o(yp)q(e)424 b Fs(handle)14 b(to)g(datat)o(yp)q (e)g(ob)r(ject)166 2070 y FA(Marks)i(the)h(datat)o(yp)q(e)f(ob)s(ject)g(asso) q(ciated)h(with)g Ft(datat)o(yp)q(e)i FA(for)d(deallo)q(cation.)26 b(The)17 b(ob)s(ject)f(will)75 2126 y(b)q(e)g(deallo)q(cated)h(after)d(an)o (y)i(p)q(ending)h(comm)o(unication)f(that)e(uses)i(this)g(ob)s(ject)f (completes,)g(at)g(whic)o(h)75 2183 y(p)q(oin)o(t)21 b Ft(datat)o(yp)q(e)g FA(b)q(ecomes)g(n)o(ull.)36 b(It)20 b(is)h(erroneous)f(to)f(use)i(a)f(datat)o (yp)q(e)f(as)h(argumen)o(t)f(to)h(an)g(MPI)75 2239 y(function)c(after)e(it)i (w)o(as)e(freed.)166 2372 y Fn(Implemen)o(tati)o(on)e(note:)166 2422 y Fs(The)e(implemen)o(tati)o(on)d(ma)o(y)g(k)o(eep)j(a)f(reference)j (coun)o(t)e(of)f(activ)o(e)g(comm)o(unications)d(that)k(use)g(the)g(datat)o (yp)q(e,)75 2471 y(in)15 b(order)i(to)f(decide)h(when)f(to)g(free)h(it.)24 b(Also,)15 b(one)h(ma)o(y)e(implemen)o(t)g(constructors)k(of)d(deriv)o(ed)h (datat)o(yp)q(es)h(so)75 2521 y(that)e(they)h(k)o(eep)g(p)q(oin)o(ters)g(to)f (their)h(datat)o(yp)q(e)f(argumen)o(ts,)f(rather)j(then)e(cop)o(ying)g(them.) 21 b(In)16 b(suc)o(h)g(case,)g(one)75 2571 y(needs)e(to)e(k)o(eep)h(trac)o(k) f(of)g(activ)o(e)g(datat)o(yp)q(e)g(de\014nition)g(references)j(in)d(order)h (to)f(kno)o(w)g(when)h(a)f(datat)o(yp)q(e)g(ob)r(ject)75 2621 y(can)i(b)q(e)h(freed.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 37 39 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(37)75 49 y Fr(1.13.5)49 b(Use)16 b(of)h(general)f(datat)o(yp)q (es)g(in)h(comm)n(unication)75 135 y FA(Handles)g(to)f(deriv)o(ed)h(datat)o (yp)q(es)f(can)h(b)q(e)f(passed)h(to)f(a)g(comm)o(unication)h(call)g(wherev)o (er)f(a)g(datat)o(yp)q(e)75 192 y(argumen)o(t)22 b(is)h(required.)43 b(A)23 b(call)h(of)e(the)h(form)f Ft(MPI)p 1050 192 14 2 v 15 w(SEND\(sta)o(rt,)j(count,)g(datat)o(yp)q(e,)h(...\))p FA(,)d(where)75 248 y Fp(count)12 b FA(=)h Fp(1)h FA(uses)h(the)f(send)h(bu\013er)f (de\014ned)i(b)o(y)e Ft(sta)o(rt)g FA(and)h Ft(datat)o(yp)q(e)p FA(;)h(i.e.)k(if)29 b Ft(datat)o(yp)q(e)15 b FA(is)g(a)f(handle)i(to)75 304 y(the)i(datat)o(yp)q(e)f(with)h(t)o(yp)q(e)g(map)g Fl(f)p FA(\()p Fi(ty)r(pe)789 311 y Fh(0)807 304 y Fi(;)8 b(disp)912 311 y Fh(0)930 304 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)1130 311 y Fg(n)p Fj(\000)p Fh(1)1193 304 y Fi(;)g(disp)1298 311 y Fg(n)p Fj(\000)p Fh(1)1362 304 y FA(\))p Fl(g)p FA(,)17 b(\(with)h(size)h (zero)f(en)o(tries)75 361 y(deleted\))h(then)g(the)g(send)g(bu\013er)f (consists)h(of)f(the)g(en)o(tries)h(of)f(t)o(yp)q(es)h Fi(ty)r(pe)1400 368 y Fg(i)1414 361 y FA(,)g(at)e(addresses)i Fp(start)12 b FA(+)75 417 y Fi(disp)159 424 y Fg(i)172 417 y FA(,)17 b Fi(i)e FA(=)g(0)p Fi(;)8 b(:::;)g(n)g Fl(\000)k FA(1.)24 b(The)17 b(message)f(generated)g(consists)h(of)f(a)h(sequence)h(of)e Fi(n)h FA(en)o(tries,)g(of)f(t)o(yp)q(es)75 474 y Fi(ty)r(pe)159 481 y Fh(0)178 474 y Fi(;)8 b(:::;)g(ty)r(pe)343 481 y Fg(n)p Fj(\000)p Fh(1)406 474 y FA(.)30 b(The)19 b(receiv)o(e)h(bu\013er)f(for)f(a)g (call)i Ft(MPI)p 1119 474 V 16 w(RECV\()f(sta)o(rt,)h(count,)g(datat)o(yp)q (e)g(,)g(...\))p FA(,)d(is)75 530 y(similarly)g(de\014ned,)g(for)e Fp(count)24 b(=1)p FA(.)d(The)15 b(same)h(applies)h(to)e(all)i(other)e(send)h (and)g(receiv)o(e)h(op)q(erations)75 587 y(that)d(ha)o(v)o(e)h(a)g Ft(datat)o(yp)q(e)i FA(parameter.)166 643 y(A)10 b(call)h(of)f(the)g(form)g Ft(MPI)p 596 643 V 16 w(SEND\(sta)o(rt,)h(count,)h(datat)o(yp)q(e)g(,)e (...\))p FA(,)g(where)g Fp(count)i Fi(>)h Fp(1)p FA(,)e(is)f(in)o(terpreted) 75 700 y(as)21 b(if)h(the)f(call)i(w)o(as)d(passed)i(a)f(new)h(datat)o(yp)q (e)e(whic)o(h)j(is)f(the)f(concatenation)h(of)e Ft(count)j FA(copies)f(of)75 756 y Ft(datat)o(yp)q(e)p FA(.)f(The)14 b(same)g(holds)g (true)g(for)f(the)h(other)g(comm)o(unication)h(functions)f(that)f(ha)o(v)o(e) h(a)f(datat)o(yp)q(e)75 813 y(argumen)o(t.)19 b(Th)o(us,)166 869 y Ft(MPI)p 251 869 V 16 w(SEND\(sta)o(rt,)c(count,)h(datat)o(yp)q(e,)g (dest,)h(tag,)e(comm)m(\))d FA(is)j(equiv)m(alen)o(t)i(to)75 975 y Fp(MPI_TYPE_CONTIGUOUS\(count,)j(datatype,)j(newtype\))75 1032 y(MPI_TYPE_COMMIT\(newtype\))75 1088 y(MPI_SEND\(start,)f(1,)h(newtype,) g(dest,)g(tag,)h(comm\))166 1194 y FA(If)e(a)f(comm)o(unication)i(op)q (eration)e(uses)h(as)g(argumen)o(t)f(a)g(general)h(datat)o(yp)q(e)f Ft(t)o(yp)q(e)i FA(then)f(eac)o(h)75 1251 y(en)o(try)13 b(in)g(the)g(t)o(yp)q (e)g(map)g(should)h(matc)o(h)e(the)h(datat)o(yp)q(e)f(of)h(the)g(corresp)q (onding)h(v)m(ariable)g(in)g(the)f(caller)75 1307 y(memory)g(\(see)g(Section) i(1.5\).)j(If)13 b(a)h(send)g(op)q(eration)f(using)h(a)g(general)g(datat)o (yp)q(e)e Ft(t)o(yp)q(e1)j FA(is)f(matc)o(hed)f(b)o(y)75 1364 y(a)j(receiv)o(e)i(op)q(eration)f(using)g(a)f(general)h(datat)o(yp)q(e)f Ft(t)o(yp)q(e2)i FA(then)f(the)f(sequence)i(of)e(\(nonempt)o(y\))g(basic)75 1420 y(t)o(yp)q(e)e(en)o(tries)g(in)g(eac)o(h)g(t)o(yp)q(e)f(map)h(should)g (b)q(e)g(iden)o(tical.)22 b(Ho)o(w)o(ev)o(er,)12 b(the)i(sequences)g(of)g (displacemen)o(ts)75 1477 y(ma)o(y)i(di\013er.)24 b(Th)o(us,)16 b(t)o(yp)q(e)g(matc)o(hing)h(is)g(de\014ned)h(b)o(y)e(the)g(t)o(yp)q(e)h (map,)f(and)h(do)q(es)f(not)g(dep)q(end)j(on)d(the)75 1533 y(construction)f(of)g(the)h(datat)o(yp)q(e.)166 1590 y(Example:)75 1683 y Fp(...)75 1740 y(MPI_TYPE_CONTIGUOUS\()21 b(MPI_REAL,)i(2,)g(type2\)) 75 1796 y(MPI_TYPE_CONTIGUOUS\()e(MPI_REAL,)i(4,)g(type4\))75 1853 y(MPI_TYPE_CONTIGUOUS\()e(type2,)i(2,)h(type22\))166 1947 y FA(A)15 b(message)g(sen)o(t)g(with)g(datat)o(yp)q(e)g Ft(t)o(yp)q(e4)h FA(can)f(b)q(e)h(receiv)o(ed)h(with)e(datat)o(yp)q(e)30 b Ft(t)o(yp)q(e22)p FA(.)166 2003 y(A)18 b(datat)o(yp)q(e)g(used)h(in)g(a)g(receiv)o(e)g(op)q (eration)g(ma)o(y)e(not)h(ha)o(v)o(e)g(o)o(v)o(erlapping)h(en)o(tries)g(in)g (its)g(t)o(yp)q(e)75 2060 y(map:)h(the)15 b(same)g(lo)q(cation)h(in)g(the)f (receiv)o(e)h(bu\013er)f(cannot)g(b)q(e)h(written)g(t)o(wice)f(during)h(a)f (receiv)o(e.)166 2116 y(T)o(yp)q(e)23 b(matc)o(hing)g(requiremen)o(ts)h (imply)g(that)f(\(nonempt)o(y\))f(en)o(tries)i(in)f(the)h(t)o(yp)q(e)f(map)g (of)f(a)75 2172 y(datat)o(yp)q(e)f(cannot)g(o)o(v)o(erlap,)h(unless)h(they)e (coincide)j(and)d(ha)o(v)o(e)g(the)g(same)g(basic)h(t)o(yp)q(e,)h(or)e(one)g (of)75 2229 y(them)d(ha)o(v)o(e)f(basic)h(t)o(yp)q(e)g Ft(b)o(yte)p FA(:)25 b(otherwise,)18 b(when)g(the)g(datat)o(yp)q(e)f(is)h(used)g(in)h(a)e (comm)o(unication,)i(at)75 2285 y(least)e(on)g(of)g(the)g(t)o(w)o(o)f(o)o(v)o (erlaping)h(en)o(tries)h(will)h(not)d(b)q(e)i(t)o(yp)q(e-consisten)o(t)g (with)f(the)g(comm)o(unication)75 2342 y(bu\013er.)166 2398 y(The)h(v)m(alue)h(returned)g(in)f Ft(count)h FA(b)o(y)f(a)g(call)h(to)e Ft(MPI)p 1085 2398 V 16 w(GET)p 1189 2398 V 17 w(COUNT\(status,)k(count\))d FA(is)h(equal)g(to)75 2455 y(the)13 b(n)o(um)o(b)q(er)g(of)f(basic)h(elemen)o (ts)h(receiv)o(ed)g(b)o(y)e(the)h(op)q(eration)g(that)f(set)g Ft(status)p FA(;)j(this)e(is)h(the)e(n)o(um)o(b)q(er)h(of)75 2511 y(nonempt)o(y)f(en)o(tries)h(in)g(the)f(t)o(yp)q(e)h(map)f(used)h(to)e (create)h(the)h(message.)18 b(Consider)13 b(again)f(the)h(datat)o(yp)q(es)75 2568 y(de\014ned)k(in)f(the)f(last)h(example.)21 b(If)16 b(a)f(message)g(is)h (sen)o(t)f(using)h(either)g(datat)o(yp)q(e)f Ft(t)o(yp)q(e22)h FA(or)f Ft(t)o(yp)q(e4)h FA(and)75 2624 y(is)i(receiv)o(ed)g(using)g(either)f (of)g(these)g(t)o(w)o(o)f(datat)o(yp)q(es,)h(then)g(the)g(n)o(um)o(b)q(er)h (of)e(elemen)o(ts)i(receiv)o(ed)g(will)75 2681 y(b)q(e)e(four.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 38 40 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(38)166 49 y Fn(Discussion:)166 99 y Fs(The)17 b(curren)o(t)g(de\014nition)f(implies)e(that)j(a)f(receiv)o(e)h(cannot)f(c)o (hange)h(the)g(v)n(alue)e(of)h(storage)h(b)q(et)o(w)o(een)g(the)75 149 y(en)o(tries)k(de\014ned)f(to)g(comp)q(ose)f(the)h(comm)o(unication)c (bu\013er.)37 b(In)19 b(particular,)i(the)f(de\014nition)f(implies)f(that)75 199 y(padding)12 b(space)i(in)e(a)g(structure)j(should)e(not)f(b)q(e)i(mo)q (di\014ed)d(when)i(suc)o(h)h(structure)h(is)d(copied)h(from)e(one)i(pro)q (cess)75 249 y(to)19 b(another.)34 b(This)19 b(prev)o(en)o(ts)i(the)f(ob)o (vious)e(optimization)e(of)j(cop)o(ying)f(the)i(structure,)i(together)f(with) d(the)75 298 y(padding,)12 b(as)g(one)h(con)o(tiguous)f(blo)q(c)o(k)g(\(the)i (additional)c(comm)o(unication)g(o)o(v)o(erhead)i(of)g(sending)h(the)g (padding)f(is)75 348 y(lik)o(ely)e(to)h(b)q(e)h(smaller)d(then)j(the)g (computation)d(o)o(v)o(erhead)j(of)f(disassem)o(bling,)e(next)j(reassem)o (bling)e(the)i(structure)75 398 y(comp)q(onen)o(ts\).)19 b(Ho)o(w)o(ev)o(er,) 14 b(this)g(optimization)e(is)i(not)g(p)q(ossible)h(in)f(general,)g(since)h (it)f(is)g(not)g(kno)o(wn)g(when)g(the)75 448 y(send)h(is)g(executed)h(that)f (the)g(receiv)o(e)g(is)g(going)e(to)h(store)i(the)f(incoming)d(message)i(in)g (a)g(structure)j(of)d(the)h(same)75 498 y(t)o(yp)q(e;)e(also,)e(across)i (heterogeneous)i(mac)o(hines,)c(similarly)e(de\014ned)k(structures)i(ma)o(y) 10 b(ha)o(v)o(e)i(a)g(di\013eren)o(t)h(memory)75 547 y(la)o(y)o(out.)23 b(In)16 b(a)g(homogeneous)f(en)o(vironmen)o(t,)g(when)i(comm)o(unicatio)o(n)d (is)h(kno)o(wn)h(to)g(in)o(v)o(olv)o(e)f(structures)j(with)75 597 y(the)e(same)f(la)o(y)o(out,)g(one)h(can)g(a)o(v)o(oid)e(assem)o(bly)h (and)g(disassem)o(bly)g(b)o(y)g(sending)h(the)h(data)e(using)g(the)i(datat)o (yp)q(e)75 647 y Fm(MPI)p 152 647 13 2 v 14 w(BYTE)p Fs(.)c(This)h(problem)e (ma)o(y)g(b)q(e)j(addressed)h(b)o(y)d(future)i(extensions)g(of)e(MPI.)75 851 y Fr(1.13.6)49 b(Examples)75 937 y FA(The)15 b(follo)o(wing)h(examples)g (illustrate)h(the)e(use)g(of)g(deriv)o(ed)h(datat)o(yp)q(es.)75 1057 y Ft(First)f(example:)43 b FA(Send)16 b(and)g(receiv)o(e)g(a)e(section)i (of)f(a)g(3D)g(arra)o(y)l(.)218 1164 y Fp(REAL)23 b(a\(100,100,100\),)f (e\(9,9,9\))218 1220 y(INTEGER)h(oneslice,)g(twoslice,)f(threeslice,)h (sizeofreal,)f(myrank,)h(ierr)218 1276 y(MPI_STATUS)g(status)75 1389 y(C)143 b(extract)23 b(the)g(section)g(a\(1:17:2,)g(3:11,)g(2:10\))75 1446 y(C)143 b(and)23 b(store)h(it)f(in)h(e\(*,*,*\).)218 1559 y(CALL)f(MPI_COMM_RANK\(MPI_ALL,)e(myrank\))218 1672 y(CALL)i (MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 1785 y(C)119 b(create)23 b(datatype)g(for)g(a)h(1D)g(section)218 1841 y(CALL)f(MPI_TYPE_VECTOR\()f(9,)i(MPI_REAL,)46 b(2,)24 b(1,)f(oneslice,)g(ierr\))75 1954 y(C)119 b(create)23 b(datatype)g(for)g(a)h (2D)g(section)218 2010 y(CALL)f(MPI_TYPE_HVECTOR\()f(9,)i(oneslice,)46 b(100*sizeofreal,)22 b(1,)i(twoslice,)e(ierr\))75 2123 y(C)119 b(create)23 b(datatype)g(for)g(the)h(entire)f(section)218 2180 y(CALL)g(MPI_TYPE_HVECTOR\()f(9,)i(twoslice,)46 b(100*100*sizeofreal,)21 b(1,)767 2236 y(threeslice,)h(ierr\))218 2349 y(CALL)h(MPI_TYPE_COMMIT\()f (threeslice,)h(ierr\))218 2406 y(CALL)g(MPI_SENDRECV\(a\(1,3,2\),)e(1,)j (threeslice,)e(myrank,)h(0,)h(e,)f(9*9*9,)648 2462 y(MPI_REAL,)f(myrank,)h (0,)h(MPI_ALL,)f(status,)g(ierr\))75 2582 y Ft(Second)17 b(example:)42 b FA(Cop)o(y)15 b(the)g(\(strictly\))g(lo)o(w)o(er)g(triangular)g(part)g(of)g (a)g(matrix.)218 2688 y Fp(REAL)23 b(a\(100,100\),)g(b\(100,100\))-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 39 41 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(39)218 49 y Fp(INTEGER)47 b(disp\(100\),)22 b(blocklen\(100\),)g(ltype,)h(myrank,)g(ierr)218 106 y(MPI_STATUS)g(status)75 219 y(C)119 b(copy)23 b(lower)h(triangular)e(part)h(of)h(array)f(a)75 275 y(C)119 b(onto)23 b(lower)h(triangular)e(part)h(of)h(array)f(b)218 388 y(CALL)g(MPI_COMM_RANK\(MPI_ALL,)e(myrank\))75 501 y(C)119 b(compute)23 b(start)g(and)h(size)f(of)h(each)f(column)218 557 y(DO)h(i=1,)f(100)266 614 y(disp\(i\))g(=)h(100*\(i-1\))e(+)i(i)266 670 y(block\(i\))f(=)g(100-i)218 727 y(END)h(DO)75 840 y(C)119 b(create)23 b(datatype)g(for)g(lower)h(triangular)e(part)218 896 y(CALL)h(MPI_TYPE_INDEX\()f(100,)i(MPI_REAL,)e(disp,)h(block,)g(ltype,)h (ierr\))218 1009 y(CALL)f(MPI_TYPE_COMMIT\(ltype,)e(ierr\))218 1065 y(CALL)i(MPI_SENDRECV\()g(a,)g(1,)h(ltype,)f(myrank,)g(0,)g(b,)h(1,)552 1122 y(ltype,)f(myrank,)g(0,)h(MPI_ALL,)f(status,)f(ierr\))75 1242 y Ft(Third)15 b(example:)43 b FA(T)l(ransp)q(ose)15 b(a)g(matrix)218 1348 y Fp(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 1405 y(INTEGER)g(row,)g (xpose,)g(sizeofreal,)g(myrank,)g(ierr)218 1461 y(MPI_STATUS)g(status)75 1574 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 1687 y(CALL)g(MPI_COMM_RANK\(MPI_ALL,)e(myrank\))218 1800 y(CALL)i (MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 1913 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row)218 1969 y(CALL)f (MPI_TYPE_VECTOR\()f(100,)h(MPI_REAL,)g(100,)g(1,)h(row,)f(ierr\))75 2082 y(C)119 b(create)23 b(datatype)g(for)g(matrix)g(in)h(row-major)f(order) 218 2139 y(CALL)g(MPI_TYPE_HVECTOR\()f(100,)h(row,)h(sizeofreal,)e(1,)i (xpose,)f(ierr\))218 2252 y(CALL)g(MPI_TYPE_COMMIT\()f(xpose,)h(ierr\))75 2364 y(C)119 b(send)23 b(matrix)h(in)f(row-major)g(order)g(and)g(receive)g (in)h(column)f(major)g(order)218 2421 y(CALL)g(MPI_SENDRECV\()g(a,)g(1,)h (xpose,)f(myrank,)g(0,)g(b,)h(100*100,)457 2477 y(MPI_REAL,)e(myrank,)h(0,)h (MPI_ALL,)f(status,)g(ierr\))166 2584 y FA(Another)15 b(approac)o(h)g(to)f (the)i(transp)q(ose)f(problem:)218 2677 y Fp(REAL)23 b(a\(100,100\),)g (b\(100,100\))-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 40 42 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(40)218 49 y Fp(INTEGER)47 b(disp\(2\),)23 b(blocklen\(2\),)f (type\(2\),)h(row,)g(row1,)g(sizeofreal)218 106 y(INTEGER)47 b(myrank,)23 b(ierr)218 162 y(MPI_STATUS)g(status)218 275 y(CALL)g (MPI_COMM_RANK\(MPI_ALL,)e(myrank\))75 388 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 501 y(CALL)g(MPI_TYPE_EXTENT\()f(MPI_REAL,)h (sizeofreal,)f(ierr\))75 614 y(C)119 b(create)23 b(datatype)g(for)g(one)h (row)218 670 y(CALL)f(MPI_TYPE_VECTOR\()f(MPI_REAL,)h(100,)g(100,)g(1,)h (row,)f(ierr\))75 783 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row,)f (with)h(the)f(extent)g(of)h(one)f(real)g(number)218 840 y(disp\(1\))g(=)h(0) 218 896 y(disp\(2\))f(=)h(sizeofreal)218 953 y(type\(1\))47 b(=)24 b(row)218 1009 y(type\(2\))47 b(=)24 b(MPI_UB)218 1065 y(blocklen\(1\))46 b(=)24 b(1)218 1122 y(blocklen\(2\))46 b(=)24 b(1)218 1178 y(CALL)f(MPI_TYPE_STRUC\()f(2,)i(type,)f(disp,)g(blocklen,)g (row1\))218 1291 y(CALL)g(MPI_TYPE_COMMIT\()f(row1,)h(ierr\))75 1404 y(C)119 b(send)23 b(100)h(rows)f(and)h(receive)f(in)g(column)g(major)g (order)218 1461 y(CALL)g(MPI_SENDRECV\()g(a,)g(100,)g(row1,)h(myrank,)f(0,)g (b,)h(100*100,)457 1517 y(MPI_REAL,)e(myrank,)h(0,)h(MPI_ALL,)f(status,)g (ierr\))75 1637 y Ft(F)o(ourth)15 b(example:)43 b FA(manipulate)17 b(an)e(arra)o(y)f(of)g(structures.)75 1743 y Fp(struct)23 b(Partstruct)147 1800 y({)147 1856 y(int)95 b(class;)47 b(/*)23 b(particle)g(class)g(*/)147 1913 y(double)g(d[6];)71 b(/*)23 b(particle)g(coordinates)f(*/)147 1969 y(char)71 b(b[7];)g(/*)23 b(some)g(additional)g(information)f(*/)147 2026 y(};)75 2139 y(struct)h(Partstruct)94 b(particle[1000];)75 2252 y(int)429 b(i,)24 b(dest,)f(rank;)75 2308 y(MPI_communicator)117 b(comm;)75 2477 y(/*)24 b(build)f(datatype)g(describing)f(structure)h(*/)75 2590 y(MPI_datatype)f(Particletype;)75 2647 y(MPI_datatype)g(type[3])h(=)h ({MPI_int,)e(MPI_double,)h(MPI_char};)75 2703 y(int)238 b(blocklen[3])23 b(=)g({1,)h(6,)f(7};)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 41 43 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(41)75 49 y Fp(int)238 b(disp[3];)75 219 y(/*)24 b(compute)e(displacements)h(of)g(structure)g(components)f(*/)75 332 y(MPI_address\()g(\(void)h(*\)particle,)g(disp++\);)75 388 y(MPI_address\()f(\(void)h(*\)particle[0].d,)f(disp++\);)75 444 y(MPI_address\()g(\(void)h(*\)particle[0].b,)f(disp\);)75 501 y(for)h(\(i=0;)h(i)f(<2;)h(i++\))f(disp[i])g(-=)h(disp[0];)75 614 y(MPI_type_struct\()e(3,)h(type,)g(disp,)h(blocklen,)e(Particletype\);) 147 727 y(/*)h(If)h(compiler)f(does)g(padding)g(in)g(mysterious)g(ways,)147 783 y(the)g(following)g(may)g(be)h(safer)f(*/)75 896 y(MPI_datatype)f (type1[4])h(=)h({MPI_int,)e(MPI_double,)h(MPI_char,)f(MPI_UB};)75 953 y(int)238 b(blocklen1[4])22 b(=)i({1,)g(6,)f(7,)h(1};)75 1009 y(int)238 b(disp1[4];)75 1122 y(/*)24 b(compute)e(displacements)h(of)g (structure)g(components)f(*/)75 1235 y(MPI_address\()g(\(void)h(*\)particle,) g(disp1++\);)75 1291 y(MPI_address\()f(\(void)h(*\)particle[0].d,)f (disp1++\);)75 1348 y(MPI_address\()g(\(void)h(*\)particle[0].b,)f (disp1++\);)75 1404 y(MPI_address\()g(\(void)h(*\)\(particle+1\),)f(disp1\);) 75 1461 y(for)h(\(i=0;)h(i)f(<3;)h(i++\))f(disp1[i])g(-=)g(disp1[0];)75 1574 y(/*)h(build)f(datatype)g(describing)f(structure)h(*/)75 1686 y(MPI_type_struct\()f(4,)h(type1,)g(disp1,)g(blocklen1,)g (Particletype\);)409 1856 y(/*)h(4.1:)266 1912 y(send)f(the)h(entire)f(array) g(*/)75 2025 y(MPI_commit\()f(Particletype\);)75 2082 y(MPI_send\()h (particle,)f(1000,)h(Particletype,)g(dest,)g(tag,)g(comm\);)409 2251 y(/*)h(4.2:)266 2307 y(send)f(only)h(the)f(entries)g(of)g(class)h(zero)f (particles,)266 2364 y(preceded)g(by)g(the)h(number)f(of)g(such)h(entries)f (*/)75 2477 y(MPI_datatype)f(Zparticles;)70 b(/*)24 b(datatype)f(describing)f (all)i(particles)791 2533 y(with)f(index)g(zero)h(\(needs)f(to)g(be)h (recomputed)791 2590 y(if)f(classes)g(change\))g(*/)75 2646 y(MPI_datatype)f(Ztype;)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 42 44 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(42)75 49 y Fp(int)23 b(zdisp[1000],)g(zblock[1000],)f(j,)h(k;) 75 106 y(int)g(zzblock[2])g(=)h({1,1};)75 162 y(int)f(zztype[2],)g (zzindex[2];)75 332 y(/*)h(compute)e(indices)h(of)h(class)f(zero)g(particles) g(*/)75 388 y(j)h(=)f(0;)75 444 y(for\(i=0;)g(i)g(<)h(1000;)f(i++\))123 501 y(if)g(\(particle[i].class==0\))194 557 y({)194 614 y(zindex[j])g(=)h(i;) 194 670 y(zblock[j])f(=)h(1;)194 727 y(j++;)194 783 y(})75 896 y(/*)g(create)f(datatype)f(for)i(class)f(zero)g(particles)47 b(*/)75 953 y(MPI_type_indexed\()22 b(j,)h(Particletype,)46 b(zindex,)23 b(zblock,)g(Zparticles\);)75 1065 y(/*)h(prepend)e(particle)h (count)g(*/)75 1122 y(MPI_address\(\(void)f(*\)&j,)h(zzindex\);)75 1178 y(zzindex[1])f(=)i(particle;)75 1235 y(zztype[0])f(=)g(MPI_int;)75 1291 y(zztype[1])g(=)g(Zparticles;)75 1348 y(MPI_type_struct\(2,)e(zztype,)i (zzindex,)g(zzblock,)g(Ztype\);)75 1461 y(MPI_type_commit\()f(Ztype\);)75 1517 y(MPI_send\()h(MPI_bottom,)f(1,)i(Ztype,)f(dest,)g(tag,)g(comm\);)242 1686 y(/*)h(A)f(probably)g(more)g(efficient)g(way)h(of)f(defining)g (Zparticles)f(*/)75 1799 y(/*)i(consecutive)e(particles)h(with)g(index)g (zero)g(are)h(handled)f(as)g(one)h(block)f(*/)75 1856 y(j=0;)75 1912 y(for)g(\(i=0;)h(i)f(<)h(1000;)f(i++\))123 1969 y(if)g (\(particle[i].index==0\))170 2025 y({)170 2082 y(for)h(\(k=i+1;)f(\(k)g(<)h (1000\)&&\(particle[k].index)d(==)i(0\))h(;)f(k++\);)170 2138 y(zindex[j])g(=)h(i;)170 2195 y(zblock[j])f(=)h(k-i;)170 2251 y(j++;)170 2307 y(i)g(=)g(k;)170 2364 y(})75 2420 y(MPI_type_indexed\()e(j,)h (Particletype,)f(zindex,)h(zblock,)g(Zparticles\);)457 2590 y(/*)g(4.3:)314 2646 y(send)g(the)g(first)h(two)f(coordinates)f(of)i(all)f (entries)g(*/)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 43 45 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(43)75 49 y Fp(MPI_datatype)22 b(Allpairs;)142 b(/*)24 b(datatype)f(for)g(all)g(pairs)h(of)f(coordinates)g(*/)75 162 y(int)238 b(sizeofentry;)75 275 y(MPI_extent\()22 b(Particletype,)g (sizeofentry\);)194 388 y(/*)i(sizeofentry)e(can)i(also)f(be)h(computed)e(by) i(subtracting)e(the)i(address)266 444 y(of)f(particle[0])g(from)g(the)h (address)f(of)g(particle[1])f(*/)75 557 y(MPI_type_hvector\()g(1000,)h (MPI_real,)f(sizeofentry,)h(2,)g(Allpairs\);)75 614 y(MPI_type_commit\()f (Allpairs\);)75 670 y(MPI_send\()h(particle.d,)f(1,)i(Allpairs,)e(dest,)h (tag,)h(comm\);)218 783 y(/*)g(an)f(alternative)g(solution)f(to)i(4.3)g(*/)75 896 y(MPI_datatype)e(Onepair;)71 b(/*)23 b(datatype)g(for)h(one)f(pair)g(of)h (coordinates,)e(with)695 953 y(the)i(extent)f(of)h(one)f(particle)g(entry)g (*/)75 1009 y(int)g(disp2[3];)75 1065 y(int)g(type2[3])g(=)h({MPI_lb,)f (MPI_double,)f(MPI_ub};)75 1122 y(int)h(blocklen2[3])g(=)g({1,)h(2,)f(1};)75 1235 y(MPI_address\()f(\(void)h(*\)particle,)g(disp2++\);)75 1291 y(MPI_address\()f(\(void)h(*\)particle[0].d,)f(disp2++\);)75 1348 y(MPI_address\()g(\(void)h(*\)\(particle+1\),)f(disp2\);)75 1404 y(for)h(\(i=0;)h(i<2;)f(i++\))g(disp2[i])g(-=)h(disp2[0];)75 1517 y(MPI_type_struct\()e(3,)h(type2,)g(disp2,)g(blocklen2,)g(Onepair\);)75 1574 y(MPI_commit\()f(Onepair\);)75 1630 y(MPI_send\()h(particle[0].d,)f (1000,)h(Onepair,)g(dest,)g(tag,)g(comm\);)75 1807 y Ft(Fifth)18 b(example:)43 b FA(same)17 b(arra)o(y)g(manipulations)i(as)f(in)g(the)g (previous)g(example,)h(but)f(using)h(absolute)75 1863 y(addresses)c(in)h (datat)o(yp)q(es.)75 1969 y Fp(struct)23 b(Partstruct)147 2026 y({)147 2082 y(int)g(class;)147 2139 y(double)g(d[6];)147 2195 y(char)g(b[7];)147 2252 y(};)75 2364 y(struct)g(Partstruct)g(particle[1000];) 337 2477 y(/*)h(build)f(datatype)g(describing)g(first)g(array)g(entry)g(*/)75 2590 y(MPI_datatype)f(Particletype;)75 2647 y(MPI_datatype)g(type[3])h(=)h ({MPI_int,)e(MPI_double,)h(MPI_char};)75 2703 y(int)238 b(block[3])23 b(=)h({1,)f(6,)h(7};)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 44 46 bop 75 -100 a Fu(SECTION)16 b(1.)30 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)637 b FA(44)75 49 y Fp(int)238 b(disp[3];)75 162 y(MPI_address\()22 b(\(void)h(*\)particle,)g(disp++\);)75 219 y(MPI_address\()f(\(void)h (*\)particle[0].d,)f(disp++\);)75 275 y(MPI_address\()g(\(void)h (*\)particle[0].b,)f(disp\);)75 332 y(MPI_type_struct\()g(3,)h(stype,)g (sindex,)g(sblock,)g(Particletype\);)75 388 y(/*)h(Particletype)e(describes)g (first)i(array)f(entry)g(--)h(using)f(absolute)147 444 y(addresses)f(*/)505 557 y(/*)h(5.1:)361 614 y(send)h(the)f(entire)g(array)g(*/)75 727 y(MPI_commit\()f(Particletype\);)75 783 y(MPI_send\()h(MPI_bottom,)f (1000,)h(Particletype,)f(dest,)h(tag,)h(comm\);)481 953 y(/*)f(5.2:)290 1009 y(send)g(the)h(entries)e(of)i(class)f(zero,)290 1065 y(preceded)g(by)g (the)h(number)f(of)g(such)h(entries)e(*/)75 1178 y(MPI_datatype)g (Zparticles,)h(Ztype;)75 1291 y(int)g(zindex[1000],)g(zblock[1000],)f(i,)h (j,)h(k;)75 1348 y(int)f(zzblock[2])g(=)h({1,1};)75 1404 y(int)f(zztype[2],)g (zzindex[2];)75 1574 y(j=0;)75 1630 y(for)g(\(i=0;)h(i)f(<)h(1000;)f(i++\)) 123 1686 y(if)g(\(particle[i].index==0\))170 1743 y({)170 1799 y(for)h(\(k=i+1;)f(\(k)g(<)h(1000\)&&\(particle[k].index)d(==)i(0\))h(;)f (k++\);)170 1856 y(zindex[j])g(=)h(i;)170 1912 y(zblock[j])f(=)h(k-i;)170 1969 y(j++;)170 2025 y(i)g(=)g(k;)170 2082 y(})75 2138 y(MPI_type_indexed\()e (j,)h(Particletype,)46 b(zindex,)23 b(zblock,)g(Zparticles\);)75 2195 y(/*)h(Zparticles)e(describe)h(particles)f(with)i(class)f(zero,)g(using) 147 2251 y(their)g(absolute)g(addresses*/)75 2364 y(/*)h(prepend)e(particle)h (count)g(*/)75 2420 y(MPI_address\(\(void)f(*\)&j,)h(zzindex\);)75 2477 y(zzindex[1])f(=)i(MPI_bottom;)75 2533 y(zztype[0])f(=)g(MPI_int;)75 2590 y(zztype[1])g(=)g(Zparticles;)75 2646 y(MPI_type_struct\(2,)e(zztype,)i (zzindex,)g(zzblock,)g(Ztype\);)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 45 47 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(45)75 49 y Fp(MPI_type_commit\()22 b(Ztype\);)75 106 y(MPI_send\()h(MPI_bottom,)f(1,)i(Ztype,)f(dest,)g(tag,)g(comm\);)75 284 y Fr(1.13.7)49 b(Co)o(rrect)15 b(use)h(of)g(addresses)75 370 y FA(Successiv)o(ely)23 b(declared)f(v)m(ariables)h(in)e(C)g(or)g(F)l (ortran)e(are)i(not)g(necessarily)h(stored)f(at)f(con)o(tiguous)75 426 y(lo)q(cations.)i(Th)o(us,)16 b(care)g(m)o(ust)f(b)q(e)h(exercised)i (that)d(displacemen)o(ts)i(do)f(not)f(cross)h(from)f(one)h(v)m(ariable)75 483 y(to)e(another.)20 b(Also,)15 b(in)g(mac)o(hines)h(with)f(segmen)o(ted)g (address)g(space,)g(addresses)g(are)g(not)f(unique)j(and)75 539 y(address)11 b(arithmetic)h(has)g(some)f(p)q(eculiar)i(prop)q(erties.)19 b(Th)o(us,)12 b(use)g(of)f Fw(addresses)p FA(,)f(i.e.)20 b(displacemen)o(ts) 75 596 y(relativ)o(e)c(to)e(the)i(start)e(address)29 b Fm(MPI)p 733 596 13 2 v 14 w(BOTTOM)p FA(,)14 b(has)h(to)g(b)q(e)h(restricted.)166 652 y(V)l(ariables)k(b)q(elong)f(to)f(the)h(same)f Fw(sequen)o(tial)k (storage)d FA(if)g(they)g(b)q(elong)g(to)f(the)h(same)f(arra)o(y)l(,)75 708 y(to)g(the)g(same)g Fp(COMMON)g FA(blo)q(c)o(k)h(in)g(F)l(ortran,)f(or)g (to)f(the)i(same)f(structure)g(in)h(C.)f(V)l(alid)i(addresses)e(are)75 765 y(de\014ned)f(recursiv)o(ely)f(as)f(follo)o(ws:)131 871 y(1.)22 b(The)g(function)g Ft(MPI)p 558 871 14 2 v 16 w(ADDRESS)h FA(returns)e(a)h(v)m(alid)h(address,)h(when)e(passed)g(as)f(argumen)o(t)g(a) 189 928 y(v)m(ariable)16 b(of)f(the)g(calling)i(program.)131 1021 y(2.)22 b(The)16 b Ft(sta)o(rt)h FA(parameter)e(of)h(a)f(comm)o (unication)i(function)g(ev)m(aluates)g(to)f(a)g(v)m(alid)h(address,)f(when) 189 1078 y(passed)f(as)g(argumen)o(t)f(a)h(v)m(ariable)i(of)e(the)g(calling)i (program.)131 1172 y(3.)22 b(If)15 b Ft(v)g FA(is)h(a)f(v)m(alid)i(address,)d (and)i Ft(i)f FA(is)g(an)h(in)o(teger,)f(then)g Ft(v+i)g FA(is)h(a)f(v)m (alid)i(address,)e(pro)o(vided)h Ft(v)f FA(and)189 1228 y Ft(v+i)g FA(are)g(in)h(the)f(same)g(sequen)o(tial)h(storage.)131 1322 y(4.)22 b(If)15 b Ft(v)g FA(is)h(a)f(v)m(alid)i(address)e(then)29 b Fm(MPI)p 820 1322 13 2 v 15 w(BOTTOM)13 b(+)h(v)h FA(is)h(a)f(v)m(alid)i (address.)166 1428 y(A)22 b(correct)g(program)g(uses)g(only)h(v)m(alid)i (addresses)d(to)g(iden)o(tify)i(the)e(lo)q(cations)h(of)f(en)o(tries)h(in)75 1485 y(comm)o(unication)16 b(bu\013ers.)k(F)l(urthermore,)14 b(if)h Ft(u)h FA(and)f Ft(v)g FA(are)g(t)o(w)o(o)f(v)m(alid)j(addresses,)e (then)g(the)g(\(in)o(teger\))75 1541 y(di\013erence)j Ft(u)g(-)f(v)g FA(can)g(b)q(e)h(computed)g(only)f(if)h(b)q(oth)f Ft(u)g FA(and)h Fp(v)f FA(are)f(in)i(the)g(same)e(sequen)o(tial)j(storage;)75 1598 y(no)c(other)g(arithmetic)h(op)q(erations)f(can)g(b)q(e)h(meaningfully)i (executed)e(on)f(addresses.)166 1654 y(W)l(e)f(sa)o(y)g(that)g(a)g(datat)o (yp)q(e)g(is)h Fw(absolute)g FA(if)g(all)h(displacemen)o(ts)g(within)f(this)g (datat)o(yp)q(e)f(are)g(v)m(alid)75 1710 y(\(absolute\))h(addresses;)g(it)g (is)h Fw(relativ)o(e)g FA(otherwise.)166 1767 y(A)f(correct)g(program)f(ob)q (eys)i(the)f(follo)o(wing)h(constrain)o(ts)e(in)i(the)g(use)f(of)g(datat)o (yp)q(es:)143 1873 y Fl(\017)23 b FA(If)10 b(the)g Ft(oldt)o(yp)q(e)i FA(argumen)o(t)d(used)i(in)g Ft(MPI)p 881 1873 14 2 v 15 w(TYPE)p 1013 1873 V 17 w(CONTIGUOUS,)h(MPI)p 1425 1873 V 15 w(TYPE)p 1557 1873 V 17 w(VECTOR,)f(MPI)p 1859 1873 V 16 w(TYPE)p 1992 1873 V 17 w(HVECTOR,)189 1930 y(MPI)p 274 1930 V 15 w(TYPE)p 406 1930 V 17 w(INDEX)20 b FA(or)38 b Ft(MPI)p 739 1930 V 16 w(TYPE)p 872 1930 V 16 w(HINDEX)20 b FA(is)g(absolute,)g(then)g(all)h (addresses)e(within)i(it)189 1986 y(m)o(ust)12 b(b)q(e)h(to)f(v)m(ariables)h (con)o(tained)g(within)h(the)f(same)f(arra)o(y)f(or)h(structure;)h(the)g (result)f(datat)o(yp)q(e)189 2043 y(is)17 b(also)g(absolute,)g(and)g(all)g (computed)g(addresses)g(m)o(ust)g(also)f(fall)i(within)g(the)f(same)f(arra)o (y)f(or)189 2099 y(structure.)143 2193 y Fl(\017)23 b FA(If)e(all)g(en)o (tries)g(of)f(the)h Ft(a)o(rra)o(y)p 698 2193 V 15 w(of)p 750 2193 V 16 w(t)o(yp)q(es)h FA(argumen)o(t)d(of)i Ft(MPI)p 1233 2193 V 15 w(TYPE)p 1365 2193 V 17 w(STRUC)h FA(are)e(absolute)h(ad-)189 2249 y(dresses)d(\(computed)h(b)o(y)g Ft(MPI)p 727 2249 V 15 w(ADDR)p FA(\),)g(then)f(the)h(result)g(datat)o(yp)q(e)f(is)h(also)g (absolute.)30 b(Eac)o(h)189 2306 y(new)14 b(address)h(computed)g(from)f(an)g (old)h(address)g(m)o(ust)f(fall)h(within)h(the)f(same)f(arra)o(y)f(or)h (record)189 2362 y(as)h(the)i(old)f(address.)23 b(\(Addresses)16 b(in)h(the)f(old)h(t)o(yp)q(e)f(ma)o(y)f(not)h(b)q(e)h(all)g(within)g(the)f (same)g(arra)o(y)189 2419 y(or)e(record.\))143 2512 y Fl(\017)23 b FA(If)18 b(a)g(comm)o(unication)g(call)i(is)e(in)o(v)o(ok)o(ed)h(with)f (parameters)f Ft(sta)o(rt)i FA(and)36 b Ft(datat)o(yp)q(e)p FA(,)20 b(then)e(either)189 2569 y Ft(sta)o(rt)c FA(=)26 b Fm(MPI)p 427 2569 13 2 v 14 w(BOTTOM)14 b FA(and)f Ft(datat)o(yp)q(e)j FA(is)e(a)f(handle)i(to)e(an)h(absolute)g(datat)o(yp)q(e,)f(or)g Ft(sta)o(rt)h FA(is)g(set)189 2625 y(to)j(the)i(address)f(of)g(a)g(program)f (v)m(ariable,)j(and)f(all)g(displacemen)o(ts)h(in)37 b Ft(datat)o(yp)q(e)20 b FA(relativ)o(e)f(to)189 2682 y(this)c(base)g(yield)i(addresses)e(that)f (are)g(within)j(the)e(same)f(sequen)o(tial)i(storage)e(as)h(this)g(v)m (ariable.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 46 48 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(46)166 49 y(In)21 b(summary)l(,)g(the)f(t)o(yp)q(e)h (constructors)e Ft(MPI)p 981 49 14 2 v 16 w(TYPE)p 1114 49 V 17 w(CONTIGUOUS,)j(MPI)p 1536 49 V 15 w(TYPE)p 1668 49 V 17 w(VECTOR,)75 106 y(MPI)p 160 106 V 16 w(TYPE)p 293 106 V 17 w(HVECTOR,)15 b(MPI)p 631 106 V 16 w(TYPE)p 764 106 V 17 w(INDEX)f FA(and)29 b Ft(MPI)p 1114 106 V 16 w(TYPE)p 1247 106 V 17 w(HINDEX)14 b FA(can)h(b)q(e)g(recursiv)o(ely)h(ap-)75 162 y(plied)22 b(to)e(build)j(datat)o(yp)q(es)d(that)f(will)k(com)o(bine)e(v) m(ariables)h(that)e(b)q(elong)h(to)f(the)h(same)f(sequen)o(tial)75 219 y(storage;)d(v)m(ariables)i(that)e(do)g(not)g(b)q(elong)i(to)d(the)i (same)f(sequen)o(tial)i(storage)d(can)i(b)q(e)g(com)o(bined)g(to-)75 275 y(gether)d(using)h(one)f(application)i(of)e Ft(MPI)p 788 275 V 16 w(TYPE)p 921 275 V 16 w(STRUCT)p FA(.)166 408 y Fn(Discussion:)166 457 y Fs(It)f(is)g(not)g(exp)q(ected)j(that)d(MPI)g(implemen)o(tations)d (will)i(b)q(e)i(able)f(to)g(detect)i(erroneous,)f(\\out)f(of)f(b)q(ound")75 507 y(displacemen)o(ts)f({)h(unless)g(those)h(o)o(v)o(er\015o)o(w)e(the)i (user)f(address)i(space)e({)g(since)g(the)h(MPI)f(call)f(ma)o(y)e(not)j(kno)o (w)f(the)75 557 y(exten)o(t)j(of)e(the)i(arra)o(ys)f(and)f(records)j(in)d (the)i(host)f(program.)166 607 y(In)o(teger)k(argumen)o(ts)e(of)g(t)o(yp)q(e) i(constructor)h(functions)e(that)g(represen)o(t)i(displacemen)o(ts)e(should)f (b)q(e)i(of)e(a)75 657 y(t)o(yp)q(e)f(that)f(can)h(b)q(e)g(used)g(to)f (represen)o(t)j(an)d(address)i(on)e(the)h(host)g(system)f(\(e.g.,)f(64)h (bits)h(on)f(a)g(64)g(bit)g(address)75 706 y(mac)o(hine\).)166 922 y Fn(Implemen)o(tati)o(on)e(note:)166 971 y Fs(There)i(is)f(no)f(need)i (to)f(distinguish)f(\(absolute\))h(addresses)i(and)e(\(relativ)o(e\))g (displacemen)o(ts)f(on)h(a)f(mac)o(hine)75 1021 y(with)k(con)o(tiguous)h (address)g(space:)25 b Fm(MPI)p 742 1021 13 2 v 14 w(BOTTOM)16 b Fs(is)h(zero,)g(and)g(b)q(oth)f(addresses)j(and)d(displacemen)o(ts)h(are)75 1071 y(in)o(tegers.)h(On)12 b(mac)o(hines)e(where)j(the)f(distinction)f(is)g (required,)h(addresses)i(are)e(recognized)g(as)g(expressions)h(that)75 1121 y(in)o(v)o(olv)o(e)g Fm(MPI)p 292 1121 V 14 w(BOTTOM)p Fs(.)75 1347 y Fx(1.14)60 b(Universal)19 b(comm)o(unication)e(functions)75 1448 y FA(This)g(section)f(is)g(not)g(part)f(of)h(the)g(MPI)g(de\014nition.) 24 b(Its)15 b(goal)h(is)h(to)e(indicate)i(ho)o(w)f(the)g(de\014nition)h(or)75 1505 y(implemen)o(tation)i(of)e(the)g(p)q(oin)o(t)h(to)f(p)q(oin)o(t)g(comm)o (unication)h(functions)h(can)e(b)q(e)h(reduced)g(to)f(a)g(small)75 1561 y(n)o(um)o(b)q(er)i(of)e(primitiv)o(es.)31 b(W)l(e)18 b(assume)g(the)h(a)o(v)m(ailabilit)o(y)h(of)e(the)g(primitiv)o(e)i(functions) f(listed)g(at)f(the)75 1618 y(end)e(of)f(the)g(section,)g(and)h(the)f(abilit) o(y)h(to)f(allo)q(cate)h(memory)f(for)f(new)i(v)m(ariables.)166 1674 y(W)l(e)g(assume)g(that)f(the)i(function)g Ft(MPI)p 846 1674 14 2 v 15 w(TYPE)p 978 1674 V 17 w(COMMIT)f FA(is)g(not)g(c)o(hanging)h (the)f(represen)o(tation)75 1730 y(of)f(a)g(datat)o(yp)q(e,)f(so)h(that)f (this)i(function)g(is)g(redundan)o(t.)166 1787 y(Comm)o(unication)g(with)h(a) e Ft(count)j FA(argumen)o(t)d(that)g(is)i Fi(>)d FA(1)i(can)g(b)q(e)h (reduced)g(to)e(comm)o(unication)75 1843 y(with)h Ft(count)h(=1)p FA(,)f(b)o(y)f(replicating)j(the)31 b Ft(datat)o(yp)q(e)17 b FA(argumen)o(t)e Ft(count)i FA(times.)22 b(Comm)o(unication)16 b(with)g(a)75 1900 y Ft(a)f(sta)o(rt)h FA(argumen)o(t)f(that)g(is)h(not)29 b Fm(MPI)p 735 1900 13 2 v 15 w(BOTTOM)15 b FA(can)h(b)q(e)g(reduced)h(to)e (comm)o(unication)h(with)g Ft(sta)o(rt)g FA(=)75 1956 y Fm(MPI)p 152 1956 V 14 w(BOTTOM)f FA(b)o(y)g(creating)h(a)e(suitable)j(displaced)g (new)f(datat)o(yp)q(e.)j(Th)o(us:)166 2013 y Ft(MPI)p 251 2013 14 2 v 16 w(SEND\()c(sta)o(rt,)g(count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f (comm)m(\))d FA(is)75 2107 y Fp(Type[1])23 b(=)h(datatype)75 2163 y(MPI_ADDRESS\()e(start,)h(Disp[1]\);)75 2219 y(Blocklen[1])f(=)i (count;)75 2276 y(MPI_TYPE_STRUCT\()e(1,)h(Type,)g(Disp,)h(Blocklen,)e (Newtype\))75 2332 y(MPI_SEND\()h(MPI_BOTTOM,)f(1,)i(Newtype,)e(dest,)i(tag,) f(comm\))166 2426 y FA(The)18 b(same)g(construction)h(applies)g(to)f(all)h (other)f(comm)o(unication)g(functions)h(with)g(argumen)o(ts)75 2483 y Ft(count)e FA(and)f Ft(datat)o(yp)q(e)p FA(.)24 b(W)l(e)15 b(shall)i(henceforth)g(restrict)e(ourselv)o(es)i(to)e(comm)o(unication)h (that)f(in)o(v)o(olv)o(es)75 2539 y(only)h(one)f(elemen)o(t)h(with)g(an)f (absolute)g(address)g(\()p Ft(count)i(=1)p FA(,)d(and)i Ft(sta)o(rt)f FA(=)30 b Fm(MPI)p 1491 2539 13 2 v 14 w(BOTTOM)p FA(\).)166 2596 y(It)10 b(is)h(con)o(v)o(enien)o(t)g(to)e(to)h(mak)o(e)f(explicit)k(the) d Fw(p)q(ersistence)g FA(attribute)g(of)g(comm)o(unication)h(ob)s(jects.)75 2652 y(A)f Ft(p)q(ersistent)j FA(comm)o(unication)e(ob)s(ject)e(need)i(b)q(e) g(explicitly)i(deallo)q(cated)f(b)o(y)e(a)g Ft(MPI)p 1516 2652 14 2 v 15 w(COMMOBJ)p 1757 2652 V 18 w(FREE)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 47 49 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(47)75 49 y(op)q(eration.)19 b(On)14 b(the)e(other)h(hand,)g (an)g Ft(ephemeral)e FA(comm)o(unication)i(ob)s(ject)f(is)i(go)q(o)q(d)e(for) g(a)h(single)h(com-)75 106 y(m)o(unication.)33 b(It)19 b(is)h(deallo)q(cated) g(b)o(y)f(the)g(system)g(when)h(the)f(\014rst)g(comm)o(unication)h(it)f(is)h (used)g(for)75 162 y(completes.)166 219 y(The)15 b(function)h Ft(MPI)p 522 219 14 2 v 16 w(COMM)p 680 219 V 16 w(INIT)f FA(is)g(a)g(new)g (univ)o(ersal)i(function)f(for)e(the)h(creation)h(of)e(comm)o(u-)75 275 y(nication)i(ob)s(jects:)75 379 y Ft(MPI)p 160 379 V 16 w(COMM)p 318 379 V 16 w(INIT\(handle,)f(datat)o(yp)q(e,)i(source-dest,)g (tag,)e(comm)m(,)d(op-mo)q(de,)g(p)q(ersistence\))117 456 y Fs(OUT)108 b Ft(handle)465 b Fs(new)15 b(comm)o(uni)o(cation)c(ob)r(ject)k (\(handle\))117 530 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(elemen)o(t)f(sen)o(t)i(or)f(receiv)o(ed)117 603 y(IN)155 b Ft(source-dest)377 b Fs(rank)14 b(of)f(destination)h(or)g(source)h(\(in)o (teger\))117 677 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o (teger\))117 751 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 824 y(IN)155 b Ft(op-mo)q(de)420 b Fs(one)10 b(of)18 b Fm(MPI)p 1106 824 13 2 v 14 w(ST)m(AND)o(ARD)p Fs(,)f Fm(MPI)p 1443 824 V 14 w(READ)o(Y)p Fs(,)g Fm(MPI)p 1698 824 V 15 w(SYNCHRONOUS)905 881 y Fs(or)28 b Fm(MPI)p 1047 881 V 14 w(RECV)117 955 y Fs(IN)155 b Ft(p)q(ersistence)383 b Fs(one)14 b(of)27 b Fm(MPI)p 1119 955 V 15 w(PERSISTENT)13 b Fs(or)28 b Fm(MPI)p 1534 955 V 14 w(EPHEMERAL)75 1143 y Fr(1.14.1)49 b(P)o(ersistent)14 b(comm)o(unication)g(objects)75 1229 y Ft(MPI)p 160 1229 14 2 v 16 w(CREA)l(TE)p 345 1229 V 17 w(SEND\(handle,)j(MPI)p 731 1229 V 15 w(BOTTOM,)f(1,)e(datat)o(yp)q(e,)j(dest,)f(tag,)f(comm)m(\))d FA(is)75 1325 y Fp(MPI_COMM_INIT\(handle,)21 b(datatype,)i(dest,)g(tag,)g (comm,)g(MPI_STANDARD,)f(MPI_PERSISTENT\))166 1421 y FA(The)f(functions)g Ft(MPI)p 551 1421 V 16 w(CREA)l(TE)p 736 1421 V 17 w(RSEND)p FA(,)g Ft(MPI)p 1015 1421 V 16 w(CREA)l(TE)p 1200 1421 V 17 w(RSEND)g FA(and)g Ft(MPI)p 1560 1421 V 16 w(CREA)l(TE)p 1745 1421 V 17 w(RECV)75 1478 y FA(are)15 b(dealt)h(in)g(a)e(similar)j(manner.)166 1534 y(The)e(functions)h Ft(MPI)p 540 1534 V 16 w(ST)l(ART)g FA(and)g Ft(MPI)p 885 1534 V 15 w(COMMOBJ)p 1126 1534 V 18 w(FREE)g FA(are)f(primitiv)o(e.)75 1655 y Fr(1.14.2)49 b(Nonblo)q(cking)19 b(comm)n(unication)14 b(initiation)75 1740 y Ft(MPI)p 160 1740 V 16 w(ISEND\(handle,)i(MPI)p 557 1740 V 15 w(BOTTOM,)g(1,)f(datat)o(yp)q(e,) h(dest,)g(tag,)f(comm)m(\))d FA(is)75 1837 y Fp(MPI_COMM_INIT\(handle,)21 b(datatype,)i(dest,)g(tag,)g(comm,)g(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1893 y(MPI_START\(handle\))166 1989 y FA(The)g(functions)g Ft(MPI)p 553 1989 V 16 w(IRSEND)p FA(,)f Ft(MPI)p 844 1989 V 16 w(ISSEND)h FA(and)g Ft(MPI)p 1214 1989 V 16 w(IRECV)g FA(are)f(handled)i(in)g(a)e(similar)75 2046 y(manner.)75 2166 y Fr(1.14.3)49 b(Comm)o(unication)14 b(completion)75 2252 y FA(The)h(t)o(w)o(o)f(primitiv)o(e)j(completion)f(op)q(erations)g(are)e Ft(MPI)p 1063 2252 V 16 w(W)l(AIT)l(ANY)i FA(and)g Ft(MPI)p 1467 2252 V 16 w(TEST)l(ALL)p FA(.)166 2308 y(The)c(function)h Ft(MPI)p 516 2308 V 15 w(W)l(AIT)f FA(can)g(b)q(e)h(implemen)o(ted)g(b)o(y)f (a)f(call)i(to)e Ft(MPI)p 1368 2308 V 16 w(W)l(AIT)l(ANY)i FA(with)f(an)24 b Ft(a)o(rra)o(y-)75 2365 y(of-handles)17 b FA(argumen)o(t)d(of)h(length)h(one.)166 2421 y(The)f(function)g Ft(MPI)p 521 2421 V 16 w(W)l(AIT)l(ALL)g FA(can)g(b)q(e)g(implemen)o(ted)h (as)f(a)f(lo)q(op)h(where)g(an)g Ft(MPI)p 1621 2421 V 16 w(W)l(AIT)g FA(call)g(is)75 2478 y(executed)h(for)f(eac)o(h)g(successiv)o(e)h(handle)h (in)f(the)f Ft(a)o(rra)o(y-of-handles)p FA(.)166 2534 y(The)f(function)g Ft(MPI)p 519 2534 V 16 w(TEST)g FA(can)f(b)q(e)h(implemen)o(ted)i(b)o(y)d(a)g (call)i(to)e Ft(MPI)p 1387 2534 V 16 w(TEST)l(ALL)g FA(with)h(an)g Ft(a)o(rra)o(y-)75 2591 y(ofhandles)j FA(of)d(length)i(one.)166 2647 y(The)h(function)h Ft(MPI)p 526 2647 V 16 w(TEST)l(ANY)h FA(can)e(b)q(e)h(implemen)o(ted)h(as)d(a)h(lo)q(op)h(where)f(an)h Ft(MPI)p 1663 2647 V 15 w(TEST)g FA(call)75 2704 y(is)e(executed)g(for)e(eac) o(h)i(successiv)o(e)g(handle)h(in)f(the)f Ft(a)o(rra)o(y-of-handle)p FA(.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 48 50 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(48)75 49 y Fr(1.14.4)49 b(Blo)q(cking)18 b(comm)o(unication)75 135 y Ft(MPI)p 160 135 14 2 v 16 w(SEND\(MPI)p 393 135 V 16 w(BOTTOM,)d(1,)g(datat)o(yp)q(e,)h(dest,)h(tag,)d(comm)m(\))e FA(is)75 241 y Fp(MPI_COMM_INIT\(handle,)21 b(datatype,)i(dest,)g(tag,)g (comm,)g(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 298 y(MPI_START\(handle\))75 354 y(MPI_WAIT\(handle,)g(dontcarestatus\))166 461 y FA(The)15 b(functions)h Ft(MPI)p 540 461 V 16 w(RSEND)g FA(and)f Ft(MPI)p 888 461 V 16 w(SSEND)h FA(are)f(handled)i(in)f(a)f(similar)h(manner.)166 517 y Ft(MPI)p 251 517 V 16 w(RECV\(MPI)p 482 517 V 16 w(BOTTOM,)g(1,)e (datat)o(yp)q(e,)j(source,)e(tag,)g(comm)m(\))d FA(is)75 623 y Fp(MPI_COMM_INIT\(handle,)21 b(datatype,)i(dest,)g(tag,)g(comm,)g (MPI_RECV,)g(MPI_EPHEMERAL\))75 680 y(MPI_START\(handle\))75 736 y(MPI_WAIT\(handle,)f(status\))75 858 y Fr(1.14.5)49 b(Prob)q(e)17 b(and)f(cancel)75 944 y FA(The)f(functions)h Ft(MPI)p 449 944 V 16 w(PROBE)h FA(and)e Ft(MPI)p 800 944 V 16 w(CANCEL)g FA(are)g(primitiv)o (e.)75 1065 y Fr(1.14.6)49 b(Return)16 b(status)75 1151 y FA(The)k(functions) g Ft(MPI)p 458 1151 V 16 w(GET)p 562 1151 V 16 w(SOURCE)p FA(,)h Ft(MPI)p 868 1151 V 16 w(GET)p 972 1151 V 16 w(T)l(A)o(G)p FA(,)f Ft(MPI)p 1189 1151 V 16 w(GET)p 1293 1151 V 16 w(LEN)p FA(,)f Ft(MPI)p 1507 1151 V 16 w(PROBE)p 1671 1151 V 18 w(LEN)p FA(,)g(and)75 1208 y Ft(MPI)p 160 1208 V 16 w(TEST)p 290 1208 V 16 w(CANCELLED)d FA(are)e(primitiv)o(e.)166 1264 y(These)h(functions)h(are) f(simple)i(macros)d(that)h(access)g(records)g(in)h(a)f(structure.)75 1386 y Fr(1.14.7)49 b(send-receive)15 b(and)i(exchange)75 1472 y Ft(MPI)p 160 1472 V 16 w(SENDRECV\()22 b(send)p 531 1472 V 18 w(sta)o(rt,)g(send)p 753 1472 V 18 w(count,)h(send)p 995 1472 V 18 w(t)o(yp)q(e,)g(dest,)h(recv)p 1319 1472 V 16 w(sta)o(rt,)e(recv)p 1533 1472 V 17 w(count,)h(recv)p 1768 1472 V 16 w(t)o(yp)q(e,)75 1528 y(source,)15 b(tag,)g(comm)m(,)d(status\))17 b FA(is)75 1634 y Fp(MPI_ISEND\(handle[0],)k(send_start,)h(send_count,)h(send_type,)f (dest,)i(tag,)f(comm\))75 1691 y(MPI_IRECV\(handle[1],)e(recv_start,)h (recv_count,)h(recv_type,)f(source,)h(tag,)h(comm\))75 1747 y(MPI_WAITALL\(2,)e(handle,)h(status_array\))75 1804 y(status)g(=)h (status_array[1])166 1910 y FA(The)13 b(non)o(blo)q(c)o(king)h(sends)f(and)g (receiv)o(es)h(can)f(b)q(e)g(replaced)h(b)o(y)f(their)g(primitiv)o(e)h (implemen)o(tation.)166 1966 y(The)f(function)i Ft(MPI)p 519 1966 V 15 w(EX)o(CHANGE)g FA(can)e(b)q(e)h(handled)h(in)f(a)f(similar)i (manner;)e(a)g(temp)q(orary)g(bu\013er)75 2023 y(need)20 b(b)q(e)h(allo)q (cated)f(to)f(replicate)i(the)e(send)h(and)g(receiv)o(e)h(bu\013er.)32 b(A)20 b(size)g(for)f(this)h(bu\013er)g(can)f(b)q(e)75 2079 y(computed)d(using)g(the)f(function)h Ft(MPI)p 745 2079 V 16 w(TYPE)p 878 2079 V 16 w(EXTENT)p FA(.)75 2201 y Fr(1.14.8)49 b(Derived)16 b(datat)o(yp)q(es)75 2287 y FA(W)l(e)22 b(ha)o(v)o(e)f(outlined) i(in)f(Section)g(1.13.1)e(ho)o(w)h(eac)o(h)h(datat)o(yp)q(e)e(constructor)h (can)h(b)q(e)g(expressed)g(in)75 2343 y(terms)d(of)g(the)g(next)g(one.)32 b(Th)o(us)20 b(all)g(datat)o(yp)q(e)f(constructors)f(can)h(b)q(e)h(expressed) g(in)g(terms)f(of)g(the)75 2400 y(constructor)11 b Ft(MPI)p 397 2400 V 16 w(TYPE)p 530 2400 V 17 w(STRUCT)p FA(;)h(the)h(functions)g Ft(MPI)p 1099 2400 V 15 w(TYPE)p 1231 2400 V 17 w(FREE)p FA(,)f Ft(MPI)p 1464 2400 V 16 w(TYPE)p 1597 2400 V 17 w(EXTENT)h FA(and)75 2456 y Ft(MPI)p 160 2456 V 16 w(ADDRESS)j FA(are)f(primitiv)o(e.) -32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 49 51 bop 75 -100 a Fu(SECTION)16 b(1.)35 b(POINT)15 b(TO)h(POINT)f(COMMUNICA)l (TION)632 b FA(49)75 49 y Fr(1.14.9)49 b(Summ)o(a)o(r)o(y)75 135 y FA(In)13 b(order)e(to)g(implemen)o(t)j(the)e(MPI)f(p)q(oin)o(t-to-p)q (oin)o(t)i(comm)o(unication)g(one)f(needs)g(16)g(functions;)h(ab)q(out)75 192 y(half)j(of)e(those)h(are)g(trivial)h(macros:)131 298 y(1.)22 b Ft(MPI)p 274 298 14 2 v 15 w(COMM)p 431 298 V 17 w(INIT)131 392 y FA(2.)g Ft(MPI)p 274 392 V 15 w(ST)l(ART)131 485 y FA(3.)g Ft(MPI)p 274 485 V 15 w(W)l(AIT)l(ALL)131 579 y FA(4.)g Ft(MPI)p 274 579 V 15 w(TEST)l(ANY)131 673 y FA(5.)g Ft(MPI)p 274 673 V 15 w(PROBE)131 767 y FA(6.)g Ft(MPI)p 274 767 V 15 w(CANCEL)131 861 y FA(7.)g Ft(MPI)p 274 861 V 15 w(GET)p 377 861 V 17 w(SOURCE)131 954 y FA(8.)g Ft(MPI)p 274 954 V 15 w(GET)p 377 954 V 17 w(T)l(A)o(G)131 1048 y FA(9.)g Ft(MPI)p 274 1048 V 15 w(GET)p 377 1048 V 17 w(LEN)108 1142 y FA(10.)g Ft(MPI)p 274 1142 V 15 w(PROBE)p 437 1142 V 18 w(LEN)108 1236 y FA(11.)g Ft(MPI)p 274 1236 V 15 w(IS)p 327 1236 V 17 w(CANCELLED)108 1330 y FA(12.)g Ft(MPI)p 274 1330 V 15 w(TYPE)p 406 1330 V 17 w(STRUCT)108 1424 y FA(13.)g Ft(MPI)p 274 1424 V 15 w(TYPE)p 406 1424 V 17 w(EXTENT)108 1517 y FA(14.)g Ft(MPI)p 274 1517 V 15 w(ADDRESS)108 1611 y FA(15.)g Ft(MPI)p 274 1611 V 15 w(TYPE)p 406 1611 V 17 w(FREE)108 1705 y FA(16.)g Ft(MPI)p 274 1705 V 15 w(COMM)p 431 1705 V 17 w(FREE)166 1811 y FA(On)17 b(the)f(other)g(hand,)h(w)o(e)f(exp)q(ect)h (most)f(implemen)o(tations)h(to)f(use)h(more)e(primitiv)o(es,)j(in)f(order)75 1868 y(to)e(reduce)h(comm)o(unication)g(o)o(v)o(erheads,)e(reduce)i (bu\013ering,)f(and)h(impro)o(v)o(e)f(error)f(rep)q(orting.)-32 46 y Fv(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 15 15:01:45 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23808; Wed, 15 Sep 93 15:01:45 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29593; Wed, 15 Sep 93 14:59:30 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 15 Sep 1993 14:59:28 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29585; Wed, 15 Sep 93 14:59:27 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13522; Wed, 15 Sep 1993 14:59:25 -0400 Date: Wed, 15 Sep 1993 14:59:25 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309151859.AA13522@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: MPI_address in C code In a C program is there ever any need to call MPI_address to find the address of something? Surely one can always get the address of a variable in a C code. In the fourth example in the point-to-point chapter couldn't one say: sindex[0] = particle; sindex[1] = &(particle[0].d); sindex[2] = &(particle[0].b); instead of MPI_address( (void *)particle, sindex); MPI_address( (void *)particle[0].d, sindex+1); MPI_address( (void *)particle[0].b, sindex+2); David From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 28 19:53:31 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19210; Tue, 28 Sep 93 19:53:31 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09577; Tue, 28 Sep 93 19:52:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 28 Sep 1993 19:52:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09569; Tue, 28 Sep 93 19:52:07 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 7643; Tue, 28 Sep 93 19:52:12 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 8677; Tue, 28 Sep 1993 19:52:12 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 28 Sep 93 19:52:10 EDT Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA19017; Tue, 28 Sep 1993 19:52:03 -0400 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9309282352.AA19017@snir.watson.ibm.com> To: mpi-pt2pt@cs.utk.edu Reply-To: snir@watson.ibm.com Subject: Order semantics, yet again Date: Tue, 28 Sep 93 19:52:02 -0500 -:) -:) -:) -:) -:) Consider the following program process 0 MPI_isend(a,1, MPI_REAL, 1, 0, MPI_WORLD) MPI_isend(b,1, MPI_REAL, 1, 0, MPI_WORLD) process 1 MPI_irecv(a,1, MPI_REAL, 0, MPI_ANYTAG, MPI_WORLD) MPI_irecv(b,1, MPI_REAL, 0, 0, MPI_WORLD) If the two sends of process 0 are matched by the two receives of process 1, then 1st send should match 1st receive and second send should match 2nd receive. This precludes an implementation where receives with fixed tag are posted in a different structure than receives with wildcard tags, and where the first structure is always searched first when a message arrives (with such implementation the first send could match the second receive, if it arrives after both receives where posted). Some implementers feel this could lead to significant performance loss, since an incoming message cannot be matched to a receive with a determinate tag and source, without checking for earlier receives with wildcard parameter(s) that might also also match the arriving message. It runs against our philosophy not to penalize the more frequent case (no wildcards), where a hashing algorithm can be used for fast matching of sends and receives in order to handle the less frequent case (wildcards). The alternative is to specify that order is preserved only for sends (receives) with identical destination, tag and communicator parameters. I think we have already discussed this issue, but I promised to make a further round because of the potential impact. Anybody feeling strongly that we should not weaken the order requirement (in the sense provided by the alternative definition)? Any other implementor feeling a weaker requirement would make its life simpler (or faster)? Anybody cares? From owner-mpi-pt2pt@CS.UTK.EDU Tue Sep 28 20:12:12 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19278; Tue, 28 Sep 93 20:12:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA10836; Tue, 28 Sep 93 20:11:21 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 28 Sep 1993 20:11:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA10828; Tue, 28 Sep 93 20:11:18 -0400 Received: from t.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA00608; Tue, 28 Sep 93 17:11:12 PDT Message-Id: <9309290011.AA00608@SSD.intel.com> To: snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: Order semantics, yet again In-Reply-To: Your message of "Tue, 28 Sep 93 19:52:02 CDT." <9309282352.AA19017@snir.watson.ibm.com> Date: Tue, 28 Sep 93 17:11:11 -0700 From: prp@SSD.intel.com We (Intel) have an implementation that will work better with the relaxed ordering requirement, and proposed implementations that would be even more sensitive. I support making the rules such that order is guaranteed only for exactly matching destination, tag, and communicator parameters. Paul Pierce From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 01:55:10 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20920; Wed, 29 Sep 93 01:55:10 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA04548; Wed, 29 Sep 93 01:54:12 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 01:54:11 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA04540; Wed, 29 Sep 93 01:54:08 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Tue, 28 Sep 93 22:41 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA18343; Tue, 28 Sep 93 22:38:32 PDT Date: Tue, 28 Sep 93 22:38:32 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: Order semantics, yet again To: prp@SSD.intel.com, snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu, rj_littlefield@pnlg.pnl.gov Message-Id: <9309290538.AA18343@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Marc Snir asks: > Anybody feeling strongly that we should not weaken the order > requirement (in the sense provided by the alternative definition)? > Any other implementor feeling a weaker requirement would make its > life simpler (or faster)? Anybody cares? and Paul Pierce writes: > We (Intel) have an implementation that will work better with the relaxed > ordering requirement, and proposed implementations that would be even more > sensitive. I support making the rules such that order is guaranteed only for > exactly matching destination, tag, and communicator parameters. I care, and I urge us to be very cautious here. Consider: process 0 MPI_isend(a,1, MPI_REAL, 1, 0, MPI_WORLD) MPI_isend(b,1, MPI_REAL, 1, 0, MPI_WORLD) process 1: scenario A MPI_irecv(a,1, MPI_REAL, 0, MPI_ANYTAG, MPI_WORLD) MPI_irecv(b,1, MPI_REAL, 0, MPI_ANYTAG, MPI_WORLD) process 1: scenario B MPI_irecv(a,1, MPI_REAL, 0, 0, MPI_WORLD) MPI_irecv(b,1, MPI_REAL, 0, 0, MPI_WORLD) By the weaker ordering spec, scenario B would have guaranteed order, while scenario A would not. Note that both scenarios are simple point-to-point. I personally find this counterintuitive and suspect that it would be the source of some portability Heisenbugs that would be very hard to find. I also am not convinced that the penalty for forced ordering is large enough to worry about. I think it should be a matter of one test for the receive handler to find out whether there are wildcard receives posted. Zero tests, in fact, if you go to the trouble to install one handler for the general case and another for when there are no wildcards posted. Why is this not correct? (I.e., can the vendors convince me that the extra cost is significant, and cannot be eliminated in the common case by being a bit more clever?) Thanks, --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 03:13:09 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA21440; Wed, 29 Sep 93 03:13:09 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09166; Wed, 29 Sep 93 03:12:20 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 03:12:19 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09158; Wed, 29 Sep 93 03:12:18 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08423; Wed, 29 Sep 93 02:11:36 CDT Date: Wed, 29 Sep 93 02:11:36 CDT From: Tony Skjellum Message-Id: <9309290711.AA08423@Aurora.CS.MsState.Edu> To: prp@SSD.intel.com, snir@watson.ibm.com Subject: Re: Order semantics, yet again Cc: mpi-pt2pt@cs.utk.edu I concur with Paul. I was in fact expecting these semantics, but it is good that this issue comes up now, and we make that decision. -Tony From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 04:22:40 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA21618; Wed, 29 Sep 93 04:22:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA13532; Wed, 29 Sep 93 04:21:59 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 04:21:58 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA13524; Wed, 29 Sep 93 04:21:57 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08656; Wed, 29 Sep 93 03:21:14 CDT Date: Wed, 29 Sep 93 03:21:14 CDT From: Tony Skjellum Message-Id: <9309290821.AA08656@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, snir@watson.ibm.com Subject: Re: Order semantics, yet again Let me moderate my comment to say that, at least I am convinced that weakening to be within a context, pairwise ordering is preserved. I have not seen any reason to also allow the relaxation to include TAG. Rik makes good points which I summarize to mean that TAG is not a field over which we relax pairwise ordering, BUT I do want to see it relaxed to be pairwise within a communicator only. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 09:03:56 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22496; Wed, 29 Sep 93 09:03:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09048; Wed, 29 Sep 93 09:02:49 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 09:02:48 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA09040; Wed, 29 Sep 93 09:02:46 -0400 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0661; Wed, 29 Sep 93 09:02:51 EDT Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 2885; Wed, 29 Sep 1993 09:02:50 EDT Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 29 Sep 93 09:02:50 EDT Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA15694; Wed, 29 Sep 1993 09:02:42 -0400 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9309291302.AA15694@snir.watson.ibm.com> To: rj_littlefield@pnlg.pnl.gov Cc: mpi-pt2pt@cs.utk.edu Date: Wed, 29 Sep 93 09:02:41 -0500 ------- Forwarded Message Date: Tue, 28 Sep 93 22:38:32 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: Order semantics, yet again To: prp@SSD.intel.com, snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu, rj_littlefield@pnlg.pnl.gov Message-Id: <9309290538.AA18343@snacker.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Marc Snir asks: > Anybody feeling strongly that we should not weaken the order > requirement (in the sense provided by the alternative definition)? > Any other implementor feeling a weaker requirement would make its > life simpler (or faster)? Anybody cares? and Paul Pierce writes: > We (Intel) have an implementation that will work better with the relaxed > ordering requirement, and proposed implementations that would be even more > sensitive. I support making the rules such that order is guaranteed only for > exactly matching destination, tag, and communicator parameters. I care, and I urge us to be very cautious here. Consider: process 0 MPI_isend(a,1, MPI_REAL, 1, 0, MPI_WORLD) MPI_isend(b,1, MPI_REAL, 1, 0, MPI_WORLD) process 1: scenario A MPI_irecv(a,1, MPI_REAL, 0, MPI_ANYTAG, MPI_WORLD) MPI_irecv(b,1, MPI_REAL, 0, MPI_ANYTAG, MPI_WORLD) process 1: scenario B MPI_irecv(a,1, MPI_REAL, 0, 0, MPI_WORLD) MPI_irecv(b,1, MPI_REAL, 0, 0, MPI_WORLD) By the weaker ordering spec, scenario B would have guaranteed order, while scenario A would not. Note that both scenarios are simple point-to-point. **** ** Both scenarios would have a guaranteed order, since in each case ** both receives have identical parameters. The scenario with no ** guaranteed order is: process 1: scenario C MPI_irecv(a,1, MPI_REAL, 0, MPI_ANY_TAG, MPI_WORLD) MPI_irecv(a,1, MPI_REAL, 0, 0, MPI_WORLD) ** i.e., two receives that match the same message, but have ** distinct matching parameters. ** The overhead is at least one additional test if no wildcard receive ** is posted, and some (significant?) additional work if a wildcard ** receive is posted, even when it does not match the incoming ** message. ** Practically, the current semantics probably means using a linear ** search structure, rather than a hash table. ** On the other hand, the current semantics are cleaner, no doubt. I personally find this counterintuitive and suspect that it would be the source of some portability Heisenbugs that would be very hard to find. I also am not convinced that the penalty for forced ordering is large enough to worry about. I think it should be a matter of one test for the receive handler to find out whether there are wildcard receives posted. Zero tests, in fact, if you go to the trouble to install one handler for the general case and another for when there are no wildcards posted. Why is this not correct? (I.e., can the vendors convince me that the extra cost is significant, and cannot be eliminated in the common case by being a bit more clever?) Thanks, - --Rik - ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 ------- End of Forwarded Message From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 09:56:26 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22682; Wed, 29 Sep 93 09:56:26 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA12999; Wed, 29 Sep 93 09:55:19 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 09:55:18 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA12982; Wed, 29 Sep 93 09:55:16 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA28059 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Wed, 29 Sep 1993 06:55:09 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA22873; Wed, 29 Sep 93 06:47:50 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA00560; Wed, 29 Sep 93 06:57:55 PDT Date: Wed, 29 Sep 93 06:57:55 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9309291357.AA00560@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: I care too. I agree with Rik that relaxing the ordering here may well result in very unintuitive behavior. In particular I often write code that looks like MPI_irecv(a,1, MPI_INTEGER, 0, MPI_ANYTAG, MPI_WORLD) get_actual_tag switch(tag) { case tag1: MPI_irecv(a,1,MPI_INTEGER, 0, yet_another_tag, MPI_WORLD); break; ... } which presumably wouldn't work properly. This seems to one of the basic uses of the tag field (to me, at least). Jon From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 10:15:15 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA22878; Wed, 29 Sep 93 10:15:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA14291; Wed, 29 Sep 93 10:14:24 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 10:14:23 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA14283; Wed, 29 Sep 93 10:14:19 -0400 Received: from canopus (canopus.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA21339 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Wed, 29 Sep 1993 15:14:14 +0100 Date: Wed, 29 Sep 1993 15:14:14 +0100 From: James Cownie Message-Id: <199309291414.AA21339@hub.meiko.co.uk> Received: by canopus (4.1/SMI-4.1) id AA23124; Wed, 29 Sep 93 15:17:02 BST To: mpi-pt2pt@cs.utk.edu In-Reply-To: <9309291302.AA15694@snir.watson.ibm.com> Subject: Message ordering promises Marc writes ** Practically, the current semantics probably means using a linear ** search structure, rather than a hash table. ** On the other hand, the current semantics are cleaner, no doubt. You can still use a hash structure to match where you know the tag. However you ALSO have to chain messages (and receives) in arrival order so that the wildcard receive can get the first one. This is not TOO bad, it costs a doubly linked list insertion and removal on each receive. Of course the best implementation choice is highly dependent on such (unknown, or highly program dependent) parameters as 1) How many unreceived messages are pending ? 2) How many unmatched receives are pending ? I seem to remember Tony claiming that the answer to 1 is "less than six". If both numbers are small , then a linear search structure will win every time despite its poor scaling properties. I'd keep the current semantics, mostly because I don't want to have to explain the other semantics to a naive user ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Sep 29 11:36:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23487; Wed, 29 Sep 93 11:36:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA20413; Wed, 29 Sep 93 11:34:51 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 29 Sep 1993 11:34:50 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA20405; Wed, 29 Sep 93 11:34:49 -0400 Received: by msr.EPM.ORNL.GOV (4.1/1.34) id AA12758; Wed, 29 Sep 93 11:34:48 EDT Date: Wed, 29 Sep 93 11:34:48 EDT From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9309291534.AA12758@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: Re: Order semantics, yet again >Anybody feeling strongly that we should not weaken the order >requirement (in the sense provided by the alternative definition)? I feel strongly that we should not weaken the message order requirement. The weaker requirement would mean that any program that uses wildcard would be nondetermanistic from run to run. I feel this is a bigger penalty than the potential speed gain. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 23 10:27:20 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id KAA03633; Tue, 23 Nov 1993 10:27:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id PAA18298; Tue, 23 Nov 1993 15:26:37 GMT X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Nov 1993 15:26:36 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id KAA18291; Tue, 23 Nov 1993 10:26:31 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 3651; Tue, 23 Nov 93 10:26:31 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 7231; Tue, 23 Nov 1993 10:26:28 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 23 Nov 93 10:26:27 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA42073; Tue, 23 Nov 1993 10:26:29 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9311231526.AA42073@snir.watson.ibm.com> To: mpi-pt2pt@CS.UTK.EDU Cc: cypher@watson.ibm.com Reply-To: snir@watson.ibm.com Subject: order requirements for point to point communication Date: Tue, 23 Nov 93 10:26:29 -0500 Pls look at the following examples and let me know if we agree on the interpretation of the semantics for pt2pt communication A switch(myid) { case 0: { MPI_Ssend(...,1,0,MPI_Comm_world); MPI_Rsend(...,1,1,MPI_Comm_World); } case 1: { MPI_Irecv(...,0,1,MPI_Comm_world); MPI_Irecv(...,0,0,MPI_Comm_world); } } .... Program is correct: 1st send matches second receive, and vice-versa. The 1st send can complete only if the 2nd receive has started, which makes sure that 1st receive is posted. Therefore, it is correct to use a ready-receive send. (Implication: the operation that posts a receive cannot return before it is ready to start accepting the message) B switch(myid) { case 0: { MPI_Send(...,2,0,MPI_Comm_world); MPI_Send(...,1,0,MPI_Comm_World); } case 1: { MPI_Recv(...,0,0,MPI_Comm_world); MPI_Send(...,2,0,MPI_Comm_world); } case 2: { MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); } } .... This is an example of causality violation: Process 0 sends first to 2, next to 1; process 1 receives from 0, next sends to 2; yet process 2 may receive the message of process 1 before it receives the message of process 0: the two receives of 2 may match the two messages sent to it in either order. I.e., order is preserved only on direct links, but need not be preserved when we have indirect causality. C switch(myid) { case 0: { MPI_Ssend(...,2,0,MPI_Comm_world); MPI_Send(...,1,0,MPI_Comm_world); } case 1: { MPI_Recv(...,0,0,MPI_Comm_world); MPI_send(...,2,0,MPI_Comm_world); } case 2: { MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); } } Same code as in example B, with the first send replaced by a synchronous send. A strict interpretation of our draft will imply that in this case, too, either message sent to process 2 may match either receive. However, an "out of order" receive is somewhat strange: The first send of process 0 can complete only if a receive has started at process 2. The message of process 1 arrives at process 2 only after a receive of the message of process 0 has already started. Thus, an "out of order" matching corresponds to a situation where both receives have already been posted at process 2, the message from process 0 arrives before any other message, and even though it matches both posted receives, it is accepted by the second posted receive. I find such behavior somewhat counterintuitive. From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 23 12:46:32 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id MAA04226; Tue, 23 Nov 1993 12:46:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id RAA26347; Tue, 23 Nov 1993 17:45:47 GMT X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Nov 1993 17:45:45 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from epcc.ed.ac.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id MAA26328; Tue, 23 Nov 1993 12:45:40 -0500 Date: Tue, 23 Nov 93 17:45:12 GMT Message-Id: <13121.9311231745@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: order requirements for point to point communication To: snir@watson.ibm.com, mpi-pt2pt@CS.UTK.EDU In-Reply-To: Marc Snir's message of Tue, 23 Nov 93 10:26:29 -0500 Reply-To: lyndon@epcc.ed.ac.uk Cc: cypher@watson.ibm.com Marc Snir writes: > > > Pls look at the following examples and let me know if we agree on > the interpretation of the semantics for pt2pt communication > > A > > switch(myid) > { > > > case 0: { MPI_Ssend(...,1,0,MPI_Comm_world); > MPI_Rsend(...,1,1,MPI_Comm_World); > } > > case 1: { MPI_Irecv(...,0,1,MPI_Comm_world); > MPI_Irecv(...,0,0,MPI_Comm_world); > } > } > .... > > > Program is correct: 1st send matches second receive, and > vice-versa. The 1st send can complete only if the 2nd receive has > started, which makes sure that 1st receive is posted. Therefore, it > is correct to use a ready-receive send. (Implication: the > operation that posts a receive cannot return before it is ready to > start accepting the message) > Agree program is correct. Dont quite understand "Implication:" - if you mean that in the above example the second MPI_Irecv in "case 1" cannot return until the MPI_Ssend in "case 0" has started then this is not consistent with my interpretation which is that MPI_Irecv does not block, however if you mean that said MPI_Irecv cannot return until it has locally prepared itself then that seems okay since it must take bounded time. The example is good. If we agree on the semantics that the program is correct then we are saying that the Isend (Irecv) functions are not equivalent to starting a thread which performs the Send (Recv) function. > > > B > > switch(myid) > { > > case 0: { MPI_Send(...,2,0,MPI_Comm_world); > MPI_Send(...,1,0,MPI_Comm_World); > } > > case 1: { MPI_Recv(...,0,0,MPI_Comm_world); > MPI_Send(...,2,0,MPI_Comm_world); > } > > case 2: { MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); > MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); > } > } > .... > > This is an example of causality violation: Process 0 sends first to > 2, next to 1; process 1 receives from 0, next sends to 2; yet > process 2 may receive the message of process 1 before it receives > the message of process 0: the two receives of 2 may match > the two messages sent to it in either order. > I.e., order is preserved only on direct > links, but need not be preserved when we have indirect causality. > Yeah, pairwise ordering preserved but nothing said about such indirect causality, this is my interpretation. Interesting that if you do replace all of the MPI_Send with MPI_Ssend then the above fragment does not violate "global" causality. > > > C > > switch(myid) > { > > case 0: { MPI_Ssend(...,2,0,MPI_Comm_world); > MPI_Send(...,1,0,MPI_Comm_world); > } > > case 1: { MPI_Recv(...,0,0,MPI_Comm_world); > MPI_send(...,2,0,MPI_Comm_world); > } > > case 2: { MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); > MPI_recv(...,MPI_Any_source,0,MPI_Comm_world); > } > } > > Same code as in example B, with the first send replaced by a > synchronous send. A strict interpretation of our draft will imply > that in this case, too, either message sent to process 2 may match > either receive. However, an "out of order" receive is somewhat > strange: The first send of process 0 can complete only if a > receive has started at process 2. The message of process 1 arrives > at process 2 only after a receive of the message of process 0 has > already started. Thus, an "out of order" matching corresponds to a > situation where both receives have already been posted at process 2, > the message from process 0 arrives before any other message, > and even though it matches both > posted receives, it is accepted by the second posted receive. I > find such behavior somewhat counterintuitive. > Yes, counterintuitive. My interpretation/understanding ... In the above example the two receives in process 2 were not immediate (nonblocking) therefore the second cannot be started before the first has completed and the fragment is deterministic in that the first MPI_Recv in process 2 has accepted the MPI_Ssend in process 0 even before the second recv in process 2 has started (which must then accept the MPI_Send in process 1). If we make the modification you appear to suggest - i.e. make the two MPI_Recv in process 1 into MPI_Irecv switch(myid) { case 0: { MPI_Ssend(...,2,0,MPI_Comm_world); MPI_Send(...,1,0,MPI_Comm_world); } case 1: { MPI_Recv(...,0,0,MPI_Comm_world); MPI_Send(...,2,0,MPI_Comm_world); } case 2: { MPI_Irecv(...,MPI_Any_source,0,MPI_Comm_world); MPI_Irecv(...,MPI_Any_source,0,MPI_Comm_world); } } then it is intuitive that the same matching of sends with receives occurs. The message from process 0 must have *started* to arrive at process 2 *before* the message from process 1, however it could be that the message from process 0 *finished* arriving at process 2 *after* the message from process 1. It seems the challenge is to find the words which will express the desired semantics without ambiguity. Something like "when a message begins to arrive at a process and there is more than one nonblocking receive which can match that message then the nonblocking receive chosen to match is that which was posted earliest" Discussion: Does this have too strong implementation requirements? It is like saying that the behaviour must be as though posted receive requests are stored in a linked list with new requests added to the back of the list and matches taken from the nearest matching request to the front of the list. Now if we do think this is too strong then do we also make a step back to say that the ordering is not defined? If so then do we not have to revise our opinion of the first example Marc gave? Nice hoop, Marc, nice hoop! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 14 06:08:00 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id GAA02313; Tue, 14 Dec 1993 06:07:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id GAA21952; Tue, 14 Dec 1993 06:06:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 14 Dec 1993 06:06:13 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id GAA21945; Tue, 14 Dec 1993 06:06:08 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA22221 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 14 Dec 1993 11:06:00 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA12616; Tue, 14 Dec 1993 11:04:05 +0000 Date: Tue, 14 Dec 1993 11:04:05 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9312141104.AA12616@tycho.co.uk> To: mpi-pt2pt@CS.UTK.EDU Subject: Suggested binding change for "get" functions Content-Length: 2560 People, I propose that we change C and Fortran bindings of the functions MPI_GET_SOURCE, MPI_GET_TAG and MPI_GET_COUNT, so that they may be implemented more easily as macros in C or as statement functions in Fortran. (We discussed this at one of the meetings, but then it got lost in the rush to get areasonable text ou for SC93). Rationale ========= At least one of these three functions is likely to be called with each receive which is preformed, thterefore minimising their cost is extremely important, since it is effectively added to the message passing latency. The loss of an error indication from these functions is a small price to pay, since there are unlikely to be any error conditions detectable at this point anyway. Suggested changes ================= First various misstatements which require correction anyway, the text states that the status object must have been set by a receive. What we want is a receive or a probe. Therefore :- Page 18: line 26 : "by a receive" should read "by a receive or probe " line 27 : "by a receive" should read "by a receive or probe " Similar changes required to definitions of the functions (line 39, page 19 line 1). Need some discussion in GET_COUNT of legal datatypes when used after probe, (since probe doesn't pass in a datatype the statement here needs to be expanded to handle probe). Add new paragraph at page 18 line 28: Since these functions are expected to be very cheap (as they are simply unpicking a structure in user space), called very often, and are incapable of detecting errors, we choose to adopt a different binding for these functions, in which the result of the function is the "natural" result of the query, rather than an error condition. In the C binding we expect that these "functions" will be implemented as macros, therefore users should NOT rely on single evaluation of the arguments to these functions. (i.e. code like tag = MPI_Get_tag(statuses[i++]); is erroneous because of the side effect in the argument). New bindings in the obvious way int MPI_Get_source(int *status) INTEGER FUNCTION MPI_GET_SOURCE(STATUS) INTEGER STATUS(MPI_STATUS_SIZE) and similarly for the other two. Also need a statement in the profiling chapter that you can't expect to profile these macros ! --Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 14 07:45:02 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id HAA02600; Tue, 14 Dec 1993 07:45:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id HAA27717; Tue, 14 Dec 1993 07:46:41 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 14 Dec 1993 07:46:40 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id HAA27710; Tue, 14 Dec 1993 07:46:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA19159 (5.65c/IDA-1.4.4 for ); Tue, 14 Dec 1993 06:46:27 -0600 Message-Id: <199312141246.AA19159@antares.mcs.anl.gov> To: jim@meiko.co.uk (James Cownie) Cc: mpi-pt2pt@CS.UTK.EDU Subject: Re: Suggested binding change for "get" functions In-Reply-To: Your message of "Tue, 14 Dec 1993 11:04:05 GMT." <9312141104.AA12616@tycho.co.uk> Date: Tue, 14 Dec 1993 06:46:27 -0600 From: Rusty Lusk Although I agree with what you say, it is not strictly necessary to change the bindings to be able to define them as macros. The model implementation currently defines them this way: #define MPI_Get_source(status,src) ((src) = (status).source,MPI_OK) #define MPI_Get_tag(status,tg) ((tg) = (status).tag,MPI_OK) #define MPI_Get_count(status,cnt) ((cnt) = (status).count,MPI_OK) Admittedly, this does not address the issue that they are more naturally implemented as functions in Fortran, and hides the fact that no bad return codes are possible. Rusty From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 14 08:16:27 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA02683; Tue, 14 Dec 1993 08:16:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA29711; Tue, 14 Dec 1993 08:18:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 14 Dec 1993 08:18:07 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA29704; Tue, 14 Dec 1993 08:18:03 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA22830 (5.65c/IDA-1.4.4 for mpi-pt2pt@CS.UTK.EDU); Tue, 14 Dec 1993 13:17:58 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA12655; Tue, 14 Dec 1993 13:15:55 +0000 Date: Tue, 14 Dec 1993 13:15:55 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9312141315.AA12655@tycho.co.uk> Cc: mpi-pt2pt@CS.UTK.EDU In-Reply-To: <199312141246.AA19159@antares.mcs.anl.gov> (message from Rusty Lusk on Tue, 14 Dec 1993 06:46:27 -0600) Subject: Re: Suggested binding change for "get" functions Content-Length: 1902 > Although I agree with what you say, it is not strictly necessary to change the > bindings to be able to define them as macros. You are correct, though I'd like to make it explicit that they may be macros, so that 1) we can outlaw side effects in the arguments (though actually it's probably safe anyway provided the macros are similar to yours, where each argument is expanded once only) 2) we can make clear that trying to intercept these using the profiling tricks won't work ! > The model implementation > currently defines them this way: > > #define MPI_Get_source(status,src) ((src) = (status).source,MPI_OK) > #define MPI_Get_tag(status,tg) ((tg) = (status).tag,MPI_OK) > #define MPI_Get_count(status,cnt) ((cnt) = (status).count,MPI_OK) Surely these are wrong, the result argumnts are supposed to be (int *), so shouldn't the macros look like #define MPI_Get_source(status,src) ((*(src) = (status).source),MPI_OK) (or am I wrong somewhere ? The code presumably works !) The other point is that even with macros as above, the current binding looks unpleasant, and may impede code optimisation, since some compilers are not very clever at handling things whose address is taken (even in an expression like *(&s)). The current binding also forces the user to create a temporary which is often unnecessary, e.g. { int tag; MPI_Get_tag(status, &tag); switch(tag) { } } compared with simply switch( MPI_Get_tag(status) ) { } Maybe consistency of interface is more important though. In any case lets say that they're probably macros... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Dec 17 15:06:48 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id PAA06243; Fri, 17 Dec 1993 15:06:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id PAA09639; Fri, 17 Dec 1993 15:07:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 17 Dec 1993 15:07:16 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id PAA09632; Fri, 17 Dec 1993 15:07:13 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA04051 (5.65c/IDA-1.4.4 for ); Fri, 17 Dec 1993 14:07:11 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA04271; Fri, 17 Dec 93 14:07:10 CST Date: Fri, 17 Dec 93 14:07:10 CST Message-Id: <9312172007.AA04271@godzilla.mcs.anl.gov> To: mpi-pt2pt@CS.UTK.EDU Subject: Proposals for MPI_GET_TAG, SOURCE, COUNT. Problems with the current specification: I. Excessive function calls in Fortran Suggested solutions: Make layout of MPI_Status visible to user Fortran: INTEGER MPI_STATUS_SOURCE, MPI_STATUS_TAG, MPI_STATUS_COUNT PARAMETER(MPI_STATUS_SOURCE=1,MPI_STATUS_TAG=2,MPI_STATUS_COUNT=3) INTEGER MPI_STATUS_SIZE PARAMETER (MPI_STATUS_SIZE=3) C: typedef struct { int source, tag, count; } MPI_Status; Version A: These are IN ADDITION to MPI_GET_TAG etc. Version B: These REPLACE MPI_GET_TAG etc. Rationale: MPI_Status already has to be visible to the user since it is allocated by the user (and there is no MPI_CREATE_STATUS routine). The function versions, while reasonable in C because they can easily be implemented as macros, cause potential difficulties for Fortran users. In addition, being able to do "switch (status.tag)" is convenient. We believe no generality is lost in taking this approach (again because MPI_Status is user-allocated). II. MPI_GET_COUNT has a datatype parameter This is needed only when used on a status returned by a probe. In othercases, the datatype would have been supplied on the RECEIVE/WAIT that returned the status. Suggested solutions: Have MPI_GET_COUNT (for receive) and MPI_GET_PROBE_COUNT (for probe). MPI_GET_COUNT( MPI_Status status, int *count ) MPI_GET_PROBE_COUNT( MPI_Status status, MPI_Datatype datatype, int *count ) (If MPI_GET_COUNT called with a status returned from a probe, the results are the same as MPI_GET_PROBE_COUNT( status, MPI_BYTE, count ).) Note that if I.B is accepted, MPI_GET_COUNT disappears but MPI_GET_PROBE_COUNT remains as a routine (that can be implemented as a macro, but because of the potential complexity in handling datatype, probably will not be). Rationale: The datatype argument to MPI_GET_COUNT is either irrelevant (supplied on the RECEIVE) or essential (status from PROBE); this suggests two different functions. Bill and Rusty From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 21 09:24:05 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id JAA21524; Tue, 21 Dec 1993 09:24:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA10374; Tue, 21 Dec 1993 09:25:01 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 21 Dec 1993 09:24:59 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA10366; Tue, 21 Dec 1993 09:24:54 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA14936 (5.65c/IDA-1.4.4 for mpi-pt2pt@CS.UTK.EDU); Tue, 21 Dec 1993 14:24:45 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA16164; Tue, 21 Dec 1993 14:22:33 +0000 Date: Tue, 21 Dec 1993 14:22:33 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9312211422.AA16164@tycho.co.uk> To: mpi-pt2pt@CS.UTK.EDU In-Reply-To: <9312172007.AA04271@godzilla.mcs.anl.gov> (message from William Gropp on Fri, 17 Dec 93 14:07:10 CST) Subject: Re: Proposals for MPI_GET_TAG, SOURCE, COUNT. Content-Length: 1460 Bill suggests making the structure of the status object visible to the user. I think this is OK, BUT we need to be very clear that we DON'T specify the full structure to the user, but only the names of certain fields she can be guaranteed to access. I strongly expect that I will want to hide other things in the status as well as the user visible fields. The Fortran argument is a strong one ! (I had hoped that one could fudge this using statement functions which are often implemented in a macro like way, but I don't now believe this is possible). MPI_GET_COUNT and datatype argument. As I remember it we went around this a few times... the arguments (I think) go like this :- 1) we want MPI_GET_COUNT to be the same whether used after probe or recv. (we were chopping functions at the time !) 2) we can't assume that the datatype can be saved in recv and used in GET_COUNT (because the user could free it in between, and the system doesn't have visibility of the status objects). If we take path 1 and delete all of these functions, then I suppose it doesn't matter anyway. If we keep them I'd like to only have one function and leave the arguments the same... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 3 20:38:41 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id UAA02503; Mon, 3 Jan 1994 20:38:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id UAA05928; Mon, 3 Jan 1994 20:38:29 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 3 Jan 1994 20:38:28 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id UAA05921; Mon, 3 Jan 1994 20:38:27 -0500 Message-Id: <199401040138.UAA05921@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 7743; Mon, 03 Jan 94 20:38:27 EST Date: Mon, 3 Jan 94 20:38:27 EST From: "Marc Snir ((914) 945-3204 (862)" To: mpi-pt2pt@CS.UTK.EDU cc:gropp at mcs.anl.gov, frankeh Subject: datatype argument in MPI_GET_ELEMENT Reply-To: SNIR@watson.ibm.com The two arguments in the favor of current design are: 1. It might be possible to receive messages faster if one does not have to count elements at receive time (Paul Pierce made this argument -- Paul are you on?) Counterargument: one just need to store in the datatype the number of bytes of the datatype and, when receiving, divide total number of bytes received by datatype length. Count is quotient, if integer, or MPI_UNDEFINED, if quotient is not integer. It seems that the saving is < 10 instructions. Moreover, one might use a running element count anyhow for complex datatypes, in order to store data in the right place. 2. Make RECV and PROBE identical. Otherwise, we still need the GET_COUNT in its current form after a call to PROBE. Comments? (Including from implementers...) From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 4 13:42:31 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id NAA07901; Tue, 4 Jan 1994 13:42:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA23314; Tue, 4 Jan 1994 13:42:31 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 4 Jan 1994 13:42:30 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from SSD.intel.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id NAA23307; Tue, 4 Jan 1994 13:42:27 -0500 From: Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA16259; Tue, 4 Jan 94 10:32:23 PST Message-Id: <9401041832.AA16259@SSD.intel.com> To: "Marc Snir ((914) 945-3204 (862)" Cc: mpi-pt2pt@CS.UTK.EDU, prp@ssd.intel.com In-Reply-To: Your message of "Mon, 03 Jan 94 20:38:27 EST." <199401040138.UAA05921@CS.UTK.EDU> Date: Tue, 04 Jan 94 10:32:23 -0800 > From: "Marc Snir ((914) 945-3204 (862)" > > The two arguments in the favor of current design are: > > 1. It might be possible to receive messages faster if one does not have to > count elements at receive time (Paul Pierce made this argument -- Paul are > you on?) Counterargument: one just need to store in the datatype the number > of bytes of the datatype and, when receiving, divide total number of bytes > received by datatype length. Count is quotient, if integer, or MPI_UNDEFINED, > if quotient is not integer. It seems that the saving is < 10 instructions. > Moreover, one might use a running element count anyhow > for complex datatypes, in order to store data in the right place. I find the counterargument adequate. In fact, if you have a GET_COUNT that is only for RECV and doesn't take a datatype, RECV can simply save the datatype in the MPI_Status data structure where it can be used by GET_COUNT when needed. Then you can implement GET_COUNT any way you want. You don't have to do the divide during the RECV itself. Of course, this option makes it harder to argue that MPI_Status should be totally user visible. > > 2. Make RECV and PROBE identical. Otherwise, we still need the GET_COUNT in > its current form after a call to PROBE. The argument for keeping the datatype in GET_COUNT for PROBE is that you can do one (potentially expensive) PROBE and then test it against one (or more) of several different datatypes as needed. I would be happy (because it would make the two calls more consistent and easier to remember) to make RECV and PROBE the same, so they both take a datatype. Then, have a simple GET_COUNT that returns the number of elements of that datatype, plus a GET_COUNT_TYPED that takes a test datatype and returns the number of elements of the test datatype. Or, we could forget GET_COUNT_TYPED and force the user to either do multiple PROBE calls for the same message to get an accurate element count or cheat by using the BYTE datatype and local knownledge of the datatype size. We could expose the first few elements of MPI_Status so that you wouldn't need the extra calls in Fortran. Of course, then you would always have to do the divide in RECV so the count would be there. I don't think its a big deal. Paul From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 5 05:44:16 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id FAA20583; Wed, 5 Jan 1994 05:44:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA03147; Wed, 5 Jan 1994 05:44:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 5 Jan 1994 05:44:00 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA03140; Wed, 5 Jan 1994 05:43:55 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA20069 (5.65c/IDA-1.4.4 for mpi-pt2pt@CS.UTK.EDU); Wed, 5 Jan 1994 10:43:52 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA24942; Wed, 5 Jan 1994 10:41:08 +0000 Date: Wed, 5 Jan 1994 10:41:08 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9401051041.AA24942@tycho.co.uk> To: mpi-pt2pt@CS.UTK.EDU In-Reply-To: <9401041832.AA16259@SSD.intel.com> (prp@ssd.intel.com) Subject: GET_COUNT Content-Length: 1163 I'm pretty much in favour of leaving it the way it is. As I read him Paul also seems happy with this. > In fact, if you have a GET_COUNT that is only for RECV and doesn't > take a datatype, RECV can simply save the datatype in the MPI_Status > data structure where it can be used by GET_COUNT when needed. Then you > can implement GET_COUNT any way you want. You don't have to do the > divide during the RECV itself. One problem with this is that the user could TYPE_FREE the datatype in between the receive and the GET_COUNT call. This makes stashing the type dangerous (note that ref count doesn't work here because the MPI system can't see creation/deletion of the user allocated status object, which is where the reference has to live). In my mind this (and simplifying to a single function rather than two) was a strong reason for having to give the type even after receive. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 6 19:41:59 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id TAA05686; Thu, 6 Jan 1994 19:41:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA15064; Thu, 6 Jan 1994 19:42:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 6 Jan 1994 19:42:07 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA15057; Thu, 6 Jan 1994 19:42:05 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1371; Thu, 06 Jan 94 19:42:05 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 8078; Thu, 6 Jan 1994 19:41:52 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Thu, 06 Jan 94 19:41:52 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA37874; Thu, 6 Jan 1994 19:42:02 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9401070042.AA37874@snir.watson.ibm.com> To: mpi-pt2pt@CS.UTK.EDU Cc: frankeh@watson.ibm.com Subject: MPI_GET_COUNT, again Reply-To: snir@watson.ibm.com Date: Thu, 06 Jan 94 19:42:02 -0500 -:) -:) -:) -:) -:) Possible alternative implementations: 1. Leave everything as is. 2. Have count computed by the receive and stored in status. Then GET_COUNT does not need a datatype argument, and could be actually an access to a field of status, whereas GET_ELEMENT stays as it is now. Additional overhead of, say, 30 cycles per receive. 3. Have both count and #elements computed by the receive and stored in status. Additional overhead of, say, #element cycles per receive. Then, if we change (options 2 or 3), we need to decide what to do with PROBE: A. Have probe carry a datatype -- not pleasant because we may not know yet which datatype to use; we will need to probie again and again. B. Leave a version of GET_COUNT and/or GET_ELEMENTS that have a datatype argument, in addition to the direct access to a status count field. I don't like 3: counting number of (base) elements may be a significant overhead, and might be seldom needed. This leaves 1: use GET_COUNT(status, datatype) and GET_ELEMENT(status, datatype) when count is undefined. 2B: status.count is OK after receive; GET_COUNT(status,datatype) can always be used, in particular after PROBE. Use GET_ELEMENT(status, datatype) when count is undefined. From owner-mpi-pt2pt@CS.UTK.EDU Fri Jan 7 05:51:39 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id FAA06948; Fri, 7 Jan 1994 05:51:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA29064; Fri, 7 Jan 1994 05:51:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 7 Jan 1994 05:51:58 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id FAA29057; Fri, 7 Jan 1994 05:51:54 -0500 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA29837 (5.65c/IDA-1.4.4 for mpi-pt2pt@CS.UTK.EDU); Fri, 7 Jan 1994 10:51:49 GMT Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00815; Fri, 7 Jan 1994 10:49:00 +0000 Date: Fri, 7 Jan 1994 10:49:00 +0000 From: jim@meiko.co.uk (James Cownie) Message-Id: <9401071049.AA00815@tycho.co.uk> To: mpi-pt2pt@CS.UTK.EDU In-Reply-To: <9401070042.AA37874@snir.watson.ibm.com> Subject: Re: MPI_GET_COUNT Content-Length: 4925 I'll run over what I think the arguments and objectives are here, mostly to clarify them to myself ! Please correct me if I'm wrong :-( Objectives ========== 1) Access to status should be CHEAP. (This is the reason for having store for the status allocated by the user, we want to avoid calls to the MPI system as far as possible, and especially extra calls on each communication op). 2) You should only have to pay for status values if you use them. (We want to avoid a large cost of calculating COUNT or ELEMENTS if you never use these values). Route to where we are now ========================= Given that we want to make access to status fields as cheap as possible, it seems reasonable to DEFINE at least a part of the status structure, so that the user can access the fields directly. (This is more important in Fortran, in C it can be hidden in a macro, but in Fortran this option is not available, and we can replace a real function call by an inline array access if we expose the structure.) Once we accept this, then we have to define which of the values accessable from the status object are truly stored in it (and therefore rapidly accessable as structure dereference), and which values really do need calculation at the point of reference. Currently we have the following enquiries on the status object (I may have missed some... beware !) GET_ELEMENTS Leaf element count GET_COUNT Top datatype count GET_SOURCE Rank of sender GET_TAG Tag of message I think SOURCE and TAG are uncontentious, a receive or probe must know these values, so there is no problem in putting them into the status object and allowing the user direct access to them. COUNT and ELEMENTS are a different kettle of fish for the following reasons 1) They both depend on the datatype used to interpret the message. 2) They may either or both be expensive to calculate. Since PROBE does not take a datatype (and I believe this is *correct*), it cannot calculate either of these and store the result in the status object. Therefore we *cannot* allow direct access to a structure field for these values. This seems correct if we're trying to achieve the second objective too. If the user can directly access these values they must always have been calculated and stored, BUT we believe calculating them may be expensive, therefore it should be postponed until we know the user wants the value. Note that this argument applies to RECEIVE as well. It is NOT solely PROBE which leads to this result. Conclusion ========== 1) GET_TAG and GET_SOURCE could be replaced by named fields (or fixed offsets) in the status object. 2) GET_COUNT and GET_ELEMENTS should both be "functions", and not status object fields. 3) GET_COUNT and GET_ELEMENTS both need the datatype argument. (This is clear if the op was PROBE: there is no datatype lying around to use. It is also the case for RECEIVE, because we want to allow implementation freedom, and the implementation will probably only store one of the two values (or maybe even a more fundamental value such as byte count), and evaluate the other as needed) Extreme implementations ======================= 1) Do it all in receive Receive always evaluates COUNT and ELEMENTS, stashes reference to datatype in status object as well. Probe stashes byte count. GET_COUNT and GET_TAG check datatype given matches one in status object and return value, else (probe case) calculate required value from byte count. First half can be done in a macro in C, second half will need function call. Costs : Calculating both values on every receive Benefits: The actual GET_COUNT will be quick after a receive Dangers: Need to be careful comparing stashed datatype with one given at GET time. User may have deleted and re-created datatypes in-between receive and GET. 2) Do nothing in receive Both receive and probe simply stash a byte count in the status object. GET_COUNT and GET_ELEMENTS both calculate their result from byte count. Costs : Calculating the value when needed Benefits: No work is done until required Dangers: None ? Intermediate implementations are possible where (say) COUNT is evaluated by receive but ELEMENTS is not (or the converse). In any case to get PROBE to work we *must* have the datatype in these routines, and have them as functions. Ultimate conclusion =================== GET_COUNT and GET_ELEMENTS in the draft are fine ! GET_TAG and GET_SOURCE could go away if we want to expose something about the status object. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 12 19:09:25 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id TAA20147; Wed, 12 Jan 1994 19:09:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA02912; Wed, 12 Jan 1994 19:09:36 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 12 Jan 1994 19:09:35 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA02904; Wed, 12 Jan 1994 19:09:31 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1027; Wed, 12 Jan 94 19:09:33 EST Received: from YKTVMV by watson.vnet.ibm.com with "VAGENT.V1.0" id 5697; Wed, 12 Jan 1994 19:09:32 EST Received: from snir.watson.ibm.com by yktvmv.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 12 Jan 94 19:09:30 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA29194; Wed, 12 Jan 1994 19:09:25 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9401130009.AA29194@snir.watson.ibm.com> To: mpi-pt2pt@CS.UTK.EDU Cc: frankeh@watson.ibm.com, johns@ccsf.caltech.edu, edlesohn@snir.watson.ibm.com Subject: point to point section draft Reply-To: snir@watson.ibm.com Date: Wed, 12 Jan 94 19:09:22 -0500 -:) -:) -:) -:) -:) Next message will contain the psotscript for the point to point section as is now. I have included most (but not all!) corrections and suggestions that were sent to me. This is the last version I shall circulate before the Europe meeting next week. I made changes to correct mistakes and make clarifications/interpretations. I have indicated in discussions possible alternatives. Significant changes and points for discussions are indicated by margin signs. Note, in particular: pg 7 -- summary of discussion that went over the net about MPI_GET_COUNT and related functions. pg 11 -- two proposals on how to deal with FORTRAN characters. pg 21 and elsewhere -- argument STATUS is always required in calls to MPI_WAIT and related calls (because CANCEL may use it). also, MPI_WAIT, MPI_TEST are changed to well-behave when passed a null request. pg 27 -- a proposal of Salmon and Edelsohn to replace array of request in WAITALL and related calls by an opaque object that represents a set of requests. Elegance and performance, at the expense of an additional object and additional implementation work. pg 36 -- some clarification on the behavior of WAIT, WAITALL when passed a persistent handle with no active communication. I droped the last section which was no part of MPI. From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 12 19:16:31 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id TAA20159; Wed, 12 Jan 1994 19:16:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA03720; Wed, 12 Jan 1994 19:16:25 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 12 Jan 1994 19:16:18 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA03660; Wed, 12 Jan 1994 19:15:37 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1123; Wed, 12 Jan 94 19:15:37 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 6412; Wed, 12 Jan 1994 19:15:06 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Wed, 12 Jan 94 19:14:52 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA27734; Wed, 12 Jan 1994 19:14:45 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9401130014.AA27734@snir.watson.ibm.com> To: mpi-pt2pt@CS.UTK.EDU Cc: frankeh@watson.ibm.com, johns@ccsf.caltech.edu, c1dje@watson.ibm.com Subject: mpi pt2pt section draft postscript Reply-To: snir@watson.ibm.com Date: Wed, 12 Jan 94 19:14:44 -0500 -:) -:) -:) -:) -:) %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: pt2pt941.dvi %%Pages: 62 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 5 118 df<3C7EE2C0C0C0E27E3C07097F880A>99 D110 D<3C7EE7C3C3C3E77E3C08097F880B>I<303030FCFC30303030323E1C070C808B08> 116 DI E /Fc 2 50 df<1E007F806180C0C0C0C0C0C0C0 C0C0C0C0C0C0C061807F801E000A0D7E8C0E>48 D<18F8F81818181818181818FFFF080D7D8C0E >I E /Fd 1 106 df<04040000000038FCD8181832333E3C080F7E8E0B>105 D E /Fe 5 118 df<0F803FE078607000E000E000E000E000E000702078603FE00F800B0D7F8C 0E>99 D110 D<0F003FC070E070E0E070E070E070E070E07070E079E03FC00F000C0D7F8C0F>I<38383838FF FF3838383838383838393F1E08117F900B>116 DI E /Ff 2 81 df<00001C00003C0000F80001E00003C00007 80000F00000E00001E00003C00003C00003C000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 0000780000780000780000780000780000F00000F00000F00001E00001E00003C0000380000700 000E00001C0000780000E00000E000007800001C00000E000007000003800003C00001E00001E0 0000F00000F00000F0000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 00007800007800003C00003C00003C00001E00000E00000F000007800003C00001E00000F80000 3C00001C167C7B8121>40 D80 D E /Fg 4 111 df<60F0F07030303060E040040A7E830A>59 D<030003800300000000000000 00001C003E006600C600CC000C000C0018001980318033003F001C00091480930C>105 D<0060007000600000000000000000038007C00CE018C018C000C000C001800180018001800300 0300030003006600EE00FC0078000C1A81930E>I<387C007CFE006F8600CF0600CE06000E0600 0C06001C0C00180CC01818C0181980301F80300E00120D808C15>110 D E /Fh 2 4 df0 D<040006000600C660F7E03F801F003F80F7E0C6 600600060004000B0D7E8D11>3 D E /Fi 27 123 df<007E01FE07800E001E003C003C007800 7FF87FF8F000F000F00070007000700038001C180FF807E00F147E9312>15 D<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C181838306040060E7C840D>I62 D<0001FC000007FF0000 1F07C0003C03E000F001E001E000F003C000F007C00078078000780F0000781F0000781E000078 3E0000783E0000787C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001E0F80003 E0F80003C0F80007C07800078078000F007C001F003C003E003E007C001E00F0000F83E00007FF 800001FC00001D217E9F23>79 D<0FFFFFFC1FFFFFF81E03C0781803C0383803C0183007801830 0780186007801860078030C00F0030C00F0030000F0000000F0000001E0000001E0000001E0000 001E0000003C0000003C0000003C0000003C00000078000000780000007800000078000000F000 0000F0000000F0000001F000007FFFC0007FFFC0001E1F7F9E1B>84 D<00F18003FDC0078F800E 07801C07803C07803C0700780700780700780700F00E00F00E00F00E00F00E18F01C30F03C3070 7C3078FC603FCFC00F078015147E9318>97 D<07803F803F000700070007000E000E000E000E00 1C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01EE01EE03CE03CE038607071 E03FC01F0010207E9F14>I<0000780003F80003F00000700000700000700000E00000E00000E0 0000E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700 F00E00F00E00F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015207E9F18>100 D<007C01FE07870E031E033C033C06781C7FF87FC0F000F000F000F000700070027007381E1FF8 07E010147E9315>I<00007C0000FE00019E00039E00030C000700000700000700000700000E00 000E00000E0000FFF001FFF0000E00001C00001C00001C00001C00001C00003800003800003800 00380000380000700000700000700000700000700000E00000E00000E00000E00000C00001C000 31C000798000F300007E00003C000017297E9F16>I<001E30007FB800F1F001C0F00380F00780 F00780E00F00E00F00E00F00E01E01C01E01C01E01C01E01C01E03801E07800E0F800F1F8007FF 0001E700000700000700000E00000E00300E00781C00F078007FF0003FC000151D809316>I<00 7000F000F00060000000000000000000000000000007001F8039C031C061C061C0C38003800380 070007000E000E000E181C301C301C601CE00FC007000D1F7F9E10>105 D<0000C00001E00001E00001C0000000000000000000000000000000000000000000003E00007F 0000C380018380030380030380060700000700000700000700000E00000E00000E00000E00001C 00001C00001C00001C0000380000380000380000380000700000700030700078E000F3C0007F80 003E00001328819E13>I<01E0000FE0000FC00001C00001C00001C00003800003800003800003 80000700000700000701C00707E00E0C300E18700E30F00E60F01CC0E01F80001F80001FE00038 F000383800383800381C307038607038607038407018C0E01F80600F0014207E9F18>I<03C01F C01F8003800380038007000700070007000E000E000E000E001C001C001C001C00380038003800 38007000700070007180E300E300E300E6007E003C000A207E9F0E>I<0F03E03E001F8FF0FF00 31DC39C38061F03B038061E01E01C061C01C01C0C3C03C03800380380380038038038003803803 800700700700070070070007007007000700700E0C0E00E00E180E00E01C180E00E01C300E00E0 0C701C01C00FE00C00C0078026147F9328>I<0F07C0001F8FE00031D8700061F0380061E03800 61C03800C3C070000380700003807000038070000700E0000700E0000700E0000701C1800E01C3 000E0383000E0386000E018E001C01FC000C00F00019147F931B>I<007C0001FF000383800F01 C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E 003838001FF00007C00013147E9316>I<03C1E007E7F80C7E3C187C1C18781E18701E30E01E00 E01E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E0007 00000700000E00000E00000E00000E00001C0000FFC000FFC000171D819317>I<0F0F001F9F80 31F1C061E1C061E3C061C3C0C3C3800380000380000380000700000700000700000700000E0000 0E00000E00000E00001C00000C000012147F9315>114 D<007C01FE0387070F070F070E0F0007 8007F803FC01FE001E000F700FF00EF00EE01C70383FF01FC010147E9315>I<00C000E001C001 C001C001C003800380FFF8FFF807000700070007000E000E000E000E001C001C001C001C183830 3830386038E01FC00F000D1C7F9B10>I<0780301FC03838C07030E07060E07061C070C1C0E003 80E00380E00380E00701C00701C00701C00701C3070386070386070786070F8C03F9F801F0F018 147F931A>I<07C3C00FE7E0187C703038706038F06038F0C070E000700000700000700000E000 00E00000E00070E030F1C060F1C060E1C0C0E3E1807E7F003C3E0014147E931A>120 D<0780301FC03838C07030E07060E07061C070C1C0E00380E00380E00380E00701C00701C00701 C00701C0070380070380070780070F8003FF0001F700000700000700000E003C0C003C1C003838 0030F0001FE0000F8000151D7F9316>I<01E06003F0C007FFC00E1F800C0300000600000C0000 180000300000600000C0000180000300000600C00C01801803803F87007FFE0061FC00C0700013 147E9315>I E /Fj 4 62 df<0FC01FE0387070386018E01CE01CE01CE01CE01CE01CE01CE01C E01C6018703838701FE00FC00E137F9211>48 D<06001E00FE00EE000E000E000E000E000E000E 000E000E000E000E000E000E000E00FFE0FFE00B137D9211>I<1F003FC061E0C0E0E070E07000 70007000E000C001C0030006000C30183030707FE0FFE0FFE00C137E9211>I<7FFFE0FFFFF000 0000000000000000000000000000000000FFFFF07FFFE0140A7E8B19>61 D E /Fk 2 42 df<0001C0000000000380000000000380000000000700000000000E0000000000 0E00000000001FFFFFFF80007FFFFFFF8000FFFFFFFF8001E0000000000780000000003F000000 0000FC00000000003F000000000007800000000001E00000000000FFFFFFFF80007FFFFFFF8000 1FFFFFFF80000E00000000000E000000000007000000000003800000000003800000000001C000 000029197D9630>40 D<000001C00000000000E00000000000E000000000007000000000003800 00000000380000FFFFFFFC0000FFFFFFFF0000FFFFFFFF800000000003C00000000000F0000000 00007E00000000001F80000000007E0000000000F00000000003C000FFFFFFFF8000FFFFFFFF00 00FFFFFFFC0000000000380000000000380000000000700000000000E00000000000E000000000 01C0000029197D9630>I E /Fl 15 118 df<387CFEFEFE7C3800000000387CFEFEFE7C380712 7D910D>58 D68 D73 D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07C0FC07 C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F007C1F00 7C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00007E01803E03801F07000FFE0003F800 11127E9115>99 D<01FC000FFF001F0F803E07C07C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC 0000FC00007C00007E00603E00C01F81C00FFF0001FC0013127F9116>101 D<1E003F003F007F003F003F001E0000000000000000000000FF00FF001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D108 DII<01FC000FFF801F07 C03E03E07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F87C01F07C01F03E03E01F07 C00FFF8001FC0015127F9118>II<1FD83FF87038E018E018F000FF807FE07FF0 1FF807FC007CC01CC01CE01CF038FFF0CFC00E127E9113>115 D<030003000300070007000F00 0F003F00FFFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F9807F003 E00E1A7F9913>II E /Fm 2 42 df<0000200000000000700000000000F00000000001E00000000001C00000000003 C0000000000780000000000FFFFFFFF0001FFFFFFFF8007FFFFFFFF800F80000000003E0000000 001FC000000000FF0000000000FE00000000001F800000000007E00000000001F0000000000078 00000000003FFFFFFFF8001FFFFFFFF8000FFFFFFFF00007800000000003C00000000001C00000 000001E00000000000F00000000000700000000000200000002D1D7D9934>40 D<0000002000000000007000000000007800000000003C00000000001C00000000001E00000000 000F00007FFFFFFF8000FFFFFFFFC000FFFFFFFFF00000000000F800000000003E00000000001F C00000000007F80000000003F8000000000FC0000000003F00000000007C0000000000F000FFFF FFFFE000FFFFFFFFC0007FFFFFFF80000000000F00000000001E00000000001C00000000003C00 000000007800000000007000000000002000002D1D7D9934>I E /Fn 8 104 df0 D<70F8F8F87005057C8D0D>I<018001C001800180 C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C0018010147D9417>3 D<07E01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E01012 7D9317>15 D<000000C0000003C000000F0000003C000000F0000003C000000F0000001C000000 78000001E00000078000001E00000078000000E0000000780000001E0000000780000001E00000 00780000001C0000000F00000003C0000000F00000003C0000000F00000003C0000000C0000000 00000000000000000000000000000000000000000000000000FFFFFFC0FFFFFFC01A247C9C23> 20 D91 D<001F003F00F800E001C001C001C001C001C001C001C0 01C001C001C001C001C001C001C001C003800700FE00F800FE000700038001C001C001C001C001 C001C001C001C001C001C001C001C001C001C001C000E000F8003F001F102D7DA117>102 DI E /Fo 53 122 df<01F1C003F1C007F1C00701 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFF1C0FFF1C0FFF1C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0121D 809C16>13 D<00F0000001F8000003FC0000039C0000070E0000070E0000070E0000070E000007 0E0000071E0000071C0000073C00000778000003F0040003E0070007C00F000FC00E001FC00E00 3DE01E0078E01C0070F03C00E0703800E0387800E03CF000E01FE000E00FC00070078000781FE0 C03FFFFFC01FF8FFC007E03F001A1F7E9D1F>38 D<00E001C0038007000E000E001C001C003800 380038007000700070007000E000E000E000E000E000E000E000E000E000E000E000E000700070 00700070003800380038001C001C000E000E000700038001C000E00B2A7E9E10>40 DI<00060000000600000006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000FFFFFFE0FFFFFFE000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 0000060000000600001B1C7E9720>43 D<7878787838307060E005097D830C>III58 D<7FFFFFC0FFFFFFE0000000000000000000000000000000000000000000000000000000 0000000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<001C0000003E0000003E0000002E00000067 00000067000000E7800000C7800000C3800001C3C0000183C0000181C0000381E0000381E00007 00F0000700F0000600F0000E0078000FFFF8000FFFF8001FFFFC001C003C0018003C0038001E00 38001E0070001F0070000F0070000F00E0000780191D7F9C1C>65 DI<003FC000FFF003FFF007C0700F80001E00003E00003C0000780000780000780000F0 0000F00000F00000F00000F00000F00000F00000F00000F000007800007800007800003C00003E 00001E00000F800807C07803FFF800FFF0003F80151F7D9D1B>IIII<003F80 01FFF003FFF807C0F80F00181E00003E00003C0000780000780000780000F00000F00000F00000 F00000F00000F00000F007F8F007F8F007F87800387800387800383C00383E00381E00380F0038 07C07803FFF801FFF0003F80151F7D9D1C>III76 DII<003F000001FFE00003FFF00007C0F8000F807C001E001E003E001F003C000F0078000780 7800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003 C0F80007C078000780780007807C000F803C000F003E001F001F003E000F807C0007C0F80003FF F00001FFE000003F00001A1F7E9D1F>II82 D<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F0 0000F800007C00007F00003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C000 03C00003C00003C0C00780E00780FC1F007FFE001FFC0007F000121F7E9D17>II< F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070 F00070F00070F00070F00070F00070F00070F00070F00070F00070F000F07800E07801E03C03C0 1F07800FFF8007FE0001F800141E7C9C1D>III89 D<0FC03FF07FF87038401C00 1C001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 DI<07E00FF81FFC3C1C70047000 E000E000E000E000E000E000700070043C1C1FFC0FF807E00E127E9112>I<000E000E000E000E 000E000E000E000E000E000E000E0F8E1FEE3FFE7C3E700E700EE00EE00EE00EE00EE00EE00EF0 0E701E7C3E3FFE1FEE0F8E0F1D7E9C15>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000E0 00E000700070043C1C3FFC1FF807E00E127E9112>I<00FC01FC03FC07000E000E000E000E000E 000E000E00FFE0FFE0FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E1D809C0D>I<03C3C00FFFC01FFFC01C3800381C00381C00381C00381C00381C001C3800 1FF8001FF0003BC0003800003800001FFC001FFF003FFF80700780E001C0E001C0E001C0F003C0 7C0F803FFF001FFE0007F800121B7F9115>III107 DIII<03F0000FFC001FFE003C0F00780780700380E001C0E001C0E001C0E001C0E001C0F003C0 7003807807803C0F001FFE000FFC0003F00012127F9115>II114 D<1FC03FF07FF0F030E000E000F0007F003FC01FE000F00038003880 38F078FFF07FE01FC00D127F9110>I<1C001C001C001C001C001C00FFE0FFE0FFE01C001C001C 001C001C001C001C001C001C001C001C001C201FF00FF007C00C187F970F>II I<7003807807003C0E001C1C000E1C0007380003F00001E00001C00001E00003F0000738000E18 000E1C001C0E00380700700380F003C01212809113>120 DI E /Fp 26 122 df<7FF0FFE0FFE00C037D8A10>45 D<70F8F8F0E005057B840E>I< 000007000000070000000F0000000F0000001F0000003F0000003F0000006F0000006F000000CF 000000CF0000018F0000038F0000030F0000060F0000060F00000C0F80000C0780001807800018 0780003FFF80007FFF800060078000C0078000C00780018007800180078003000780070007800F 0007807FC07FF8FFC07FF81D207E9F22>65 D<01FFFFFE01FFFFFC001E003C001E001C001E001C 003C000C003C000C003C000C003C001800780C1800780C1800780C0000781C0000F0380000FFF8 0000FFF80000F0380001E0300001E0300001E0303001E0306003C0006003C0006003C000C003C0 01C007800180078003800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69 D<01FFFF0001FFFFC0001E01E0001E00F0001E0078003C0078003C0078003C0078003C00780078 00F0007800F0007801E0007803C000F00F0000FFFE0000FFF80000F03C0001E01E0001E00E0001 E00F0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C1807803C1807803C30 FFF81E30FFF00FE0000007C01D207D9E21>82 D<00F18003FDC0078F800E07801C07803C07803C 0700780700780700780700F00E00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F 078014147C9317>97 D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F 1E3E0E3C0F380F700F700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207B9F 15>I<007E0001FF000383800F07801E07801C07003C0200780000780000780000F00000F00000 F00000F00000F00000700200700700381E001FF80007E00011147C9315>I<0000780003F80003 F00000700000700000700000E00000E00000E00000E00001C00001C000F1C003FDC0078F800E07 801C07803C07803C0700780700780700780700F00E00F00E00F00E00F00E30F01C60F03C60707C 6078FCC03FCFC00F078015207C9F17>I<007C0001FF000783000F01801E01803C01803C030078 0E007FFC007FE000F00000F00000F00000F000007000007002007807003C1E001FF80007E00011 147C9315>I<0000F80001FC0003BC00033C000718000700000700000E00000E00000E00000E00 000E0001FFE001FFE0001C00001C00001C00003800003800003800003800003800007000007000 00700000700000700000700000E00000E00000E00000E00001C00001C00001C000018000338000 7B8000F300007E00003C00001629829F0E>I<003C6000FF7001E3E00381E00701E00F01E00F01 C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C0F001C1F001E3F000FFE0003CE 00000E00000E00001C00001C00301C00783800F0F0007FE0003F8000141D7E9315>I<006000F0 00F000E000000000000000000000000000000F001F80318031C063806380C3800700070007000E 000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>105 D<01E0000FE0000FC00001C0 0001C00001C0000380000380000380000380000700000700000703C00707E00E0C600E10E00E21 E00E61E01CC1C01F80001F00001FC00039E0003870003870003838607070C07070C07070C07031 80E03F00601E0013207D9F15>107 D<03C01FC01F8003800380038007000700070007000E000E 000E000E001C001C001C001C0038003800380038007000700070007180E300E300E300E6007E00 3C000A207C9F0C>I<1E07C0F8003F1FE1FC0033B8730E0063E076070063C03C07006380380700 C780780E000700700E000700700E000700700E000E00E01C000E00E01C000E00E01C000E00E038 601C01C038C01C01C038C01C01C071801C01C031803803803F001801801E0023147D9325>I<1E 07C03F1FE033B87063E07063C038638038C780700700700700700700700E00E00E00E00E00E00E 01C31C01C61C01C61C038C1C018C3801F81800F018147D931A>I<007C0001FF000383800F01C0 1E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E00 3838001FF00007C00013147C9317>I<03C1E007E7F8067E3C0C7C1C0C781E0C701E18E01E00E0 1E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E000700 000700000E00000E00000E00000E00001C0000FFC000FFC000171D809317>I<1E0F003F3F8033 F1C063C1C063C3C06383C0C783800700000700000700000E00000E00000E00000E00001C00001C 00001C00001C000038000018000012147D9313>114 D<00FC03FE07070E0F0E0F0E0E1E000F80 0FF007F803FC003E001E701EF01CF01CE03860703FE01F8010147D9313>I<018001C003800380 0380038007000700FFF0FFF00E000E000E000E001C001C001C001C003800380038003830706070 6070C071803F001E000C1C7C9B0F>I<0F00601F80703180E031C0E06380E06380E0C381C00701 C00701C00701C00E03800E03800E03800E038C0E07180E07180E0F180E1F3007F3F003E1E01614 7D9318>I<0F01C01F83C03183E031C1E06380E06380E0C380C00700C00700C00700C00E01800E 01800E01800E03000E03000E06000E06000F0C0007F80001E00013147D9315>I<0787800FCFC0 18F8E03070E06071E06071E0C0E1C000E00000E00000E00001C00001C00001C00071C060F380C0 F380C0E38180C7C3007CFE00387C0013147D9315>120 D<0F00601F80703180E031C0E06380E0 6380E0C381C00701C00701C00701C00E03800E03800E03800E03800E07000E07000E0F000E1F00 07FE0003EE00000E00000E00001C00781C0078380070700060E0003FC0001F0000141D7D9316> I E /Fq 45 122 df45 DI<00C001C007 C0FFC0FFC0FBC003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C003C0FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E001E06001E02001E00001E00001 E00001E00003C00003C0000780000780000F00001E00003C0000780000F00001E00001C0000380 000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001F FF003E0F003807807003C02003C02003C00003C00003C00003C0000780000780000F00001E0003 FC0003F80003FE00000F000007800003C00003C00001E00001E00001E00001E00001E08001E0C0 03C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00 006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00 3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00000F00 000F00000F00000F0015217FA018>I<3FFF803FFF803FFF803C00003C00003C00003C00003C00 003C00003C00003C00003C00003CF8003FFE003FFF003F0F803E07803C03C03803C00001E00001 E00001E00001E00001E00001E00001E04003C04003C0E003C07007807C1F003FFE000FFC0003F0 0013227EA018>I<007E0001FF0003FF0007C1000F00001E00001E00003C00003C000078000078 000078F800F3FE00F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F001E0F001E0F001E0F0 01E07001E07001E07801E07803C03803C03C03801C07801F0F000FFE0007FC0001F00013237EA1 18>II<01F00007FC000FFE 001E0F003C07803C07807803C07803C07803C07803C07803C03803803C07801E0F000F1E0007FC 0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F001E07803 C07803C03C07803E0F801FFF0007FC0001F00013237EA118>I<001F0000001F0000003F800000 3F8000003B8000007BC0000073C0000071C00000F1E00000F1E00000E0E00001E0F00001E0F000 01C0F00003C0780003C078000380780007803C0007803C0007003C000F001E000F001E000FFFFE 001FFFFF001FFFFF001C000F003C0007803C00078038000780780003C0780003C0700003C0F000 01E0F00001E0E00001E01B237EA220>65 D<000FF000007FFC0000FFFF0001F01F0003C0070007 8002000F0000001E0000003E0000003C0000003C000000780000007800000078000000F0000000 F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000078000000780000 00780000003C0000003C0000003E0000001E0000000F0000800780018003C0038001F00F8000FF FF00007FFC00000FF00019257DA31F>67 DII76 DII<001FC000007FF00001FFFC0003 F07E0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000F0 70000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F0000078F00000 78780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007C00F800F8007C0 1F0003F07E0001FFFC00007FF000001FC0001D257DA324>I82 D<00FF0003FFC007FFF00F83F01E00F03C0020 3C00007800007800007800007800007800007C00003C00003F00001FC0000FFC0007FF0001FF80 003FC00007E00001F00000F00000F8000078000078000078000078000078400078E000F0F000F0 F801E07F07C03FFF800FFF0001FC0015257EA31B>III<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780F F00FF00FF00FF80F7C3F7FFF3FFF1F8F10167E9517>97 DI<01FC0007FF000FFF801F07803C01807800007800 00700000F00000F00000F00000F00000F00000F000007800007800007800403C00C01F07C00FFF C007FF8001FC0012167E9516>I<0003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003 E3C012237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0 FFFFC0F00000F00000F000007000007800007800003C00801F07800FFF8007FF0001F80012167E 9516>I<003F00FF01FF03C1038007800780078007800780078007800780FFF8FFF8FFF8078007 8007800780078007800780078007800780078007800780078007800780078007800780102380A2 0F>I<01F0F807FFF80FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F00 1FFE001FFC0039F0003800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078 F00078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>I105 D<00F000F000F000F000000000000000000000000000000000000000F000F000F000F000F000F0 00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000 F000F080F0E3E0FFE07FC01F000C2D83A20D>IIIII<01FC0007FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F0 0078F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F95 18>II 114 D<07F01FFE3FFF3C0F7803780078007C003F003FF01FF80FFC01FE001F000F000F400FE00F F81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F00 0F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12>I< F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF83F7F FF7FEF1F8F10167C9519>III<7801F07C01E03E03C01E07C00F0780 078F0007DE0003FC0001FC0000F80000700000F80001FC0003DC00039E00078F000F07801E0780 1E03C03C01E07800F0F800F81516809516>II E /Fr 80 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F870000000000070F8F8 F870051C779B18>33 D<03C00007E0000FF0001E78001C38001C38001C38001C38001C7BF01CF3 F01EE3F00FE7800FC7000F87000F0F001F0E003F8E007B9E0073DC00F1DC00E1FC00E0F800E078 70F0F87071FE707FFFF03FCFE01F03C0141C7F9B18>38 D<007000F001F003C007800F001E001C 003C003800780070007000F000E000E000E000E000E000E000E000E000F0007000700078003800 3C001C001E000F00078003C001F000F000700C24799F18>40 D<6000F00078003C001E000F0007 80038003C001C001E000E000E000F00070007000700070007000700070007000F000E000E001E0 01C003C0038007800F001E003C007800F00060000C247C9F18>I<01C00001C00001C00001C000 C1C180F1C780F9CF807FFF001FFC0007F00007F0001FFC007FFF00F9CF80F1C780C1C18001C000 01C00001C00001C00011147D9718>I<00600000F00000F00000F00000F00000F00000F00000F0 007FFFC0FFFFE0FFFFE07FFFC000F00000F00000F00000F00000F00000F00000F0000060001314 7E9718>I<3C7E7F7F7F3F0F0E1E7CF870080C788518>I<7FFF00FFFF80FFFF807FFF0011047D8F 18>I<78FCFCFCFC780606778518>I<000300000780000F80000F80000F00001F00001F00003E00 003E00003C00007C00007C0000F80000F80000F00001F00001F00003E00003E00007C00007C000 0780000F80000F80001F00001F00001E00003E00003E00007C00007C0000780000F80000F80000 F0000060000011247D9F18>I<01F00007FC000FFE001F1F001C07003803807803C07001C07001 C0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803 C03803801C07001F1F000FFE0007FC0001F000131C7E9B18>I<01800380078007800F803F80FF 80FB80638003800380038003800380038003800380038003800380038003800380038003807FFC FFFE7FFC0F1C7B9B18>I<07F8001FFE003FFF007C0F807003C0F001E0F000E0F000E0F000E000 00E00000E00001E00001C00003C0000780000F00001E00003C0000780000F00003E00007C0000F 80001E00E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007C0F807803C0 7801C03001C00001C00003C0000780000F8003FF0003FE0003FF00000F800003C00001C00001E0 0000E00000E0F000E0F001E0F001C0F003C07C0F803FFF001FFE0007F800131C7E9B18>I<001F 00003F00007F0000770000F70001E70001C70003C7000787000707000E07001E07003C07003807 00780700F00700FFFFF8FFFFF8FFFFF8000700000700000700000700000700000700007FF000FF F8007FF0151C7F9B18>I<1FFF803FFF803FFF8038000038000038000038000038000038000038 00003800003BFC003FFE003FFF003E07801803C00001E00000E00000E06000E0F000E0F001E0E0 01C0F003C07C0F803FFF001FFE0007F800131C7E9B18>I<007E0001FF0007FF800FC3C01F03C0 3C03C0380180780000700000700000F3F800EFFE00FFFF00FE0F80F803C0F001C0F001E0F000E0 F000E0F000E07000E07001E07801C03C03C01E0F800FFF0007FE0003F800131C7E9B18>II<03F8000FFC001FFE003E0F00780780F003C0E001C0E001C0E0 01E0E001E0E001E0F001E07803E03E0FE01FFFE00FFEE003F8E00001E00001C00001C00003C030 0380780780780F00783E003FFC001FF8000FE000131C7E9B18>57 D<78FCFCFCFC780000000000 00000078FCFCFCFC780614779318>I<3C7E7E7E7E3C0000000000000000387C7E7E7E3E0E1E3C 78F060071A789318>I<000300000F80001F80003F0000FE0001FC0003F00007E0001FC0003F80 007E0000FC0000FC00007E00003F80001FC00007E00003F00001FC0000FE00003F00001F80000F 8000030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FF FFE0FFFFE07FFFC0130C7E9318>I<600000F80000FC00007E00003F80001FC00007E00003F000 01FC0000FE00003F00001F80001F80003F0000FE0001FC0003F00007E0001FC0003F80007E0000 FC0000F8000060000011187D9918>I<00700000F80000F80000D80000D80001DC0001DC0001DC 00018C00038E00038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF 800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<01FCE003FEE007FFE00F07E01E03E03C01E07800E07000E07000E0F00000E00000E000 00E00000E00000E00000E00000E00000E00000F000007000E07000E07800E03C01E01E01C00F07 C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C 00E01C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C 00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>III<01F9C007FFC00FFFC01F0FC01C03C03C03C07801C070 01C07001C0F00000E00000E00000E00000E00000E00000E00FF0E01FF0E00FF0F001C07001C070 03C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C0141C7E9B18>I<7F07F0FF8FF87F07F0 1C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C0 1C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F 9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C07001C0F001C 1E001C3C001C38001C78001CF0001DF0001DF8001FF8001FBC001F1C001E1E001E0E001C0F001C 07001C07801C03801C03C01C01C07F03F0FF87F87F03F0151C7F9B18>75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E00707FFF F0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D81C0 1D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C01C39C0 1C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18> I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F007FFF 003FFE000FF800111C7D9B18>II<0FF8003FFE007FFF00780F00 700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00001C00001E00 000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03801C03C01C01 C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03801C03801C03 801C03801C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07F3801FFF803F FF807C1F80700780F00380E00380E00380E00000F000007800003F00001FF0000FFE0001FF0000 1F800003C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FFFF00E7FC0013 1C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E07038007000007000007000007000 007000007000007000007000007000007000007000007000007000007000007000007000007000 00700007FF0007FF0007FF00151C7F9B18>IIII<7F8FE07F9FE07F8FE00E07000F0700070E00078E 00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00038E 00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>II<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C00 00780000700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E0 7000E0FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18 >95 D<1FE0003FF8007FFC00783E00300F0000070000070001FF000FFF003FFF007F0700780700 F00700E00700E00700F00F00783F007FFFF03FFBF00FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000EFF800FFFC00FE3E00F80 F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFF C00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300780000700000F00000E0 0000E00000E00000E00000F000007000007801C03C01C03F07C01FFF8007FF0001FC0012147D93 18>I<001F80003F80001F8000038000038000038000038000038003F3800FFB801FFF803E1F80 780F80700780F00780E00380E00380E00380E00380E00380E00780F00780700780780F803E3F80 1FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F00780780700380F003C0E001 C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01FFF8007FF0001FC001214 7D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C0007FFFC0FFFFC0FFFFC001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F303C0F00380700380700 3807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00001FFF001FFFC03FFFE0 7801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF8003FE00151F7F9318> I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000EFF800FFFC00FE1E00F80 E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3 FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C00003800000000000000000000000 00007FC000FFC0007FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C00 3800000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C003C6038F078FFF07FE03FC00E277E9C18>II<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFFFC001F1F1C 001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C 1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E3F00FEFF80 7FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001 C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE 0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00380E00380E 00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00000E00000E 00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<03F3800FFB801FFF80 3E1F80780F80700780F00780E00380E00380E00380E00380E00380E00380F00780700780780F80 3E1F801FFF800FFB8003F380000380000380000380000380000380000380000380003FF8003FF8 003FF8151E7E9318>I<7F87E0FF9FF87FBFF803FC7803F03003E00003C00003C00003C0000380 000380000380000380000380000380000380000380007FFE00FFFF007FFE0015147F9318>I<0F F7003FFF007FFF00F81F00E00700E00700F007007C00007FF0001FFC0007FE00001F00600780E0 0380F00380F00780FC0F00FFFF00FFFE00E7F80011147D9318>I<018000038000038000038000 0380007FFFC0FFFFC0FFFFC0038000038000038000038000038000038000038000038000038000 0380400380E00380E00381E003C3C001FFC000FF80007E0013197F9818>I<7E07E0FE0FE07E07 E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01 E00F07E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E 0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000 700015147F9318>II<7F8F F07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F80001DC00039E 00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E 03800E0380070380070700070700038700038600038E0001CE0001CE0000CC0000CC0000DC0000 780000780000780000700000700000700000F00000E00079E0007BC0007F80003F00001E000015 1E7F9318>I<3FFFF07FFFF07FFFF07001E07003C0700780000F00001E00007C0000F80001F000 03E0000780000F00701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0003F E0007FE000FC0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0001E0007FC000FF8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000FC00007FE0003FE00007E013247E9F18>I<7C0000FF8000FF C00007E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 F000007FC0003FE0003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00007E000FFC000FF80007C000013247E9F18>125 D E /Fs 69 124 df<003F0F0000FFBF8003C3F3C00703E3C00703C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 7F87FC007F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C00703C00E01800E00000E00 000E00000E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D809C17>I<003FC000FFC003 C3C00703C00701C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0FFFFC00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF87F CFF8151D809C17>I<003F03F00000FFCFF80003C0FC1C000701F03C000701F03C000E00E01800 0E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC00FFFFFFFC000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF807FC7FCFF80211D 809C23>I<7070F8F8FCFCFCFC7C7C0C0C0C0C0C0C181818183030606040400E0D7F9C15>34 D<00C00180030006000E000C001C0018003800300030007000700060006000E000E000E000E000 E000E000E000E000E000E000E000E000600060007000700030003000380018001C000C000E0006 000300018000C00A2A7D9E10>40 DI<018001C0018001806186F9 9F7DBE1FF807E007E01FF87DBEF99F61860180018001C0018010127E9E15>I<70F0F8F8781818 183030706040050D7D840C>44 DI<70F8F8F87005057D840C>I<00 030003000700060006000E000C001C0018001800380030003000700060006000E000C000C001C0 01800380030003000700060006000E000C000C001C001800180038003000700060006000E000C0 00C00010297E9E15>I<030007003F00FF00C70007000700070007000700070007000700070007 0007000700070007000700070007000700070007000700FFF8FFF80D1C7C9B15>49 D<07C01FF03878603C601EF01EF80FF80FF80F700F000F000E001E001C003C0078007000E001C0 038007000E030C03180330067FFEFFFEFFFE101C7E9B15>I<07E01FF03838301C781E781E781E 381E001E003C0038007007E007E00038001C001E000E000F000F700FF80FF80FF80EF01E601C38 381FF007C0101D7E9B15>I<300C3FFC3FF83FE030003000300030003000300033E037F03C3838 1C301E300E000F000F000F000F700FF00FF00FE00E601E601C38781FF007C0101D7E9B15>53 D<70F8F8F870000000000000000070F8F8F87005127D910C>58 D<70F8F8F87000000000000000 0070F0F8F8781818183030706040051A7D910C>I<7FFFFFC0FFFFFFE000000000000000000000 00000000000000000000000000000000000000000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<0FE03FF8703C601CF01EF01EF01E001E003C007800E001C00180038003000300030003000300 0200000000000000000007000F800F800F8007000F1D7E9C14>63 D<00060000000F0000000F00 00000F0000001F8000001F8000001F8000001F80000033C0000033C0000033C0000061E0000061 E0000061E00000C0F00000C0F00000C0F000018078000180780001FFF80003FFFC0003003C0003 003C0006001E0006001E0006001E001F001F00FFC0FFF0FFC0FFF01C1D7F9C1F>65 DI<001F808000FFE18003F0338007801B800F000F801E0007801C 0003803C000380780003807800018070000180F0000180F0000000F0000000F0000000F0000000 F0000000F0000000F00000007000018078000180780001803C0001801C0003001E0003000F0006 0007800C0003F0380000FFF000001F8000191E7E9C1E>III< FFFFF8FFFFF80F00780F00380F00180F001C0F000C0F000C0F030C0F030C0F03000F03000F0700 0FFF000FFF000F07000F03000F03000F03000F03000F00000F00000F00000F00000F00000F0000 FFF800FFF800161C7E9B1B>I<001F808000FFE18003F0338007801B800F000F801E0007801C00 03803C000380780003807800018070000180F0000180F0000000F0000000F0000000F0000000F0 000000F000FFF0F000FFF07000078078000780780007803C0007801C0007801E0007800F000780 07800F8003F0398000FFF080001FC0001C1E7E9C21>III75 DIII<003F800000FFE00003E0F80007803C000E000E00 1E000F003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F00001 E0F00001E0F00001E0F00001E0F00001E0780003C0780003C0780003C03C0007803C0007801E00 0F000F001E0007803C0003E0F80000FFE000003F80001B1E7E9C20>II82 D<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F000 00F000007C00007FC0003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001 C0C001C0E00180E00380F00300FC0E00CFFC0083F800121E7E9C17>I<7FFFFFC07FFFFFC0780F 03C0700F01C0600F00C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F000003FFFC0003FFFC001B1C7F9B1E>II87 D89 D<08081818303060606060C0C0C0C0C0C0F8F8FCFCFCFC7C7C38380E0D7B9C15>92 D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE001FFE003E0E00780E00F00E00F00E 60F00E60F01E60783E603FFFC01F878013127F9115>97 D I<03F00FF81E3C383C78187000F000F000F000F000F000F000780078063C061E0C0FF803E00F12 7F9112>I<001F80001F8000038000038000038000038000038000038000038000038000038003 E3800FFB801E0F80380780780380700380F00380F00380F00380F00380F00380F0038070038078 03803807801E1F800FFBF007E3F0141D7F9C17>I<03E00FF01C38381C781E700EFFFEFFFEF000 F000F000F000700078063C061E0C0FF803E00F127F9112>I<007801FC039E071E0E0C0E000E00 0E000E000E000E00FFE0FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E 007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3DC0381C00781E00781E00781E00781E00 381C003C3C003FF00037E0007000007000003000003FFC001FFF003FFF80700780E001C0E001C0 E001C0E001C07003803C0F001FFE0007F800121C7F9215>II<18003C007C003C001800000000000000000000000000FC00FC001C001C001C001C001C001C 001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<01C003E003E003E001C000 00000000000000000000000FE00FE000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E060E0F1C0F1C07F803E000B25839C0D>IIIII<03F000 0FFC001E1E00380700780780700380F003C0F003C0F003C0F003C0F003C0F003C0700380780780 3807001E1E000FFC0003F00012127F9115>II<03E1800FF9801E1F803C078078 0780780380F00380F00380F00380F00380F00380F003807803807807803C07801E1F800FFB8007 E380000380000380000380000380000380000380001FF0001FF0141A7F9116>II<1F90 3FF07070E030E030E030F8007F803FE00FF000F8C038C038E038E038F070DFE08FC00D127F9110 >I<0C000C000C000C000C001C001C003C00FFE0FFE01C001C001C001C001C001C001C001C001C 301C301C301C301C301E600FC007800C1A7F9910>IIII<7F8FF07F8FF00F07800706 00038E0001DC0001D80000F00000700000780000F80001DC00038E00030E000607000F0380FF8F F8FF8FF81512809116>II<7FFC7FFC7838707060F060E061C063C00380070C0F 0C0E0C1C1C3C1838187078FFF8FFF80E127F9112>II E /Ft 71 123 df<0000F000F8F001F8F003F8F00780000700000F00000F00000F00000F00000F 00000F00000F0000FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0142180A018>12 D<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0FFF8 F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00 F00F00F00F00F00F00F00F00F00F00F00F00F01420809F18>I<007000E001C00380078007000E 001E001E003C003C003C0078007800780078007000F000F000F000F000F000F000F000F000F000 F000F000F000700078007800780078003C003C003C001E001E000E0007000780038001C000E000 700C2E7EA112>40 DI<000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000060000 000600000006000000060000FFFFFFF0FFFFFFF000060000000600000006000000060000000600 000006000000060000000600000006000000060000000600000006000000060000000600000006 00001C207D9A23>43 D<787878781830306060E0050A7D830D>II< F0F0F0F004047C830D>I<03F00007F8001FFE001E1E003C0F00380700780780780780700380F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F0 03C07807807807807807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C003C003C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I<07F0000FFC001FFE 00383F00700F00600780E00780E003C04003C04003C00003C00003C00003C0000780000780000F 00000E00001C00003C0000780000E00001C0000380000700000E00001C0000380000700000FFFF C0FFFFC0FFFFC0121F7E9E17>I<03F0000FFC001FFE003C1F00780F0030078020078000078000 0780000780000F00000F00003E0003FC0003F80003FC00001E00000F000007800003800003C000 03C00003C00003C08003C0C003C0C00780700F807C1F003FFE000FFC0003F00012207E9E17>I< 003E00003E00005E00005E0000DE0001DE00019E00039E00039E00079E00071E000F1E000E1E00 1E1E003C1E003C1E00781E00781E00F01E00FFFFF0FFFFF0FFFFF0001E00001E00001E00001E00 001E00001E00001E00001E00141E7F9D17>I<7FFF007FFF007FFF007800007800007800007800 0078000078000078000079F0007FFC007FFE007F1F007C07007C07807807800003C00003C00003 C00003C00003C00003C00003C0400780600780F00F007C1E003FFC001FF80007E000121F7E9D17 >I<007C0001FE0003FE000782000F00001E00003C00003C000078000078000078F800F3FC00F7 FE00FE1F00FC0F80F80780F80780F003C0F003C0F003C0F003C0F003C07003C07803C07803C078 07803807803C0F001E1E000FFE0007F80001F00012207E9E17>I<03F0000FFC001FFE003E1F00 3C0F007807807807807807807807807807807807803C0F001E1E000FFC0007F8000FFC001F3E00 3C0F00780780780780F003C0F003C0F003C0F003C0F003C0F003C07807807C0F803E1F001FFE00 0FFC0003F00012207E9E17>56 D<03F00007F8000FFC001E1E003C0F00780700780780F00780F0 0380F003C0F003C0F003C0F003C0F003C0F003C07807C07807C07C0FC03E1FC01FFBC00FF3C007 C780000780000780000700000F00001E00201E00307C007FF8003FF0000FC00012207E9E17>I< F0F0F0F0000000000000000000000000F0F0F0F004147C930D>I61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1 E00000E1E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007 803C0007003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007803C000780 780007C0780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C00 00780000780000780000F00000F00000F00000F00000F00000F00000F00000F000007800007800 007800007C00003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D >IIII<00 1FE000FFF801FFFE03E03E07800E0F00001E00003E00003C00007C0000780000780000780000F0 0000F00000F00000F00000F00000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E 001E1E001E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E>III76 DII<003F000000FFC00003FFF00007E1F8000F807C001F003E001E001E003C000F00 3C000F00780007807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003 C0F00003C0F00003C0F00003C0F80007C07800078078000780780007803C000F003C000F001E00 1E001F003E000F807C0007E1F80003FFF00000FFC000003F00001A227DA021>II<003F000000FFC00003FFF00007E1F8000F807C00 1F003E001E001E003C000F003C000F00780007807800078078000780F00003C0F00003C0F00003 C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C07800078078000780781E 07803C0F0F003C0F8F001E079E001F03FE000F83FC0007E1F80003FFF00000FFF800003F780000 007C0000003E0000001E0000001F0000000F801A277DA021>II<01FC0007FF800FFFC01F03C03C00C03C0000780000780000780000 7800007800007C00003C00003F00001FE0000FFC0007FE0001FF00003F800007C00003C00003E0 0001E00001E00001E00001E00001E00001C0C003C0F007C0FC0F807FFF001FFE0003F80013227E A019>IIII I<780007807C000F003E001F001E001E000F003C000F807C000780780003C0F00003E1F00001F1 E00000F3C000007FC000007F8000003F0000001F0000001E0000003F0000007F8000007FC00000 F3C00001F1E00001E0F00003C0F80007C0780007803C000F003E001F001E001E000F003C000F80 7C0007C0780003C0F00003E01B207F9F1E>IIII93 D<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00F F00FF00FF83F7FFF3FEF1F8F10147E9316>97 DI<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E 0F1FFF0FFE03F010147E9314>I<00078000078000078000078000078000078000078000078000 078000078000078000078007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F0 0780F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C78011207E9F17>I< 03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000F00000 7000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01FE03FE078007000F 000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F000F00 0F000F000F000F000F000F000F000F20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00780F00 780F00780F00780F00780F003C1E003E3E001FFC003FF80033E0003000003800003FFE003FFF80 1FFFC03FFFE07803F0F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F9317> III<01E001E001E0 01E00000000000000000000000000000000001E001E001E001E001E001E001E001E001E001E001 E001E001E001E001E001E001E001E001E001E001E001E001E001E001E0C3C0FFC0FF803F000B29 839F0C>IIII< F1F8F3FCF7FEFC1FF80FF80FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF0 0F10147D9317>I<01F80007FE001FFF803F0FC03C03C07801E07801E0F000F0F000F0F000F0F0 00F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F80014147F9317>II<03C7800FF7801FFF803E1F807C0F80780780780780F00780F007 80F00780F00780F00780F00780F807807807807C0F803E1F801FFF800FF78007C7800007800007 80000780000780000780000780000780000780000780111D7E9317>II<07F0 1FFC3FFC780C7800780078007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E 147F9311>I<1E001E001E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E 001E001E001E001E001E001E201FF00FF007C00C1A7F9910>IIII<7801E07C03C03E 07801E0F000F0F00079E0003FC0003F80001F80000F00001F00001F80003FC00079E000F0F000E 0F001E07803C03C07801E0F801F01414809315>II<7FFF 7FFF7FFF003E003C007800F800F001E003E007C007800F001F001E003C007C00FFFFFFFFFFFF10 147F9314>I E /Fu 34 90 df<7FE07FE0FFC00B037E8A0F>45 D<3078F8787005057C840D>I< 007E0001FF0003C3800701C00E01C00E01E01C00E01C01E03C01E03C01E07801E07801E07801E0 7801E07801E0F003C0F003C0F003C0F003C0F003C0F00380F00780E00780E00700E00F00E00E00 701E00701C003878003FF0000FC000131F7C9D17>48 D<000C001C00FC0FF80F38003800380038 003800700070007000700070007000E000E000E000E000E000E001C001C001C001C001C001C003 C07FFEFFFE0F1E7C9D17>I<003F0000FFC001C1E00300F00600780600780F007C0F807C0F807C 0F00780600780000F80000F00001F00001E00003C0000780000F00001C0000380000700000C000 0180000300600600601C00C03800C07FFFC0FFFF80FFFF80161E7E9D17>I<007F0001FFC00383 E00701E00700F00F00F00F01F00F01F00001E00001E00003C0000780000F0000FE0000F800000E 000007000007800003C00003C00003C03007C07807C0F807C0F807C0F00F80C00F00601E00383C 001FF80007E000141F7D9D17>I<0001C00003C00003C0000780000F80001B80003B8000738000 638000C7000187000307000707000E07000C0700180E00300E00600E00E00E00FFFFF0FFFFF000 1C00001C00001C00001C00001C00001C00003C0003FFC003FFC0141E7D9D17>I<03007003FFF0 03FFE003FF8003FE000600000600000600000600000600000600000C7C000DFF000F87800E0380 0C01C00C01C00001E00001E00001E00001E07003C0F003C0F003C0F00380C00780E00700600E00 383C001FF0000FC000141F7D9D17>I<000F80007FC000F0E001C0E00381E00701E00E00C01E00 001C00003C000038000079FC007BFE007E07007C0380F80380F803C0F003C0F003C0F003C0F003 C0F00780E00780E00780E00700E00F00700E00701C003838001FF00007C000131F7C9D17>I<30 00003FFFE07FFFE07FFFC0600180E00300C00600C00C0000180000180000300000600000C00001 C0000180000380000300000700000600000E00000E00001E00001C00001C00003C00003C00003C 0000780000780000780000300000131F799D17>I<003F0000FF8001C1E00380E00700700E0070 0E00700E00700E00E00F00E00F81C00FC38007F70003FC0001FC0003FF000F3F801C0F803807C0 3803C07001C07001C0E001C0E001C0E001C0E003807003007007003C1C001FF80007E000141F7D 9D17>I<007E0001FF000383800701C00E01C01E01C01C01E03C01E03C01E07801E07801E07801 E07801E07803E07803E03807C03807C01C0FC01FFBC00FE380000780000780000700000E00600E 00F01C00F03800E07000E0E0007FC0003F0000131F7C9D17>I<00001800000038000000380000 00780000007C000000FC000000FC000001BC000001BC0000033C0000033E0000061E0000061E00 000C1E00000C1E0000181E0000181F0000300F0000300F0000600F00007FFF0000FFFF0000C00F 000180078001800780030007800300078006000780060007801F0007C0FFC07FFCFFC07FFC1E20 7E9F22>65 D<07FFFF0007FFFFC0003C01E0003C01F0007800F8007800F8007800F8007800F800 7800F8007800F000F001F000F003E000F007C000F00F8000FFFE0000FFFF8001E007C001E003E0 01E003E001E001E001E001E001E001F003C001E003C003E003C003E003C007C003C007C003C00F 8007C03F007FFFFE00FFFFF0001D1F7E9E20>I<0003F808000FFE18003F0738007801F801F000 F803C000F007C00070078000700F0000701F0000701E0000703E0000603C0000607C0000007C00 00007C0000007C000000F8000000F8000000F8000000F8000000F80000C0780000C0780001807C 0001803C0003003C0003001E0006001E000C000F80180007E0700001FFC000007F00001D217B9F 21>I<07FFFF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E0078001E0078001E 0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E000 3E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003C003C00F8007C0 3F007FFFFC00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C0078007800380078 003800780038007800380078003800780C3000F0183000F0180000F0180000F0380000FFF80000 FFF80001E0700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C00060 03C000E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07FFFFF807FFFFF8 003C00F8003C0078007800380078003800780038007800380078003800780C3000F0183000F018 0000F0180000F0380000FFF80000FFF80001E0700001E0300001E0300001E0300001E0300001E0 000003C0000003C0000003C0000003C0000003C0000003C0000007C000007FFE0000FFFE00001D 1F7E9E1E>I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C00078078000380F 0000381F0000381E0000383E0000303E0000307C0000007C0000007C0000007C000000F8000000 F8000000F8007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C03C0003C01E0003 C00F0007C007800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FFC007FFC7FFC000 3C007800003C007800007800F000007800F000007800F000007800F000007800F000007800F000 00F001E00000F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E003C00001E003C0 0001E003C00001E003C00001E003C00001E003C00003C007800003C007800003C007800003C007 800003C007800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22>I<07FFE007FF E0003C00003C0000780000780000780000780000780000780000F00000F00000F00000F00000F0 0000F00001E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003C0 0007C000FFFC00FFFC00131F7F9E10>I<07FFE0FFE007FFE0FFE0003C003E00003C0038000078 007000007800E00000780180000078030000007806000000780C000000F018000000F030000000 F060000000F1F0000000F3F0000000F778000001EC78000001F878000001F03C000001E03C0000 01E01E000001E01E000003C00F000003C00F000003C00F000003C007800003C007800003C003C0 0007C007E0007FFC1FFC00FFFC3FFC00231F7E9E23>75 D<07FFF00007FFF000003C0000003C00 0000780000007800000078000000780000007800000078000000F0000000F0000000F0000000F0 000000F0000000F0000001E0000001E0000001E0000001E0018001E0018001E0030003C0030003 C0030003C0070003C0060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>I<07 FC0000FFC007FC0001FFC0003E0001F800003E00037800006E0003F000006E0006F000006E0006 F000006E000CF0000067000CF00000670018F00000C70019E00000C70031E00000C70031E00000 C70061E00000C38061E00000C380C1E000018380C3C00001838183C00001838183C0000181C303 C0000181C303C0000181C603C0000301C60780000301CC0780000301CC0780000300F807800003 00F80780000700F00780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07FC03 FFC007FC03FFC0003E007C00003E003800006F003000006F003000006F00300000678030000067 8030000063C0300000C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F0600001 80F0C000018078C000018078C00001803CC00001803CC00001803CC00003001F800003001F8000 03000F800003000F800003000F800007000780000F800700007FF0030000FFF0030000221F7E9E 22>I<0003F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F0000F0 1F0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001 F0F80001F0F80001F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E001E00 3C000F00780007C1F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0003C 01F0007800F0007800F8007800F8007800F8007800F8007800F800F001F000F001F000F001E000 F003C000F00F8000FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C00000 03C0000003C0000003C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F>I< 0003F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8001F00F0000F01F0000 F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F800 01F0F80001F0F80003E0780003E0780003C0780007C07C1E07803C3F0F003C619E001E61BC000F C0F80007E1F00003FFC04000FEC0400000E0400000E0800000E1800000FF800000FF000000FF00 00007E0000003C001D297B9F23>I<07FFFC0007FFFF00003C07C0003C03E0007801E0007801F0 007801F0007801F0007801F0007801E000F003E000F003C000F0078000F01F0000FFFC0000FFF0 0001E0380001E03C0001E01C0001E01C0001E01E0001E01E0003C03E0003C03E0003C03E0003C0 3E0003C03E0603C03E0607C03E0C7FFC1F18FFFC0FF8000003E01F207E9E21>I<003F04007FCC 01E0FC03807C03003C0700380600180E00180E00180E00180E00000F00000F00000FE00007FE00 03FF8001FFC0007FE00007E00001E00000E00000F00000F06000E06000E06000E06000E07001C0 700180F80380FE0F00C7FC0081F80016217D9F19>I<1FFFFFF81FFFFFF81E03C0F83803C03838 0780383007803870078018600780186007803860078030C00F0030000F0000000F0000000F0000 000F0000000F0000001E0000001E0000001E0000001E0000001E0000001E0000003C0000003C00 00003C0000003C0000003C0000003C0000007C00001FFFF0003FFFF0001D1F7B9E21>III<7FF803FF80FF F803FF0007C000F80007C000E00003C000C00003E001C00001E003800001F003000000F0060000 00F80E000000F80C00000078180000007C300000003C700000003E600000001EC00000001F8000 00000F800000000F000000000F000000000F000000000E000000001E000000001E000000001E00 0000001E000000001E000000001C000000003C00000003FFE0000007FFE00000211F7B9E22>89 D E /Fv 25 122 df<60F0F0703030306060C040040B7D830B>44 D<03000700FF00FF00070007 0007000700070007000700070007000700070007000700070007000700070007000700FFF0FFF0 0C197D9813>49 D<0070007000F000F001F003F00370067006700C701C701870307030706070E0 70FFFFFFFF0070007000700070007007FF07FF10197F9813>52 D<07801FE0387030306038E018 E018E01CE01CE01CE01CE01C603C703C307C3FDC1F9C00180018003830307870786071C03F801F 000E1A7E9813>57 D69 D<0FFF0FFF007800780078007800780078007800780078 007800780078007800780078007800787078F878F878F878F0F070E03FC00F80101B7F9914>74 D80 D<07C21FF6383E701E600EE00EE006E006E006F00078007F803FF01FF80FFC 01FE001E000F0007C007C007C007E007E006F00EFC1CCFF883E0101C7E9A15>83 D86 D<3F80FFC0F0E0F070607000700FF03FF07870E070E070E073E07370F37FFE3E3C10107E8F13> 97 DI<007E00007E00000E00000E00000E00000E00000E00000E00000E00000E 0007CE001FFE003C1E00700E00700E00E00E00E00E00E00E00E00E00E00E00E00E00700E00701E 003C3E001FFFC007CFC0121A7F9915>100 D<07C01FF038387018701CFFFCFFFCE000E000E000 E0007000300C3C180FF007E00E107F8F11>I<00F801FC03BC073C0E180E000E000E000E000E00 FFC0FFC00E000E000E000E000E000E000E000E000E000E000E000E007FE07FE00E1A80990C>I< FC0000FC00001C00001C00001C00001C00001C00001C00001C00001C00001CF8001FFC001F1E00 1E0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E00FF9FC0FF9FC0 121A7F9915>104 D<18003C003C001800000000000000000000000000FC00FC001C001C001C00 1C001C001C001C001C001C001C001C001C00FF80FF80091A80990A>I108 DII<07E01FF8381C700E6006E007E007E007E007 E007E007700E700E3C3C1FF807E010107F8F13>II114 D<1F207FE070E0E060E060F0007F003FC01FE0 01F0C070C070E070F0E0FFC08F800C107F8F0F>I<0C000C000C000C001C001C003C00FFC0FFC0 1C001C001C001C001C001C001C001C601C601C601C601C600FC007800B177F960F>I121 D E /Fw 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F80 1F000B107F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF 80FF8009107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FF C0FFC00B107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F 801F000B107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001F E01FE00B107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F 803F000B107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F 801F000B107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C 001C0008000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060 C03F801F000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C071 807F003E000B107F8F0F>I E /Fx 30 122 df<000FE0FE00007FF3FF0000F87F8F8001E0FF1F 8003E0FE1F8007C0FE1F8007C07E0F0007C03E000007C03E000007C03E000007C03E000007C03E 0000FFFFFFF000FFFFFFF00007C03E000007C03E000007C03E000007C03E000007C03E000007C0 3E000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007 C03E000007C03E000007C03E00003FF9FFE0003FF9FFE0002120809F1E>11 D45 D<07F8001FFE00381F80780F80FC0FC0FC0FC0FC0FC078 0FC0301F80001F00003E00007C0000700000E00000E00000C00000C00000C00000C00000C00000 C00000000000000000000000000001C00007F00007F00007F00007F00007F00001C00012207D9F 19>63 D<0000E000000000E000000001F000000001F000000001F000000003F800000003F80000 0007FC00000007FC0000000FFE0000000CFE0000000CFE000000187F000000187F000000307F80 0000303F800000703FC00000601FC00000601FC00000C01FE00000C00FE00001FFFFF00001FFFF F000018007F000030003F800030003F800060003FC00060001FC000E0001FE00FFE01FFFE0FFE0 1FFFE0231F7E9E28>65 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E 3F00000E3F00000E7F0000067E0000067E000006FE000000FE000000FE000000FE000000FE0000 00FE000000FE0000007E0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E0 003803F0007000FE01C0003FFF800007FC001F1F7D9E26>67 D 70 D<03FC080FFF381E03F83C00F8780078780038F80038F80018FC0018FC0000FF0000FFF800 7FFF007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC0000FCC0007CC0007CC0007C E00078E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83 D<07FC001FFF003F0F803F07C03F 03E03F03E00C03E00003E001FFE00FFFE03F83E07E03E07C03E0F803E0F803E0F803E0FC07E07E 0DE03FF9FE07E07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC0000FC0000FC0000FC0000FC 00007C00007E00003E00603F00C01F81C007FF0001FC0013147E9317>I<0007F80007F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F801F8F80FFEF81F83F83E01F8 7E00F87C00F87C00F8FC00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00F87C00F87E00F83E01F8 1F07F80FFEFF03F8FF18207E9F1D>I<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00 F8FFFFF8FFFFF8FC0000FC0000FC00007C00007E00003E00183F00380F807007FFE000FF801514 7F9318>I<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800F FF0019FC001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C70003EF0001EF0001EF0 001E78003C78003C3F01F80FFFE001FF00171E7F931A>103 DI<1C003F007F007F007F003F001C00000000000000000000000000FF 00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE0 0B217EA00E>I107 DIII<01FF0007FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007E FC007EFC007EFC007EFC007E7C007C7C007C3E00F83E00F81F83F007FFC001FF0017147F931A> II<01F81807FE381F87783F01F83E01F87E00F87C00F8FC 00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00F87E00F87E00F83F01F81F87F80FFEF803F8F800 00F80000F80000F80000F80000F80000F80000F80007FF0007FF181D7E931C>II<0FE63FFE701E600EE006E006F800FFC07F F83FFC1FFE03FE001FC007C007E007F006F81EFFFCC7F010147E9315>I<018001800180038003 80038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F860F86 0F860F8607CC03F801F00F1D7F9C14>IIIIII E /Fy 78 124 df<000FC3E0007FEFF000E07C 7801C0F8780380F83007007000070070000700700007007000070070000700700007007000FFFF FF80FFFFFF80070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070007FE3FF007FE3FF00 1D20809F1B>11 D<001F80007FC001E0E00381E00381E00701E007000007000007000007000007 0000070000FFFFE0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E07FC3FE7FC3FE1720809F19>I<001FE0007FE0 01E1E00381E00380E00700E00700E00700E00700E00700E00700E00700E0FFFFE0FFFFE00700E0 0700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0 0700E00700E07FE7FE7FE7FE1720809F19>I<000F80F800007FE7FE0000E06E060001C0FC0F00 0380F80F000700F00F000700700000070070000007007000000700700000070070000007007000 00FFFFFFFF00FFFFFFFF0007007007000700700700070070070007007007000700700700070070 070007007007000700700700070070070007007007000700700700070070070007007007000700 700700070070070007007007007FE3FE3FF07FE3FE3FF02420809F26>I<7038F87CFC7EFC7E7C 3E0C060C060C06180C180C381C3018603040200F0E7E9F17>34 D<0078000000FC000001860000 0386000003030000070300000703000007030000070300000706000007060000070C0000071800 0003B81FFC03F01FFC03E003E003C0018003C0030007C003000EE006001CE0060038700C007878 0C0070381800F01C3000F01E3000F00E6000F007C000F003C0187801C0187807E0303E1E78701F F83FE007E00F801E227EA023>38 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D>I<006000 C001800300070006000C001C00180038003800300070007000700060006000E000E000E000E000 E000E000E000E000E000E000E000E0006000600070007000700030003800380018001C000C0006 0007000300018000C000600B2E7DA112>II< 018001C001800180C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C001 8010147DA117>I<00060000000600000006000000060000000600000006000000060000000600 0000060000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF00006 000000060000000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000000600001C207D9A23>I<70F8FCFC7C0C0C0C1818383060 40060E7C840D>II<70F8F8F87005057C840D>I<00030003000700 060006000E000C000C001C0018001800380030003000700060006000E000C000C001C001800180 01800380030003000700060006000E000C000C001C0018001800380030003000700060006000E0 00C000C000102D7DA117>I<03F0000FFC001E1E001C0E00380700780780700380700380700380 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0 7003807003807003807807803807001C0E001E1E000FFC0003F000121F7E9D17>I<018003801F 80FF80E38003800380038003800380038003800380038003800380038003800380038003800380 038003800380038003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E00300F00600F80C0 0780F007C0F807C0F803C0F803C02003C00007C0000780000780000F00000E00001C0000380000 700000E00001C0000380000300000600C00C00C01800C03001807FFF80FFFF80FFFF80121E7E9D 17>I<03F0000FFC001C1E00300F00780F80780F80780780380F80000F80000F00000F00001E00 003C0003F80003F000001C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0 F00780600F80700F003C1E001FFC0003F000121F7E9D17>I<000E00000E00001E00003E00003E 00006E0000EE0000CE00018E00038E00030E00060E000E0E000C0E00180E00380E00300E00600E 00E00E00FFFFF0FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE000FFE0141E 7F9D17>I<3803003FFF003FFE003FF8003FE00030000030000030000030000030000030000031 F00037FC003E0E003C07003807803003800003800003C00003C00003C06003C0F003C0F003C0F0 0380C00780600700700E003C1E000FF80007E000121F7E9D17>I<007C0001FE000783000E0380 0C07801C0780380300380000780000700000700000F3F800F7FE00FC0E00F80700F80380F80380 F003C0F003C0F003C0F003C0F003C07003C07003C07003803803803807001C07000E1E0007FC00 01F000121F7E9D17>I<6000007FFFC07FFFC07FFF80600180C00300C00600C00600000C000018 0000180000300000600000600000E00000C00001C00001C00001C0000380000380000380000380 00078000078000078000078000078000078000078000030000121F7D9D17>I<03F0000FFC001E 1E003807003007007003807003807003807003807807003E07003F0E001FDC000FF80003F80007 FC001E7E00383F00700F80700780E003C0E001C0E001C0E001C0E001C0F001807003803807001E 0E000FFC0003F000121F7E9D17>I<03F00007F8001E1C00380E00380700700700700380F00380 F00380F003C0F003C0F003C0F003C0F003C07007C07007C0380FC01C0FC01FFBC007F3C0000380 000380000780000700300700780E00780E00701C003078001FF0000FC000121F7E9D17>I<70F8 F8F8700000000000000000000070F8F8F87005147C930D>I<70F8F8F870000000000000000000 0070F8F8F878181818183030606040051D7C930D>I<7FFFFFE0FFFFFFF0000000000000000000 0000000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018 F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000 C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0 0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 DI<001FC04000 7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0 780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8 1C00007FF000001FC0001A217D9F21>IIII<000FC020007FF86001F81CE003E006E0078003 E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C 0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F 24>III76 DII<001F800000FFF00001E0780007C03E000F80 1F000F000F001E0007803C0003C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F8 0001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E0 3C0003C03E0007C01E0007800F000F000F801F0007C03E0001F0F80000FFF000001F80001C217D 9F23>II82 D<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F000007800 007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0C001 C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE0780F 01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C1F7E 9E21>II II91 D<0804180C3018703860306030C060C060C060F87C FC7EFC7E7C3E381C0F0E7B9F17>II<1FE0003FF000783800781C 00300E00000E00000E00000E0003FE001FFE003E0E00780E00F80E00F00E30F00E30F00E30F01E 30783F703FEFE00F878014147E9317>97 D<0E0000FE0000FE00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E3F000EFF800FC3C00F00E00E00F00E00700E00780E0078 0E00780E00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00CFF800C3E0015207F 9F19>I<03F80FFC1E1E3C1E380C78007000F000F000F000F000F000F0007000780038033C031F 0E0FFC03F010147E9314>I<000380003F80003F80000380000380000380000380000380000380 00038000038000038003E3800FFB801E0F803C0780780380780380700380F00380F00380F00380 F00380F00380F003807003807003807807803807801E1F800FFBF803E3F815207E9F19>I<03F0 000FFC001E1E003C0F00380700780700700380F00380FFFF80FFFF80F00000F00000F000007000 007800003801801C03800F070007FE0001F80011147F9314>I<003C00FE01CF038F0306070007 0007000700070007000700FFF0FFF0070007000700070007000700070007000700070007000700 07000700070007007FF07FF01020809F0E>I<0001E003E3F00FFF701C1C703C1E00380E00780F 00780F00780F00780F00380E003C1E001C1C003FF80033E0003000003000003800003FFE001FFF 803FFFC07003E07000F0E00070E00070E00070F000F07801E03E07C00FFF0003FC00141F7F9417 >I<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E 3E000EFF000FC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C0FFE7FCFFE7FC16207F9F19>I<1C001E003E001E001C000000 000000000000000000000E007E007E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E00FFC0FFC00A1F809E0C>I<00E001F001F001F000E00000000000000000000000 00007007F007F000F0007000700070007000700070007000700070007000700070007000700070 007000700070007000706070F0E0F1C07F803E000C28829E0E>I<0E0000FE0000FE00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E1FF00E1FF00E0F800E0E000E0C00 0E18000E30000E70000EF0000FF8000F38000E1C000E1E000E0E000E07000E07800E03800E03C0 FFCFF8FFCFF815207F9F18>I<0E00FE00FE000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B 20809F0C>I<0E3F03F000FEFF8FF800FFC1DC1C000F80F80E000F00F00E000E00E00E000E00E0 0E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00 E00E000E00E00E000E00E00E000E00E00E00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FE FF00FFC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C0FFE7FCFFE7FC16147F9319>I<01F80007FE001E07803C03C03801C0 7000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07801E03801C03C03C01E0780 07FE0001F80014147F9317>I<0E3F00FEFF80FFC3C00F01E00E00F00E00F00E00780E00780E00 780E00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00EFF800E3E000E00000E00 000E00000E00000E00000E00000E0000FFE000FFE000151D7F9319>I<03E1800FF9801E1F803C 0780780780780380700380F00380F00380F00380F00380F00380F003807003807803807807803C 0F801E1F800FF38003E380000380000380000380000380000380000380000380003FF8003FF815 1D7E9318>I<0E78FEFCFF9E0F1E0F0C0F000E000E000E000E000E000E000E000E000E000E000E 000E00FFE0FFE00F147F9312>I<1F903FF07070E030E030E030F00078007F803FE00FF000F0C0 78C038C038E038E038F070DFE08F800D147E9312>I<06000600060006000E000E001E003E00FF F8FFF80E000E000E000E000E000E000E000E000E000E000E180E180E180E180E18073007E003C0 0D1C7F9B12>I<0E01C0FE1FC0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E03C00E03C0070FC007FDFC01F1FC16147F9319>III<7FC7FC7FC7FC 0703E007038003830001C70000EE0000EC00007800003800003C00007C0000EE0001C700018700 0303800701C00F01E0FF87FEFF87FE1714809318>II<3F FF3FFF380E301C703C6038607060F001E001C0038007830F030E031C073C063806701EFFFEFFFE 10147F9314>II E /Fz 43 122 df45 DI<007E0001FF8007FFE00FFFF01FC3F81F00F83E007C3E00 7C3C003C7C003E7C003E78001E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001FF800 1FF8001FF8001FF8001FF8001FF8001FF8001FF8001F78001E7C003E7C003E7C003E7C003E3E00 7C3F00FC1F00F81FC3F80FFFF007FFE001FF80007E0018297EA71D>48 D<00180000380000F800 07F800FFF800FFF800FFF800F8F80000F80000F80000F80000F80000F80000F80000F80000F800 00F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F800 00F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF07FFFF014287CA71D> I<00FE0003FF800FFFC01FFFE01F07F03C01F87800FC78007C70007CF0003EF0003E60003E2000 3E00003E00003E00003E00007C00007C0000780000F80001F00001E00003E00007C0000F80001F 00003E00007C0000F80001F00003E00003C0000780000F00001E00003C00007FFFFE7FFFFE7FFF FE7FFFFE17287EA71D>I<007E0003FF8007FFE00FFFF01F81F83E00F87C007C78007C30007C10 007C00007C00007C0000780000F80000F80001F00007E000FFC000FF8000FFC000FFF00001F800 00FC00007C00003E00003E00001F00001F00001F00001F00001F80001FC0003EE0003EF0007E7C 00FC3F01F81FFFF00FFFE003FFC000FE0018297EA71D>I<0007E0000FE0000BE0001BE0001BE0 003BE0003BE00073E00073E000F3E000F3E001E3E001E3E003E3E003C3E007C3E00F83E00F83E0 1F03E01F03E03E03E03E03E07C03E07C03E0F803E0FFFFFFFFFFFFFFFFFFFFFFFF0003E00003E0 0003E00003E00003E00003E00003E00003E00003E00003E018277EA61D>I<3FFFF83FFFF83FFF F83FFFF83E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E3E003EFF 803FFFE03FFFF03FC1F03F80F83F007C3E007C00007C00003E00003E00003E00003E00003E0000 3E00003E20003E60007C70007CF800F87C01F87F07F03FFFE00FFFC007FF0001FC0017287EA61D >I<000FE0003FF800FFF801FFF803F81807E0000F80000F80001F00003E00003E00003C00007C 00007C02007C3FC0F8FFE0F9FFF0FBFFF8FF80FCFE007CFE003EFC003EFC003FFC001FF8001FF8 001FF8001FF8001F78001F7C001F7C001F7C001E3C003E3E003E1E007C1F00FC0FC1F807FFF003 FFE001FF80007E0018297EA71D>II<007E0003FFC007FFE00FFFF01F81F83F00 FC3E007C7E007E7C003E7C003E7C003E7C003E7C003E3E007C3E007C1F00F80FC3F007FFE003FF C003FFC00FFFF01F81F83F00FC3E007C7C003E7C003EF8001FF8001FF8001FF8001FF8001FF800 1F7C003E7C003E7E007E3F00FC1F81F80FFFF007FFE003FFC0007E0018297EA71D>I<007E0001 FF8007FFC00FFFE01F83F01F00F83E00787C007C7C003C78003EF8003EF8003EF8001EF8001FF8 001FF8001FF8001FF8003FFC003F7C003F7C007F3E007F3F01FF1FFFDF0FFF9F07FF1F03FC1E00 403E00003E00003E00007C00007C0000F80000F80001F01803E01E0FC03FFF801FFF000FFE0003 F80018297EA71D>I66 D<0003FE00000FFFC0003F FFF000FFFFF001FE03F003F8006007E0002007C000000F8000001F8000001F0000003E0000003E 0000007C0000007C0000007C0000007C000000F8000000F8000000F8000000F8000000F8000000 F8000000F8000000F8000000F8000000F80000007C0000007C0000007C0000007C0000003E0000 003E0000001F0000001F8000000F80000007C0000007E0000803F8003801FE01F800FFFFF8003F FFF0000FFFC00003FE001D2C7CAA25>II< F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8 F8F8F8052A7BA910>73 D77 DI80 D<007FC00001FFF80007FFFE000FFFFF001FC07F003F000F007E0006007C0000007C000000F800 0000F8000000F8000000F8000000F8000000FC0000007E0000007F0000003F8000001FF800000F FF000007FFE00003FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000000FC0 000007C0000007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00FC003F 00FF80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>83 DI<01FC000FFF003FFF803FFFC03C07E03003E00001F00001F00001F0 0001F00001F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0F801F0F801F0F801F0FC03F0 7E0FF07FFFF03FFFF01FF9F00FE1F0141B7E9A1C>97 DI< 007F8001FFE007FFF80FFFF81FC0783F00103E00007C00007C00007C0000F80000F80000F80000 F80000F80000F80000F800007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FFE0 007F80151B7E9A1A>I<00007C00007C00007C00007C00007C00007C00007C00007C00007C0000 7C00007C00007C00007C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C7C00 7C7C007CFC007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C7E00 FC3F00FC1F83FC1FFFFC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFFC01F 87E03F01E03E00F07C00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F8000078 00007C00007C00003E00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<001FC0 007FC000FFC001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007C000 07C000FFFE00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000 07C00007C000122A7FA912>I<00F80F0003FE7F800FFFFF801FFFFF801F07C0003E03E0003E03 E0007C01F0007C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0003F FF80003BFE000038F8000078000000780000007C0000003FFF80003FFFF0001FFFF8001FFFFC00 3FFFFE007C00FE007C003F00F8001F00F8001F00F8001F00F8001F00FC003F007E007E003F81FC 001FFFF8000FFFF00007FFE00000FF000019287E9A1D>I II107 DI< F83F007E00F9FF83FF00FBFFC7FF80FFFFEFFFC0FF07FE0FC0FE03FC07E0FC01F803E0FC01F803 E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F0 03E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801 F003E0F801F003E0F801F003E0F801F003E0231B7B9A30>II<007E0001 FF8007FFE00FFFF01F81F83F00FC3E007C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8 001FF8001FF8001F7C003E7C003E7C003E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E0018 1B7E9A1D>II<00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E00FC7E007C7C00 7CFC007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CFC007C7C007C7C007C7E007C3F00 FC1FC3FC1FFFFC0FFF7C03FE7C01F87C00007C00007C00007C00007C00007C00007C00007C0000 7C00007C00007C00007C00007C16277E9A1E>II<07F8001FFF003FFF807FFF807C0780F80100F80000F80000F80000FC00007F00007FF000 3FFC001FFE0007FF0000FF80001FC0000FC00007C00007C04007C0E007C0FC1F80FFFF807FFF00 1FFE0007F800121B7E9A16>I<07C00007C00007C00007C00007C00007C00007C000FFFF80FFFF 80FFFF80FFFF8007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0 0007C00007C00007C00007C00007C00007C00007C08007E38003FFC003FFC001FF0000FC001222 7FA115>III121 D E /FA 10 118 df<000000007FFC00000E0000000FFFFFC0001E0000007FFFFFF8003E000003 FFFFFFFE007E00000FFFFFFFFF80FE00003FFFF8007FC1FE0000FFFF80000FF3FE0003FFFC0000 03FFFE0007FFF0000001FFFE000FFFC00000007FFE001FFF800000003FFE003FFF000000001FFE 007FFC000000000FFE00FFFC000000000FFE01FFF80000000007FE03FFF00000000003FE03FFE0 0000000003FE07FFE00000000001FE0FFFC00000000001FE0FFFC00000000000FE1FFF80000000 0000FE1FFF8000000000007E3FFF8000000000007E3FFF0000000000007E3FFF0000000000003E 7FFF0000000000003E7FFF0000000000003E7FFF0000000000003E7FFE000000000000007FFE00 000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000 000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000 FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00 0000000000007FFE000000000000007FFE000000000000007FFF000000000000007FFF00000000 00003E7FFF0000000000003E3FFF0000000000003E3FFF0000000000003E3FFF8000000000003E 1FFF8000000000007E1FFF8000000000007C0FFFC000000000007C0FFFC00000000000FC07FFE0 0000000000FC03FFE00000000000F803FFF00000000001F801FFF80000000003F000FFFC000000 0003F0007FFE0000000007E0003FFF000000000FC0001FFF800000001F80000FFFC00000003F00 0007FFF0000000FE000003FFFC000001FC000000FFFF80000FF80000003FFFF8007FE00000000F FFFFFFFFC000000003FFFFFFFF00000000007FFFFFFC00000000000FFFFFE00000000000007FFE 00000047497AC754>67 D80 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFFF8000003FF003FFC000007FF80 0FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007FF8001FFC00007FF8001FFC000 03FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FFE00000000000FFE00000000000 FFE00000000000FFE00000000000FFE000000000FFFFE00000007FFFFFE0000007FFFFFFE00000 3FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE0000FFF0000FFE0001FFE0000FF E0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF80000FFE000FFF00000FFE000FFF0 0000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000FFF80003FFE0007FFC0007FFE0 007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC3FFFF003FFFFF01FFFF0007FFF C00FFFF00007FE0003FFF0342E7DAD38>97 D<00001FFF80000001FFFFF8000007FFFFFE00001F FFFFFF00007FF801FF8000FFE003FFC003FFC003FFC007FF8003FFC007FF0003FFC00FFE0003FF C01FFE0003FFC01FFC0001FF803FFC0000FF003FFC00007E007FFC000000007FF8000000007FF8 00000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000007FFC 000000007FFC000000007FFC000000003FFC000000003FFE000000001FFE000001F01FFF000001 F00FFF000003F007FF800007E007FFC00007C003FFE0000FC000FFF8003F80007FFF00FF00001F FFFFFC000007FFFFF0000001FFFFC00000001FFC00002C2E7CAD34>99 D<00FC0001FF0003FF80 07FF800FFFC00FFFC00FFFE00FFFE00FFFE00FFFC00FFFC007FF8003FF8001FF0000FC00000000 000000000000000000000000000000000000000000000000000000000000000000007FC0FFFFC0 FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC820>105 D<00FF8003FF000003FF000000FFFF80 1FFFE0001FFFE00000FFFF807FFFF8007FFFF80000FFFF81FFFFFC01FFFFFC0000FFFF83F01FFE 03F01FFE0000FFFF87C007FF07C007FF000003FF8F0007FF8F0007FF800001FF9E0003FF9E0003 FF800001FFBC0003FFBC0003FF800001FFF80001FFF80001FFC00001FFF80001FFF80001FFC000 01FFF00001FFF00001FFC00001FFE00001FFE00001FFC00001FFE00001FFE00001FFC00001FFE0 0001FFE00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFF FF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80592E7BAD62>109 D<00FF8007FE0000FFFF803FFFC000FFFF80FFFFF000FFFF81FFFFFC00FFFF87F01FFC00FFFF8F C00FFE0003FF9F0007FF0001FF9E0007FF0001FFBC0007FF0001FFF80003FF8001FFF80003FF80 01FFF00003FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC000 03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003 FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF 81FFFFFFFFFFFF81FFFFFF382E7BAD41>I<00000FFE0000000001FFFFF000000007FFFFFC0000 001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF00003FF00001FF80007FE0000 0FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC000007FF803FF8000003FF803F F8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0FFF8000003FFE0FFF8000003 FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8 000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07FF8000003FFC07FF8000003FF C07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007FF001FFC000007FF000FFE00 000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FFE000FFE000007FFC07FFC000 001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF000000332E7DAD3A>I<0001F000 000001F000000001F000000001F000000001F000000001F000000003F000000003F000000003F0 00000007F000000007F000000007F00000000FF00000000FF00000001FF00000003FF00000003F F00000007FF0000001FFF0000003FFF000000FFFFFFFC0FFFFFFFFC0FFFFFFFFC0FFFFFFFFC0FF FFFFFFC0007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000 007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF000 00007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF0 0000007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007F F001F0007FF001F0007FF003F0003FF803E0003FF807E0001FFC07C0001FFE0F80000FFFFF8000 03FFFE000000FFFC0000001FE00024427EC12D>116 D<007FC00000FF80FFFFC001FFFF80FFFF C001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF8003FFC00007FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0 0003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80 01FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC000 03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00007FF8001FFC00007FF8001FFC00007 FF8001FFC0000FFF8001FFC0001FFF8000FFC0003FFF8000FFE0007FFF80007FE000FBFFC0007F F803F3FFFF003FFFFFE3FFFF000FFFFF83FFFF0003FFFE03FFFF00007FF003FFFF382E7BAD41> I E /FB 8 117 df<00001E000000003E00000000FE00000003FE0000003FFE0000FFFFFE0000 FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00007FFFFFFF 807FFFFFFF807FFFFFFF807FFFFFFF80213879B730>49 D<0000001FFE0000E0000003FFFFE001 E000001FFFFFF803E000007FFFFFFE07E00001FFFC00FF0FE00007FFC0001FDFE0000FFF000007 FFE0003FFC000001FFE0007FF0000000FFE000FFE00000007FE001FFC00000003FE003FF800000 001FE007FF800000001FE007FF000000000FE00FFE0000000007E00FFE0000000007E01FFC0000 000007E01FFC0000000003E03FFC0000000003E03FF80000000003E07FF80000000001E07FF800 00000001E07FF80000000001E07FF0000000000000FFF0000000000000FFF0000000000000FFF0 000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FF F0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000 FFF00000000000007FF00000000000007FF80000000000007FF80000000001E07FF80000000001 E03FF80000000001E03FFC0000000001E01FFC0000000001E01FFC0000000003E00FFE00000000 03C00FFE0000000007C007FF0000000007C007FF800000000F8003FF800000000F8001FFC00000 001F0000FFE00000003E00007FF00000007C00003FFC000000F800000FFF000003F0000007FFC0 000FE0000001FFFC007FC00000007FFFFFFF000000001FFFFFFC0000000003FFFFE00000000000 1FFE0000003B3D7BBB46>67 D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF8 01FF80001FF800FFC0001FF8007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF0 0003C0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000F FFFFF000007FF83FF00003FF803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0 007FE0003FF000FFE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0 007FF000FFE000FFF0007FE001DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FF E0003FE001FFE02B267DA52F>97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF 8007FC003F800FF8003FC01FF8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE000 0FF87FE0000FF8FFE0000FF8FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0 000000FFE0000000FFE0000000FFE00000007FE00000007FE00000007FF00000003FF00000783F F00000781FF80000F80FF80000F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00 000FFFFC000001FFC00025267DA52C>101 D<00FF0000000000FFFF0000000000FFFF00000000 00FFFF0000000000FFFF000000000007FF000000000003FF000000000003FF000000000003FF00 0000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000000 03FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000 00000003FF000000000003FF000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003 FF0F81FF800003FF3C00FF800003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007F E00003FFC0007FE00003FF80007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF 00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE0 0003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00 007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000 03FF00007FE00003FF00007FE00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1F FFFF80FFFFFC1FFFFF80313C7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFF FE03FF00FFFFF000FFC007FFE0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF 000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF00000FFE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF 00001FF803FF80003FF003FFC0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC 0003FF0FFFF00003FF01FF000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF00000000FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC0000003037 7EA536>112 D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FE E07FE003FEE07FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000 03FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF0000 0003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE 000023267EA528>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000 001F0000001F0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00 001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF 000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003 FF000003FF000003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C 03FF003C03FF003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377E B626>116 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 75 386 a FB(Chapter)31 b(1)75 620 y FA(P)m(oin)m(t)39 b(to)g(P)m(oin)m(t) g(Comm)m(u)o(nication)75 886 y Fz(1.1)59 b(Intro)r(duction)75 1038 y Fy(Sending)13 b(and)f(receiving)h(of)e(messages)f(b)o(y)i(pro)q (cesses)g(is)g(the)f(basic)h(MPI)g(comm)o(unication)g(mec)o(hanism.)75 1095 y(All)20 b(other)f(comm)o(unication)h(functions)g(can)f(b)q(e)h (de\014ned)h(and)e(implemen)o(ted)i(on)e(top)g(of)g(this)g(basic)75 1151 y(comm)o(unication)e(la)o(y)o(er,)f(using)h(few)f(auxiliary)i(data)e (managemen)o(t)f(routines,)i(and)g(using)g(the)f(group)75 1207 y(and)f(comm)o(unicator)g(manipulation)i(functions.)166 1290 y(The)i(basic)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unication)g(op)q (erations)g(are)f Fx(send)g Fy(and)h Fx(receiv)o(e)p Fy(.)29 b(A)18 b Fx(send)75 1346 y Fy(op)q(eration)13 b(creates)f(and)h(sends)g(a)g (message.)18 b(The)13 b(op)q(eration)g(sp)q(eci\014es)i(a)d Fx(send)i(bu\013er)f Fy(in)h(the)e(sender)75 1403 y(memory)f(from)g(whic)o(h) h(the)g(message)f(data)g(is)h(tak)o(en.)18 b(In)12 b(addition,)h(the)f(send)g (op)q(eration)g(asso)q(ciates)f(an)75 1459 y Fx(en)o(v)o(elop)q(e)16 b Fy(with)g(the)g(message.)22 b(This)16 b(en)o(v)o(elop)q(e)h(sp)q(eci\014es) h(the)e(message)g(destination)h(and)f(con)o(tains)75 1516 y(distinguishing)h (information)d(that)f(can)i(b)q(e)g(used)f(b)o(y)g(the)g Fx(receiv)o(e)g Fy(op)q(eration)g(to)g(select)h(a)e(particular)75 1572 y(message.)166 1654 y(A)22 b Fx(receiv)o(e)f Fy(op)q(eration)h(consumes)g(a)f(message.)39 b(The)22 b(message)f(to)g(b)q(e)i(receiv)o(ed)f(is)h(selected)75 1711 y(according)13 b(to)f(the)g(v)m(alue)i(on)e(its)g(en)o(v)o(elop)q(e,)i (and)f(the)f(message)g(data)g(is)h(put)f(in)o(to)g(the)h Fx(receiv)o(e)g (bu\013er)p Fy(.)166 1793 y(The)f(next)g(sections)h(describ)q(e)g(the)g (basic)f(\(blo)q(c)o(king\))h(send)g(and)f(receiv)o(e)h(op)q(erations.)19 b(W)l(e)12 b(discuss)75 1850 y(send,)21 b(receiv)o(e,)g(basic)f(comm)o (unication)g(seman)o(tics,)h(t)o(yp)q(e)e(matc)o(hing)h(requiremen)o(ts,)h(t) o(yp)q(e)e(con)o(v)o(er-)75 1906 y(sion)14 b(in)h(heterogeneous)f(en)o (vironmen)o(ts,)f(and)h(more)g(general)g(comm)o(unication)g(mo)q(des.)20 b(Non)o(blo)q(c)o(king)75 1962 y(comm)o(unication)c(is)h(addressed)f(next,)f (follo)o(w)o(ed)h(b)o(y)g(c)o(hannel-lik)o(e)i(constructs)e(and)f (send-receiv)o(e)j(op-)75 2019 y(erations.)i(W)l(e)15 b(then)g(consider)h (general)f(datat)o(yp)q(es)f(that)g(allo)o(w)h(one)g(to)g(transfer)f (heterogeneous)h(and)75 2075 y(noncon)o(tiguous)g(data.)75 2366 y Fz(1.2)59 b(Basic)20 b(send)g(op)r(eration)75 2518 y Fy(The)15 b(syn)o(tax)g(of)g(the)g(simplest)h(send)g(op)q(eration)f(is)h(giv) o(en)g(b)q(elo)o(w.)p 75 2661 720 2 v 127 2688 a Fw(1)144 2704 y Fv(V)m(ersion)e(of)f(Jan)g(4,)f(1994)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 2 2 bop 75 -100 a Fy(2)631 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(SEND\(buf,)g(count,)g (datat)o(yp)q(e,)g(dest,)g(tag,)f(comm)m(\))117 130 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 213 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)g(in)g(send)h(bu\013er)h(\(nonnegativ)o(e)e(in)o(te-)905 269 y(ger\))117 352 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 434 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 517 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 599 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))75 727 y Fr(int)23 b(MPI)p 245 727 15 2 v 17 w(Send\(void*)g(buf,)g(int)g (count,)h(MPI)p 979 727 V 16 w(Datatype)f(datatype,)g(int)g(dest,)393 784 y(int)h(tag,)f(MPI)p 683 784 V 17 w(Comm)g(comm\))75 874 y(MPI)p 150 874 V 17 w(SEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)h (COMM,)f(IERROR\))170 930 y()g(BUF\(*\))170 987 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(IERROR)75 1160 y Fq(1.2.1)49 b(Message)16 b(data)75 1253 y Fy(The)e(send)g(bu\013er)g(sp)q(eci\014ed)i(b)o (y)e(the)f Ft(MPI)p 803 1253 14 2 v 16 w(SEND)h Fy(op)q(eration)g(consists)g (of)f Ft(count)i Fy(successiv)o(e)g(en)o(tries)f(of)75 1309 y(the)f(t)o(yp)q(e)h(indicated)h(b)o(y)e Ft(datat)o(yp)q(e)p Fy(,)i(starting)e(with)h(the)f(en)o(try)g(at)g(address)h Ft(buf)p Fy(.)20 b(Note)13 b(that)g(w)o(e)g(sp)q(ecify)75 1366 y(the)19 b(message)f(length)i(in)g(terms)e(of)g(n)o(um)o(b)q(er)h(of)g Fp(elements)p Fy(,)f(not)g(n)o(um)o(b)q(er)i(of)e Fp(bytes)p Fy(.)30 b(The)19 b(former)f(is)75 1422 y(mac)o(hine)e(indep)q(enden)o(t)i (and)d(closer)h(to)e(the)h(application)i(lev)o(el.)166 1482 y(The)g(data)f(part)g(of)g(the)g(message)h(consists)f(of)h(a)f(sequence)i(of) e Ft(count)i Fy(v)m(alues,)f(eac)o(h)g(of)f(the)h(t)o(yp)q(e)75 1539 y(indicated)j(b)o(y)e Ft(datat)o(yp)q(e)p Fy(.)31 b Ft(count)20 b Fy(ma)o(y)d(b)q(e)i(zero,)g(in)g(whic)o(h)g(case)g(the)f(data)g(part)f(of)h (the)h(message)e(is)75 1595 y(empt)o(y)l(.)i(The)13 b(basic)h(datat)o(yp)q (es)e(that)h(can)g(b)q(e)h(sp)q(eci\014ed)h(for)d(message)h(data)f(v)m(alues) i(corresp)q(ond)g(to)e(the)75 1652 y(basic)j(datat)o(yp)q(es)f(of)g(the)h (host)f(language.)19 b(The)c(p)q(ossible)h(v)m(alues)g(of)e(this)h(argumen)o (t)f(for)f(F)l(ortran)h(and)75 1708 y(the)h(corresp)q(onding)h(F)l(ortran)e (t)o(yp)q(es)i(are)f(listed)h(b)q(elo)o(w)p 472 1892 1006 2 v 471 1949 2 57 v 497 1932 a(MPI)f(datat)o(yp)q(e)p 1046 1949 V 294 w(F)l(ortran)f(datat)o(yp)q(e)p 1477 1949 V 472 1951 1006 2 v 471 2007 2 57 v 497 1990 a Ft(MPI)p 582 1990 14 2 v 16 w(INTEGER)p 1046 2007 2 57 v 287 w Fr(INTEGER)p 1477 2007 V 471 2064 V 497 2047 a Ft(MPI)p 582 2047 14 2 v 16 w(REAL)p 1046 2064 2 57 v 365 w Fr(REAL)p 1477 2064 V 471 2120 V 497 2103 a Ft(MPI)p 582 2103 14 2 v 16 w(DOUBLE)p 777 2103 V 17 w(PRECISION)p 1046 2120 2 57 v 50 w Fr(DOUBLE)23 b(PRECISION)p 1477 2120 V 471 2176 V 497 2159 a Ft(MPI)p 582 2159 14 2 v 16 w(COMPLEX)p 1046 2176 2 57 v 263 w Fr(COMPLEX)p 1477 2176 V 471 2233 V 497 2216 a Ft(MPI)p 582 2216 14 2 v 16 w(LOGICAL)p 1046 2233 2 57 v 291 w Fr(LOGICAL)p 1477 2233 V 471 2289 V 497 2272 a Ft(MPI)p 582 2272 14 2 v 16 w(CHARA)o(CTER)p 1046 2289 2 57 v 211 w Fr(CHARACTER\(1\))p 1477 2289 V 471 2346 V 497 2329 a Ft(MPI)p 582 2329 14 2 v 16 w(BYTE)p 1046 2346 2 57 v 1477 2346 V 472 2347 1006 2 v 75 2549 a Fy(The)13 b(p)q(ossible)i(v)m (alues)g(for)d(this)i(argumen)o(t)f(for)f(C)h(and)h(the)f(corresp)q(onding)h (C)f(t)o(yp)q(es)g(are)g(listed)i(b)q(elo)o(w.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 3 bop 75 -100 a Fu(1.2.)34 b(BASIC)16 b(SEND)f(OPERA)l(TION)1097 b Fy(3)p 466 5 1019 2 v 465 62 2 57 v 491 45 a(MPI)15 b(datat)o(yp)q(e)p 1004 62 V 259 w(C)g(datat)o(yp)q(e)p 1483 62 V 466 63 1019 2 v 465 120 2 57 v 491 103 a Ft(MPI)p 576 103 14 2 v 16 w(CHAR)p 1004 120 2 57 v 321 w Fr(signed)23 b(char)p 1483 120 V 465 176 V 491 159 a Ft(MPI)p 576 159 14 2 v 16 w(SHORT)p 1004 176 2 57 v 291 w Fr(signed)g(short)p 1483 176 V 465 233 V 491 216 a Ft(MPI)p 576 216 14 2 v 16 w(INT)p 1004 233 2 57 v 365 w Fr(signed)g(int)p 1483 233 V 465 289 V 491 272 a Ft(MPI)p 576 272 14 2 v 16 w(LONG)p 1004 289 2 57 v 321 w Fr(signed)g(long)g(int)p 1483 289 V 465 346 V 491 329 a Ft(MPI)p 576 329 14 2 v 16 w(UNSIGNED)p 815 329 V 17 w(CHAR)p 1004 346 2 57 v 81 w Fr(unsigned)f(char)p 1483 346 V 465 402 V 491 385 a Ft(MPI)p 576 385 14 2 v 16 w(UNSIGNED)p 815 385 V 17 w(SHORT)p 1004 402 2 57 v 51 w Fr(unsigned)g(short)i(int)p 1483 402 V 465 459 V 491 442 a Ft(MPI)p 576 442 14 2 v 16 w(UNSIGNED)p 1004 459 2 57 v 218 w Fr(unsigned)e(int)p 1483 459 V 465 515 V 491 498 a Ft(MPI)p 576 498 14 2 v 16 w(UNSIGNED)p 815 498 V 17 w(LONG)p 1004 515 2 57 v 81 w Fr(unsigned)g(long)i(int)p 1483 515 V 465 571 V 491 555 a Ft(MPI)p 576 555 14 2 v 16 w(FLO)o(A)l(T)p 1004 571 2 57 v 301 w Fr(float)p 1483 571 V 465 628 V 491 611 a Ft(MPI)p 576 611 14 2 v 16 w(DOUBLE)p 1004 628 2 57 v 262 w Fr(double)p 1483 628 V 465 684 V 491 667 a Ft(MPI)p 576 667 14 2 v 16 w(LONG)p 712 667 V 16 w(DOUBLE)p 1004 684 2 57 v 126 w Fr(long)f(double)p 1483 684 V 465 741 V 491 724 a Ft(MPI)p 576 724 14 2 v 16 w(BYTE)p 1004 741 2 57 v 1483 741 V 466 742 1019 2 v 166 912 a Fy(The)16 b(datat)o(yp)q(e)f Ft(MPI)p 536 912 14 2 v 15 w(BYTE)i Fy(do)q(es)e(not)g(corresp)q(ond)h(to)f(a)g(F)l (ortran)g(or)g(C)g(datat)o(yp)q(e.)20 b(A)c(v)m(alue)g(of)75 968 y(t)o(yp)q(e)d Ft(MPI)p 260 968 V 16 w(BYTE)h Fy(consists)f(of)g(a)g(b)o (yte)g(\(8)g(binary)h(digits\).)19 b(A)13 b(b)o(yte)h(is)f(unin)o(terpreted)i (and)e(is)h(di\013eren)o(t)75 1025 y(from)i(a)g(c)o(haracter.)23 b(Di\013eren)o(t)16 b(mac)o(hines)h(ma)o(y)f(ha)o(v)o(e)g(di\013eren)o(t)h (represen)o(tations)f(for)g(c)o(haracters,)f(or)75 1081 y(ma)o(y)i(use)i (more)e(than)h(one)g(b)o(yte)g(to)f(represen)o(t)h(c)o(haracters.)28 b(On)18 b(the)g(other)g(hand,)h(a)e(b)o(yte)h(has)g(the)75 1138 y(same)d(binary)h(v)m(alue)g(on)f(all)h(mac)o(hines.)166 1194 y(MPI)11 b(supp)q(orts)g(the)f(basic)i(datat)o(yp)q(es)e(of)g(F)l (ortran)g(77)g(and)h(ANSI)h(C.)e(Additional)j(MPI)e(datat)o(yp)q(es)75 1251 y(should)32 b(b)q(e)g(pro)o(vided)f(if)h(the)f(host)f(language)h(has)g (additional)i(data)d(t)o(yp)q(es:)51 b(F)l(or)30 b(example,)75 1307 y Ft(MPI)p 160 1307 V 16 w(LONG)p 296 1307 V 17 w(LONG)p 433 1307 V 17 w(INT)p Fy(,)13 b(for)h(64)g(bit)h(in)o(tegers)g(in)g(C;)f Ft(MPI)p 1113 1307 V 16 w(DOUBLE)p 1308 1307 V 16 w(COMPLEX)h Fy(for)f(double)i(preci-)75 1364 y(sion)h(complex)g(in)g(F)l(ortran;)e Ft(MPI)p 667 1364 V 16 w(REAL2)p Fy(,)h Ft(MPI)p 928 1364 V 16 w(REAL4)g Fy(and)h Ft(MPI)p 1266 1364 V 15 w(REAL8)g Fy(for)e(2,4)g(or)h (8)g(b)o(yte)g(reals,)75 1420 y(in)g(F)l(ortran;)e Ft(MPI)p 391 1420 V 15 w(INTEGER1)i(MPI)p 716 1420 V 16 w(INTEGER2)f Fy(and)g Ft(MPI)p 1129 1420 V 16 w(INTEGER4)h Fy(for)e(1,)h(2)g(or)f(4)h(b)o (yte)g(in)o(tegers)75 1476 y(in)h(F)l(ortran;)e(etc.)189 1581 y Fp(R)n(ationale.)77 b Fy(One)23 b(goal)e(of)g(the)h(design)g(is)g(to)f (allo)o(w)h(for)e(MPI)i(to)f(b)q(e)h(implemen)o(ted)h(as)e(a)189 1637 y(library)l(,)d(with)f(no)g(need)h(for)e(additional)j(prepro)q(cessing)f (or)e(compilation.)27 b(Th)o(us,)17 b(one)g(cannot)189 1693 y(assume)c(that)g(a)g(comm)o(unication)h(call)g(has)g(information)f(on)h(the) f(datat)o(yp)q(e)g(of)g(v)m(ariables)i(in)f(the)189 1750 y(comm)o(unication) 20 b(bu\013er;)i(this)f(information)f(m)o(ust)f(b)q(e)i(supplied)h(b)o(y)e (an)g(explicit)i(argumen)o(t.)189 1806 y(The)e(need)h(for)f(suc)o(h)g(datat)o (yp)q(e)g(information)g(will)i(b)q(ecome)f(clear)f(in)h(section)g(1.6.)34 b(\()p Fp(End)20 b(of)189 1863 y(r)n(ationale.)p Fy(\))75 1984 y Fq(1.2.2)49 b(Message)16 b(envelop)q(e)75 2070 y Fy(In)g(addition)g(to)e (the)h(data)g(part,)f(messages)g(con)o(tain)h(information)h(that)e(can)h(b)q (e)h(used)f(to)g(distinguish)75 2127 y(messages)h(and)g(selectiv)o(ely)i (receiv)o(e)f(them.)23 b(This)17 b(information)f(is)h(con)o(tained)g(in)g(a)f (\014xed)h(n)o(um)o(b)q(er)f(of)75 2183 y(\014xed-format)f(\014elds,)h(whic)o (h)g(w)o(e)f(collectiv)o(ely)i(call)g(the)e Fx(message)i(en)o(v)o(elop)q(e)p Fy(.)i(These)d(\014elds)g(are)75 2275 y Fx(source)75 2368 y(destination)75 2462 y(tag)75 2555 y(con)o(text)166 2647 y Fy(The)c(message)g(source)g(is)h (implicitly)i(determined)f(b)o(y)e(the)g(iden)o(tit)o(y)h(of)f(the)g(message) g(sender.)20 b(The)75 2704 y(other)15 b(\014elds)h(are)f(sp)q(eci\014ed)i(b)o (y)f(argumen)o(ts)e(in)i(the)f(send)h(op)q(eration.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 4 bop 75 -100 a Fy(4)631 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)166 49 y Fy(The)h(in)o(teger-v)m(alued)h(message)e(tag)f (is)i(sp)q(eci\014ed)i(b)o(y)d(the)g Ft(tag)h Fy(argumen)o(t.)22 b(This)17 b(in)o(teger)g(can)f(b)q(e)75 106 y(used)c(b)o(y)g(the)f(program)g (to)g(distinguish)i(di\013eren)o(t)f(t)o(yp)q(es)g(of)f(messages.)18 b(The)12 b(range)f(of)g(v)m(alid)i(tag)e(v)m(alues)75 162 y(is)18 b(implemen)o(tation)h(dep)q(enden)o(t)g(and)f(can)f(b)q(e)h(queried)h(using)f (the)g Ft(MPI)p 1362 162 14 2 v 16 w(GET)p 1466 162 V 17 w(V)l(ALID)p 1610 162 V 16 w(T)l(A)o(G)p 1712 162 V 17 w(RANGE)75 219 y Fy(en)o(vironmen)o(tal)e(inquiry)h(function,)e(as)g(describ)q(ed)i(in)f (Chapter)f Fx(??)p Fy(.)166 275 y(The)10 b(con)o(text)g(of)g(the)h(message)e (sen)o(t)h(is)h(sp)q(eci\014ed)i(b)o(y)d(the)g Ft(comm)5 b Fy(argumen)o(t.)17 b(The)11 b(message)f(carries)75 332 y(the)21 b(con)o(text)g(asso)q(ciated)g(with)h(this)f(comm)o(unicator)g(\(see)g (Chapter)g Fx(??)p Fy(\).)37 b(Eac)o(h)21 b(comm)o(unication)75 388 y(con)o(text)11 b(pro)o(vides)i(a)e(separate)h(\\comm)o(unication)g(univ) o(erse":)19 b(messages)12 b(are)f(alw)o(a)o(ys)h(receiv)o(ed)h(within)75 444 y(the)k(con)o(text)f(they)h(w)o(ere)g(sen)o(t,)g(and)g(messages)f(sen)o (t)h(in)g(di\013eren)o(t)g(con)o(texts)g(do)g(not)f(in)o(terfere.)25 b(This)75 501 y(pro)o(vides)16 b(a)f(p)q(o)o(w)o(erful)g(encapsulation)i(mec) o(hanism)e(for)g(libraries)i(and)e(mo)q(dules.)166 557 y(The)f(message)f (destination)h(is)h(sp)q(eci\014ed)g(b)o(y)f(the)f Ft(dest)j Fy(argumen)o(t)c(as)i(a)f(rank)g(within)i(the)f(pro)q(cess)75 614 y(group)i(asso)q(ciated)h(with)g(that)f(same)h(comm)o(unicator)f (\(remote)g(pro)q(cess)h(group,)f(for)g(in)o(tercomm)o(uni-)75 670 y(cators\).)32 b(The)19 b(range)h(of)f(v)m(alid)i(v)m(alues)g(is)f Ft(0,)g(...)32 b(,)20 b(n-1)p Fy(,)h(where)e Ft(n)h Fy(is)h(the)e(n)o(um)o(b) q(er)h(of)f(pro)q(cesses)h(in)75 727 y(this)e(group.)25 b(Th)o(us,)17 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unications)g(do)f(not)g(use)g(absolute) h(addresses,)f(but)g(only)75 783 y(relativ)o(e)f(ranks)f(within)h(a)f(group.) k(This)d(pro)o(vides)g(imp)q(ortan)o(t)f(mo)q(dularit)o(y)l(.)166 840 y(The)j(message)f(en)o(v)o(elop)q(e)i(w)o(ould)f(normally)g(b)q(e)h(enco) q(ded)g(b)o(y)e(a)g(\014xed-length)j(message)d(header.)75 896 y(Ho)o(w)o(ev)o(er,)c(the)i(actual)f(mec)o(hanism)h(used)g(to)f(asso)q(ciate) g(an)g(en)o(v)o(elop)q(e)i(with)e(a)g(message)g(is)h(implemen-)75 953 y(tation)d(dep)q(enden)o(t:)20 b(some)12 b(of)g(the)g(information)g (\(e.g.,)g(source)g(or)g(destination\))g(ma)o(y)g(b)q(e)h(implicit,)i(and)75 1009 y(need)f(not)g(b)q(e)g(explicitly)i(carried)e(b)o(y)g(messages;)f(pro)q (cesses)h(ma)o(y)f(b)q(e)h(iden)o(ti\014ed)i(b)o(y)d(relativ)o(e)h(ranks,)f (or)75 1065 y(absolute)j(ids;)f(etc.)189 1168 y Fp(A)n(dvic)n(e)j(to)i (users.)60 b Fy(Users)18 b(that)g(are)g(comfortable)g(with)h(the)g(notion)g (of)f(a)g(\015at)g(name)h(space)189 1225 y(for)14 b(pro)q(cesses,)h(and)h(a)f (unique)h(comm)o(unication)g(con)o(text,)f(as)f(o\013ered)h(b)o(y)g(most)g (existing)h(com-)189 1281 y(m)o(unication)f(libraries,)h(need)f(only)h(use)e (the)h(prede\014ned)h(v)m(ariable)g Fo(MPI)p 1437 1281 13 2 v 15 w(COMM)p 1582 1281 V 14 w(W)o(ORLD)f Fy(as)f(the)189 1338 y Ft(comm)d Fy(argumen)o(t.)25 b(This)17 b(will)i(allo)o(w)f(to)e(comm)o (unicate)i(with)f(all)h(the)f(pro)q(cesses)h(a)o(v)m(ailable)h(at)189 1394 y(initialization)f(time.)i(\()p Fp(End)c(of)g(advic)n(e)g(to)h(users.)p Fy(\))75 1537 y Fz(1.3)59 b(Basic)20 b(receive)f(op)r(eration)75 1638 y Fy(The)c(syn)o(tax)g(of)g(the)g(simplest)h(receiv)o(e)g(op)q(eration)g (is)f(giv)o(en)h(b)q(elo)o(w.)75 1742 y Ft(MPI)p 160 1742 14 2 v 16 w(RECV)g(\(buf,)f(count,)h(datat)o(yp)q(e,)h(source,)e(tag,)g(comm)m (,)d(status\))117 1819 y Fs(OUT)108 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 1894 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in) h(receiv)o(e)h(bu\013er)g(\(in)o(teger\))117 1968 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h (bu\013er)g(elemen)o(t)e(\(handle\))117 2043 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o(teger\))117 2118 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2192 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2267 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 2391 y Fr(int)23 b(MPI)p 245 2391 15 2 v 17 w(Recv\(void*)g(buf,)g(int)g (count,)h(MPI)p 979 2391 V 16 w(Datatype)f(datatype,)g(int)g(source,)393 2448 y(int)h(tag,)f(MPI)p 683 2448 V 17 w(Comm)g(comm,)g(MPI)p 1034 2448 V 17 w(Status)g(*status\))75 2534 y(MPI)p 150 2534 V 17 w(RECV\(BUF,)f(COUNT,)h(DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(STATUS,)g (IERROR\))170 2591 y()g(BUF\(*\))170 2647 y(INTEGER)g(COUNT,)g (DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(STATUS\(MPI)p 1463 2647 V 16 w(STATUS)p 1623 2647 V 17 w(SIZE\),)170 2704 y(IERROR)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 5 bop 75 -100 a Fu(1.3.)34 b(BASIC)16 b(RECEIVE)g(OPERA)l(TION)1012 b Fy(5)166 49 y(The)15 b(receiv)o(e)h(bu\013er)f(consists)h(of)e(the)h (storage)f(con)o(taining)i Ft(count)g Fy(consecutiv)o(e)g(elemen)o(ts)g(of)f (the)75 106 y(t)o(yp)q(e)e(sp)q(eci\014ed)i(b)o(y)e Ft(datat)o(yp)q(e)p Fy(,)h(starting)e(at)h(address)f Ft(buf)p Fy(.)20 b(The)13 b(length)h(of)e(the)h(receiv)o(ed)h(message)f(m)o(ust)75 162 y(b)q(e)i(less)g(than)f(or)g(equal)h(to)e(the)i(length)g(of)e(the)i(receiv)o (e)g(bu\013er.)20 b(I.e.,)13 b(an)i(error)e(o)q(ccurs)i(if)g(all)g(incoming) 75 219 y(data)j(do)q(es)h(not)g(\014t,)g(without)g(truncation,)g(in)o(to)g (the)g(receiv)o(e)h(bu\013er.)30 b(The)19 b Ft(MPI)p 1531 219 14 2 v 16 w(PROBE)h Fy(function)75 275 y(describ)q(ed)d(in)f(section)g(1.9)e (can)i(b)q(e)f(used)h(to)f(receiv)o(e)h(messages)f(of)f(unkno)o(wn)i(length.) 166 355 y(The)j(selection)h(of)f(a)f(message)h(b)o(y)f(a)h(receiv)o(e)h(op)q (eration)f(is)g(done)g(uniquely)i(according)f(to)e(the)75 412 y(v)m(alue)i(of)f(the)g(message)g(en)o(v)o(elop)q(e.)32 b(The)19 b(receiv)o(e)h(op)q(eration)g(sp)q(eci\014es)h(an)d Fx(en)o(v)o(elop)q(e)k (pattern)p Fy(;)f(a)75 468 y(message)13 b(can)g(b)q(e)g(receiv)o(ed)h(b)o(y)f (that)g(receiv)o(e)h(op)q(eration)f(only)g(if)h(its)f(en)o(v)o(elop)q(e)h (matc)o(hes)f(that)f(pattern.)75 525 y(A)19 b(pattern)g(sp)q(eci\014es)i(v)m (alues)g(for)e(the)g Ft(source)p Fy(,)i Ft(tag)e Fy(and)h Ft(context)g Fy(\014elds)h(of)e(the)g(message)g(en)o(v)o(elop)q(e.)75 581 y(The)d(receiv)o(er)h(ma)o(y)e(sp)q(ecify)j(a)e(wildcard)h Fo(MPI)p 889 581 13 2 v 14 w(ANY)p 988 581 V 15 w(SOURCE)d Fy(v)m(alue)k(for)d Ft(source)p Fy(,)h(and/or)g(a)g(wildcard)75 638 y Fo(MPI)p 152 638 V 14 w(ANY)p 251 638 V 15 w(T)m(A)o(G)c Fy(v)m(alue)j(for)d Ft(tag)p Fy(,)i(indicating)h(that)d(an)o(y)h(source)h (and/or)e(tag)h(are)g(acceptable.)20 b(It)13 b(cannot)75 694 y(sp)q(ecify)h(a)f(wildcard)h(v)m(alue)g(for)f Ft(context)p Fy(.)20 b(Th)o(us,)13 b(a)g(message)f(can)i(b)q(e)f(receiv)o(ed)h(b)o(y)f(a)g (receiv)o(e)h(op)q(eration)75 751 y(only)23 b(if)g(it)g(is)g(addressed)h(to)e (the)g(receiving)j(pro)q(cess,)f(has)f(a)f(matc)o(hing)h(con)o(text,)g(has)g (matc)o(hing)75 807 y(source)14 b(unless)g(source=)p Fo(MPI)p 581 807 V 15 w(ANY)p 681 807 V 15 w(SOURCE)d Fy(in)k(the)e(pattern,)g(and)h (has)g(a)f(matc)o(hing)g(tag)g(unless)i(tag=)75 863 y Fo(MPI)p 152 863 V 14 w(ANY)p 251 863 V 15 w(T)m(A)o(G)f Fy(in)i(the)f(pattern.)166 944 y(The)d(message)g(tag)f(is)i(sp)q(eci\014ed)h(b)o(y)e(the)g Ft(tag)h Fy(argumen)o(t)e(of)h(the)g(receiv)o(e)h(op)q(eration.)19 b(The)12 b(message)75 1000 y Ft(context)20 b Fy(is)f(the)f(con)o(text)g (attac)o(hed)g(with)h(the)f(comm)o(unicator)g(sp)q(eci\014ed)j(b)o(y)d(the)h (argumen)o(t)e Ft(comm)m Fy(.)75 1057 y(The)g(message)g(source,)h(if)f (di\013eren)o(t)h(from)32 b Fo(MPI)p 929 1057 V 14 w(ANY)p 1028 1057 V 15 w(SOURCE)p Fy(,)15 b(is)j(sp)q(eci\014ed)h(as)e(a)g(rank)f (within)j(the)75 1113 y(pro)q(cess)g(group)f(asso)q(ciated)h(with)g(that)e (same)h(comm)o(unicator)h(\(remote)e(pro)q(cess)i(group,)g(for)f(in)o(ter-)75 1170 y(comm)o(unicators\).)27 b(Th)o(us,)18 b(the)g(range)g(of)f(v)m(alid)j (v)m(alues)f(for)e(the)h Ft(source)g Fy(argumen)o(t)g(is)g Fn(f)g Ft(0,)f(...)28 b(,)18 b(n-1)75 1226 y Fn(g[f)d Ft(MPI)p 251 1226 14 2 v 16 w(ANY)p 359 1226 V 17 w(SOURCE)i Fn(g)p Fy(,)d(where)h Ft(n)h Fy(is)g(the)f(n)o(um)o(b)q(er)h(of)e(pro)q(cesses)i(in) g(this)g(group.)166 1306 y(Note)i(the)i(asymmetry)d(b)q(et)o(w)o(een)j(send)f (and)g(receiv)o(e)h(op)q(erations:)28 b(A)19 b(receiv)o(e)h(op)q(eration)f (ma)o(y)75 1363 y(accept)13 b(messages)f(from)g(an)h(arbitrary)f(sender;)i (on)f(the)g(other)f(hand,)i(a)e(send)h(op)q(eration)g(m)o(ust)g(sp)q(ecify)75 1419 y(a)i(unique)j(receiv)o(er.)k(This)16 b(matc)o(hes)f(a)h(\\push")f(comm) o(unication)i(mec)o(hanism,)f(where)g(data)f(transfer)75 1476 y(is)i(e\013ected)h(b)o(y)f(the)g(sender)g(\(rather)f(than)h(a)g(\\pull")h (mec)o(hanism,)g(where)f(data)f(transfer)g(is)i(e\013ected)75 1532 y(b)o(y)d(the)g(receiv)o(er\).)166 1612 y(Source)f(=)h(destination)f(is) h(allo)o(w)o(ed:)k(a)14 b(pro)q(cess)g(can)g(send)h(a)f(message)f(to)g (itself.)21 b(\(Ho)o(w)o(ev)o(er,)12 b(it)i(is)75 1669 y(unsafe)h(to)g(do)g (so)g(with)h(the)f(basic)h(send)g(and)f(receiv)o(e)h(op)q(erations)g(describ) q(ed)h(ab)q(o)o(v)o(e,)e(since)h(this)g(ma)o(y)75 1725 y(lead)g(to)f(deadlo)q (c)o(k;)g(see)h(Section)g(1.4\))75 1983 y Fq(1.3.1)49 b(Return)16 b(status)75 2115 y Fy(The)e(source)g(or)f(the)h(tag)f(of)g(a)h(receiv)o(ed)h (message)e(ma)o(y)g(not)h(b)q(e)g(kno)o(wn)g(if)g(wildcard)h(v)m(alues)g(w)o (ere)e(used)75 2172 y(in)h(the)f(receiv)o(e)h(op)q(eration.)19 b(Also,)13 b(the)g(actual)g(length)h(of)f(the)g(message)f(receiv)o(ed)i(ma)o (y)e(not)h(b)q(e)h(kno)o(wn.)75 2228 y(Th)o(us,)j(this)h(information)f(needs) h(to)e(b)q(e)i(returned)g(b)o(y)f(the)g(receiv)o(e)h(op)q(eration.)26 b(The)17 b(information)g(is)75 2285 y(returned)12 b(b)o(y)g(the)f Ft(status)j Fy(argumen)o(t)d(of)g(the)h Ft(MPI)p 923 2285 V 16 w(RECV)g Fy(function.)20 b(This)12 b(is)g(an)g(argumen)o(t)f(of)g(a)g(sp)q (ecial)75 2341 y(MPI-de\014ned)19 b(t)o(yp)q(e.)27 b(Status)17 b(v)m(ariables)i(need)f(to)f(b)q(e)h(explicitly)i(allo)q(cated)f(b)o(y)e(the) h(user)g({)f(they)g(are)75 2397 y(not)e(system)g(ob)s(jects.)166 2478 y(The)f(status)f(v)m(ariable)i(can)f(b)q(e)g(\\deco)q(ded")h(to)e (retriev)o(e)h(the)g Ft(count)p Fy(,)g Ft(source)h Fy(and)f Ft(tag)f Fy(\014elds,)i(using)75 2534 y(the)g(query)g(functions)h(listed)g(b) q(elo)o(w.)k(The)15 b(information)g(returned)h(b)o(y)f(these)g(query)g (functions)g(is)h(the)75 2591 y(information)k(stored)e(in)j(the)e Ft(status)j Fy(v)m(ariable)e(b)o(y)g(the)f(last)g(receiv)o(e)i(\(or)d(prob)q (e)i({)f(see)h(Section)g(1.9\))75 2647 y(call)c(that)f(w)o(as)f(passed)h Ft(status)i Fy(as)e(argumen)o(t.)k(It)c(is)h(erroneous)f(to)f(call)j(these)e (query)g(functions)h(if)g(the)75 2704 y Ft(status)h Fy(v)m(ariable)g(w)o(as)d (nev)o(er)h(set)g(b)o(y)h(a)e(receiv)o(e)j(\(or)d(prob)q(e\).)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 6 bop 75 -100 a Fy(6)631 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(GET)p 264 49 V 17 w(SOURCE\(status,)i(source\))117 131 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f (\(Status\))117 215 y(OUT)108 b Ft(source)468 b Fs(source)15 b(rank)f(\(in)o(teger\))75 344 y Fr(int)23 b(MPI)p 245 344 15 2 v 17 w(Get)p 334 344 V 17 w(source\(MPI)p 591 344 V 16 w(Status)g(status,)g(int)g(*source\))75 435 y(MPI)p 150 435 V 17 w(GET)p 239 435 V 17 w(SOURCE\(STATUS,)e(SOURCE,)i(IERROR\))170 491 y(INTEGER)g(STATUS\(MPI)p 604 491 V 16 w(STATUS)p 764 491 V 17 w(SIZE\),)g(SOURCE,)f(IERROR)166 582 y Fy(Returns)13 b(the)g(rank)g(of)f (the)h(message)f(source)h(in)h(the)f(group)f(asso)q(ciated)h(with)g(the)g (comm)o(unicator)75 639 y(that)h(w)o(as)h(used)h(to)e(receiv)o(e)i(\(remote)f (group,)f(for)h(in)o(tercomm)o(unicators\).)75 747 y Ft(MPI)p 160 747 14 2 v 16 w(GET)p 264 747 V 17 w(T)l(A)o(G\(status,)i(tag\))117 829 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i (or)f(prob)q(e)h(op)q(eration)f(\(Status\))117 913 y(OUT)108 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))75 1042 y Fr(int)23 b(MPI)p 245 1042 15 2 v 17 w(Get)p 334 1042 V 17 w(tag\(MPI)p 519 1042 V 16 w(Status)g(status,)g(int)h(*tag\))75 1133 y(MPI)p 150 1133 V 17 w(GET)p 239 1133 V 17 w(TAG\(STATUS,)e(TAG,)h (IERROR\))170 1189 y(INTEGER)g(STATUS\(MPI)p 604 1189 V 16 w(STATUS)p 764 1189 V 17 w(SIZE\),)g(TAG,)g(IERROR)166 1280 y Fy(Returns)16 b(the)f(tag)f(of)h(receiv)o(ed)h(message.)75 1388 y Ft(MPI)p 160 1388 14 2 v 16 w(GET)p 264 1388 V 17 w(COUNT\(status,)h (datat)o(yp)q(e,)f(count\))117 1470 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f(\(Status\))117 1554 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e(\(handle\))117 1638 y(OUT)108 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h (elemen)o(ts)f(\(in)o(teger\))75 1767 y Fr(int)23 b(MPI)p 245 1767 15 2 v 17 w(Get)p 334 1767 V 17 w(count\(MPI)p 567 1767 V 16 w(Status)g(status,)g(MPI)p 1013 1767 V 17 w(Datatype)f(datatype,)h(int)h (*count\))75 1858 y(MPI)p 150 1858 V 17 w(GET)p 239 1858 V 17 w(COUNT\(STATUS,)e(DATATYPE,)g(COUNT,)h(IERROR\))170 1915 y(INTEGER)g(STATUS\(MPI)p 604 1915 V 16 w(STATUS)p 764 1915 V 17 w(SIZE\),)g(DATATYPE,)f(COUNT,)h(IERROR)166 2006 y Fy(Returns)15 b(the)f(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)h(receiv)o(ed.)21 b(\(Here,)14 b(again,)h(w)o(e)f(coun)o(t)g Fp(elements)p Fy(,)f(not)h Fp(bytes)p Fy(.\))75 2062 y(The)f Ft(datat)o(yp)q(e)h Fy(argumen)o(t)e (should)i(matc)o(h)e(the)h(argumen)o(t)f(pro)o(vided)i(b)o(y)f(the)g(receiv)o (e)g(call)h(that)e(set)h(the)75 2119 y Ft(status)k Fy(v)m(ariable.)43 b(\(W)l(e)16 b(shall)g(later)g(see,)f(in)h(Section)h(1.13.5,)c(that)i Ft(MPI)p 1350 2119 14 2 v 16 w(GET)p 1454 2119 V 16 w(COUNT)i Fy(ma)o(y)d(return,)-1903 b Fm(\))75 2175 y Fy(in)16 b(certain)g(situations,) f(the)g(v)m(alue)i Fo(MPI)p 773 2175 13 2 v 14 w(UNDEFINED)p Fy(.\))166 2236 y(Note)e(that)g(it)h(is)g(not)f(mandatory)g(to)g(query)g(the) h(return)g(status)e(after)h(a)g(receiv)o(e.)22 b(The)16 b(receiv)o(er)75 2293 y(will)h(use)e(these)h(calls,)g(or)e(some)h(of)g(them,)g(only)h(when)f (the)h(information)f(they)g(return)g(is)h(needed.)189 2421 y Fp(R)n(ationale.)43 b Fy(The)16 b(use)g(of)f(a)h(separate)f(status)g (argumen)o(t)g(prev)o(en)o(ts)h(errors)e(that)i(are)f(often)g(at-)189 2478 y(tac)o(hed)f(with)h Fs(INOUT)g Fy(argumen)o(t)f(\(e.g.,)f(passing)i (the)28 b Fo(MPI)p 1233 2478 V 14 w(ANY)p 1332 2478 V 15 w(T)m(A)o(G)13 b Fy(constan)o(t)h(as)g(the)h(actual)189 2534 y(argumen)o(t)k(for)g Ft(tag)p Fy(\).)32 b(The)20 b(use)g(of)g(an)f(explicit)j(user)e(ob)s(ject)f (b)q(ecomes)h(imp)q(ortan)o(t)g(for)f(non-)189 2591 y(blo)q(c)o(king)d(comm)o (unication)g(as)f(it)g(allo)o(ws)g(the)h(receiv)o(e)g(op)q(eration)f(to)g(b)q (e)g(stateless)g(and,)g(hence,)189 2647 y(reen)o(tran)o(t.)j(This)d(prev)o (en)o(ts)f(confusions)h(in)g(the)f(case)g(where)h(m)o(ultiple)h(receiv)o(es)f (can)f(b)q(e)h(p)q(osted)189 2704 y(b)o(y)g(a)g(pro)q(cess.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 7 bop 75 -100 a Fu(1.4.)34 b(SEMANTICS)16 b(OF)f(POINT-TO-POINT)i(COMMUNICA)l (TION)472 b Fy(7)189 49 y(The)18 b Ft(datat)o(yp)q(e)i Fy(argumen)o(t)d(is)i (passed)g(to)e(the)i Ft(MPI)p 1096 49 14 2 v 16 w(GET)p 1200 49 V 16 w(COUNT)g Fy(so)f(as)g(to)g(impro)o(v)o(e)g(p)q(erfor-)189 106 y(mance)k(\(a)f(message)h(migh)o(t)g(b)q(e)g(receiv)o(ed)i(without)e (coun)o(ting)g(the)g(n)o(um)o(b)q(er)g(of)g(elemen)o(ts)h(it)189 162 y(con)o(tains\),)14 b(and)i(to)e(allo)o(w)i(its)f(use)h(after)e(a)h(call) h(to)f Ft(MPI)p 1170 162 V 16 w(PROBE)p Fy(.)h(\()p Fp(End)f(of)i(r)n (ationale.)p Fy(\))166 354 y Fl(Implemen)o(tati)o(on)d(note:)44 b Fs(One)16 b(exp)q(ects)i(deco)q(de)f(functions)f(to)g(b)q(e)g(in-lined)f (in)g(man)o(y)f(implemen)o(ta-)75 411 y(tions.)166 551 y Fy(All)19 b(send)g(and)g(receiv)o(e)g(op)q(erations)f(use)h(the)f Ft(buf)p Fy(,)h Ft(count)p Fy(,)h Ft(datat)o(yp)q(e)p Fy(,)g Ft(source)p Fy(,)f Ft(dest)p Fy(,)h Ft(tag)p Fy(,)f Ft(comm)75 607 y Fy(and)e Ft(status)i Fy(argumen)o(ts)d(in)i(the)f(same)g(w)o(a)o(y)f(as)g(the)h(basic) h Ft(MPI)p 1201 607 V 16 w(SEND)f Fy(and)g Ft(MPI)p 1523 607 V 16 w(RECV)h Fy(op)q(erations)75 664 y(describ)q(ed)f(in)f(this)g(section.) 166 797 y Fl(Discussion:)56 b Fs(An)15 b(alternativ)o(e)f(design)h(for)f(the) h(query)h(functions)e(is)h(to)f(mak)o(e)f(explicit)h(\(part)h(of)s(\))f(the) 69 b Fk(\()75 847 y Fs(structure)17 b(of)c Fo(Status)i Fs(v)n(ariables,)e(so) i(that)f(the)h(use)h(can)e(access)i(status)g(information)11 b(merely)j(b)o(y)g(accessing)h(the)75 896 y(righ)o(t)h(\014eld)g(of)g(a)g (status)h(v)n(ariable.)24 b(E.g,)16 b(in)f(C,)h Fo(status.tag,)g (status.source)h Fs(etc.)26 b(w)o(ould)15 b(pro)o(vide)h(the)h(tag)f(and)75 946 y(source)h(information;)c(in)i(F)m(ortran)g Fo(ST)m(A)m(TUS\(MPI)p 891 946 13 2 v 13 w(T)m(A)o(G\),)g(ST)m(A)m(TUS\(MPI)p 1271 946 V 13 w(SOURCE\))p Fs(,)e(could)j(b)q(e)g(used)g(for)f(the)75 996 y(same)8 b(purp)q(ose.)18 b(This)9 b(has)h(the)g(adv)n(an)o(tage)e(of)h (a)o(v)o(oiding)e(a)j(function)f(call)f(\(whic)o(h)i(cannot)f(b)q(e)h (inlined)f(in)g(standard)75 1046 y(F)m(ortran,)15 b(although)g(it)g(can)h(b)q (e)g(inlined)f(in)g(C)g(and)g(in)g(F)m(ortran)h(systems)f(whic)o(h)h(supp)q (ort)g(macro)e(expansion\).)75 1096 y(The)e(use)g(of)f(this)h(design)f(in)g (replacemen)o(t)h(of)f Fo(MPI)p 873 1096 V 14 w(GET)p 968 1096 V 15 w(COUNT)p Fs(,)f(and)h(of)g(the)h(function)f Fo(MPI)p 1576 1096 V 15 w(GET)p 1672 1096 V 14 w(ELEMENT)75 1145 y Fs(of)i(section)i (1.13.5)d(w)o(ould)h(require)i(to)e(delete)i(the)g Fo(datat)o(yp)q(e)g Fs(argumen)o(t)e(from)f(the)i(function)g(call)f({)h(see)h(b)q(elo)o(w.)166 1196 y(The)j(use)g(of)f(a)h(datat)o(yp)q(e)g(argumen)o(t)e(in)h Fo(MPI)p 916 1196 V 15 w(GET)p 1012 1196 V 14 w(COUNT)g Fs(is)h(somewhat)e (inelegan)o(t.)29 b(On)18 b(the)g(other)75 1246 y(hand,)g(the)g(use)h(of)e(a) g(datat)o(yp)q(e)h(argumen)o(t)f(is)g(rational)g(after)g(a)h(call)f(to)g Fo(MPI)p 1358 1246 V 14 w(PROBE)p Fs(,)f(since)i(one)g(ma)o(y)e(not)75 1296 y(kno)o(w)f(the)g(datat)o(yp)q(e)g(of)g(the)h(incoming)c(data)j(when)h (one)f(prob)q(es,)h(and)f(it)f(is)h(inelegan)o(t)g(and)g(time)e(consuming)75 1345 y(to)h(prob)q(e)g(m)o(ultiple)e(times,)g(with)i(a)g(datat)o(yp)q(e)g (argumen)o(t,)e(with)h(a)h(call)f(to)h Fo(MPI)p 1361 1345 V 14 w(IPROBE)p Fs(.)166 1396 y(As)g(an)g(alternativ)o(e)g(to)f(the)i(curren)o (t)g(design,)f(on)f(can)75 1446 y(1.)37 b(Rede\014ne)22 b(it)e(not)g(to)g (require)i(a)e(datat)o(yp)q(e)g(argumen)o(t,)h(and)f(ha)o(v)o(e)g(a)g (distinct)h(function)f(that)h(requires)75 1495 y(a)e(datat)o(yp)q(e)h (argumen)o(t)f(to)g(use)i(after)e(prob)q(e)i(\(this)f(do)q(es)g(not)f(add)h (an)o(y)f(real)h(o)o(v)o(erhead,)h(but)e(adds)h(a)g(new)75 1545 y(function\).)75 1595 y(2.)38 b(Replace)21 b(a)g(function)f(call)g(b)o (y)h(an)f(access)j(to)d(status.coun)o(t)i(\(adds)f(a)f(division)g(and)h(a)f (few)h(additional)75 1645 y(instructions)15 b(to)e(eac)o(h)i(receiv)o(e\),)g (and)e(ha)o(v)o(e)h(the)g(curren)o(t)i Fo(MPI)p 1079 1645 V 14 w(GET)p 1174 1645 V 15 w(COUNT)d Fs(used)i(after)f(prob)q(e.)166 1695 y(The)h(sa)o(vings)f(from)f(ha)o(ving)h(a)g(datat)o(yp)q(e)h(argumen)o (t)f(are)h(more)f(signi\014can)o(t)g(for)h Fo(MPI)p 1542 1695 V 14 w(GET)p 1637 1695 V 15 w(ELEMENTS)p Fs(,)75 1745 y(where)h(one)g(w)o (ould)e(ha)o(v)o(e)h(otherwise)h(to)f(main)o(tain)e(a)i(running)f(elemen)o(t) h(coun)o(t)h(when)f(a)g(message)g(is)g(receiv)o(ed.)75 1795 y(Also,)e(an)h(elemen)o(t)f(coun)o(t)h(is)g(less)h(lik)o(ely)d(to)i(b)q(e)g (needed.)166 1852 y(An)o(y)g(app)q(etite)g(for)g(c)o(hanges?)75 2082 y Fz(1.4)59 b(Semantics)18 b(of)i(p)r(oint-to-p)r(oint)e(communication) 75 2185 y Fy(A)g(v)m(alid)h(MPI)e(implemen)o(tation)i(guaran)o(tees)e (certain)h(general)g(prop)q(erties)g(of)f(p)q(oin)o(t-to-p)q(oin)o(t)h(com-) 75 2241 y(m)o(unication,)e(whic)o(h)g(are)f(describ)q(ed)i(in)f(this)f (section.)75 2365 y Ft(Order)46 b Fy(Messages)14 b(are)h Fp(non-overtaking)p Fy(,)g(within)h(eac)o(h)f(con)o(text:)20 b(if)c(t)o(w)o(o)d(messages)i(are)g (successiv)o(ely)75 2421 y(sen)o(t)10 b(from)g(the)g(same)g(source,)h(to)e (the)i(same)f(destination,)h(with)g(the)f(same)g(con)o(text,)h(then)f(the)g (messages)75 2478 y(are)17 b(receiv)o(ed)i(in)f(the)f(order)g(they)h(w)o(ere) f(sen)o(t.)26 b(I.e.,)17 b(if)h(the)f(receiv)o(er)i(p)q(osts)e(t)o(w)o(o)f (successiv)o(e)i(receiv)o(es)75 2534 y(that)g(b)q(oth)g(matc)o(h)g(either)i (message,)e(then)h(the)f(\014rst)h(receiv)o(e)g(will)h(receiv)o(e)g(the)e (\014rst)g(message,)h(and)75 2591 y(the)e(second)h(receiv)o(e)h(will)g (receiv)o(e)f(the)f(second)h(message.)26 b(This)18 b(requiremen)o(t)g (facilitates)g(matc)o(hing)75 2647 y(of)e(sends)h(to)f(receiv)o(es.)25 b(It)16 b(guaran)o(tees)g(that)g(message-passing)g(co)q(de)h(is)g (deterministic,)i(if)e(pro)q(cesses)75 2704 y(are)h(single-threaded)j(and)e (wildcard)h Fo(MPI)p 831 2704 V 14 w(ANY)p 930 2704 V 15 w(SOURCE)d Fy(is)i(not)f(used)h(in)h(receiv)o(es.)31 b(\(Some)19 b(of)f(the)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 8 bop 75 -100 a Fy(8)636 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y Fy(calls)22 b(describ)q(ed)h(later,)g(suc)o(h)e(as) g Ft(MPI)p 771 49 14 2 v 16 w(CANCEL)g Fy(or)g Ft(MPI)p 1124 49 V 16 w(W)l(AIT)l(ANY)p Fy(,)h(are)f(additional)h(sources)f(of)75 106 y(nondeterminism.\),)166 162 y(If)c(a)g(pro)q(cess)h(has)f(a)g(single)h (thread)f(of)g(execution,)i(then)e(an)o(y)g(t)o(w)o(o)f(comm)o(unications)i (executed)75 219 y(b)o(y)e(this)h(pro)q(cess)g(are)f(ordered.)23 b(On)17 b(the)f(other)g(hand,)h(if)g(the)f(pro)q(cess)h(is)g(m)o (ulti-threaded,)g(then)g(the)75 275 y(seman)o(tics)e(of)f(thread)h(execution) h(ma)o(y)e(not)g(de\014ne)i(a)e(relativ)o(e)i(order)e(b)q(et)o(w)o(een)h(t)o (w)o(o)e(send)j(op)q(erations)75 332 y(executed)g(b)o(y)f(t)o(w)o(o)f (distinct)j(threads:)i(the)d(op)q(erations)f(are)g(logically)i(concurren)o (t,)e(ev)o(en)h(though)f(one)75 388 y(ph)o(ysically)22 b(precedes)e(the)g (other.)33 b(In)20 b(suc)o(h)g(a)f(case,)i(the)e(t)o(w)o(o)g(messages)g(sen)o (t)g(can)h(b)q(e)g(receiv)o(ed)h(in)75 444 y(an)o(y)f(order.)37 b(Similarly)l(,)24 b(if)d(t)o(w)o(o)e(receiv)o(e)j(op)q(erations)f(that)f (are)g(logically)j(concurren)o(t)e(receiv)o(e)g(t)o(w)o(o)75 501 y(successiv)o(ely)i(sen)o(t)d(messages,)h(then)g(the)g(t)o(w)o(o)f (messages)g(can)h(matc)o(h)f(the)h(t)o(w)o(o)e(receiv)o(es)j(in)f(either)75 557 y(order.)31 b(Note)19 b(that)f(the)h(receiv)o(er)h(is)g(not)e(required)i (to)f(receiv)o(e)h(messages)e(in)i(the)f(order)g(they)g(w)o(ere)75 614 y(sen)o(t,)c(if)g(they)h(ha)o(v)o(e)e(distinct)j(en)o(v)o(elop)q(es.)75 734 y Ft(Progress)45 b Fy(If)13 b(a)g(pair)g(of)f(matc)o(hing)h(send)h(and)f (receiv)o(es)g(ha)o(v)o(e)g(b)q(een)h(initiated)g(on)f(t)o(w)o(o)e(pro)q (cesses,)j(then)75 790 y(at)k(least)h(one)h(of)e(these)h(t)o(w)o(o)f(op)q (erations)h(will)i(complete,)f(indep)q(enden)o(tly)i(of)c(other)h(actions)g (in)h(the)75 847 y(system:)e(the)13 b(send)h(op)q(eration)f(will)h(complete,) g(unless)g(the)f(receiv)o(e)h(is)f(satis\014ed)h(b)o(y)e(another)h(message,) 75 903 y(and)h(completes;)g(the)g(receiv)o(e)h(op)q(eration)f(will)h (complete,)g(unless)g(the)f(message)f(sen)o(t)g(is)i(consumed)f(b)o(y)75 960 y(another)h(matc)o(hing)g(receiv)o(e)h(that)f(w)o(as)f(p)q(osted)h(at)g (the)g(same)g(destination)h(pro)q(cess.)75 1079 y Ft(F)o(airness)46 b Fy(MPI)15 b(mak)o(es)g(no)g(guaran)o(tee)g(of)g Fp(fairness)f Fy(in)i(the)g(handling)h(of)e(comm)o(unication.)21 b(Supp)q(ose)75 1136 y(that)c(a)f(send)i(w)o(as)f(p)q(osted.)26 b(Then)18 b(it)f(is)h(p)q (ossible)h(that)e(the)g(destination)h(pro)q(cess)g(rep)q(eatedly)g(p)q(osts) 75 1192 y(a)d(receiv)o(e)h(that)f(matc)o(hes)f(this)i(send,)g(y)o(et)e(the)i (message)f(is)g(nev)o(er)h(receiv)o(ed,)g(b)q(ecause)g(it)g(is)f(eac)o(h)h (time)75 1249 y(o)o(v)o(ertak)o(en)e(b)o(y)i(another)f(message,)g(sen)o(t)h (from)f(another)g(source.)21 b(Similarly)l(,)d(supp)q(ose)f(that)e(a)g (receiv)o(e)75 1305 y(w)o(as)f(p)q(osted)i(b)o(y)f(a)g(m)o(ulti-threaded)h (pro)q(cess.)k(Then)c(it)f(is)h(p)q(ossible)h(that)d(messages)h(that)f(matc)o (h)h(this)75 1362 y(receiv)o(e)20 b(are)g(rep)q(eatedly)g(receiv)o(ed,)i(y)o (et)d(the)g(receiv)o(e)i(is)f(nev)o(er)f(satis\014ed,)i(b)q(ecause)f(it)g(is) g(o)o(v)o(ertak)o(en)75 1418 y(b)o(y)15 b(other)h(receiv)o(es)g(p)q(osted)g (at)e(this)i(no)q(de)h(\(b)o(y)e(other)g(executing)h(threads\).)21 b(It)15 b(is)h(the)g(programmer's)75 1475 y(resp)q(onsibilit)o(y)i(to)c(prev) o(en)o(t)h(starv)m(ation)g(in)h(suc)o(h)g(situations.)75 1594 y Ft(Resource)e(lim)o(itations)44 b Fy(The)13 b(curren)o(t)f(practice)h(for)f (man)o(y)g(commercial)i(message-passing)e(libraries)i(is)75 1651 y(that)f(\(short\))f(messages)h(are)g(bu\013ered)g(b)o(y)h(the)f (system,)g(th)o(us)g(allo)o(wing)h(blo)q(c)o(king)h(send)f(op)q(erations)g (to)75 1707 y(complete)d(ahead)f(of)g(the)h(matc)o(hing)f(receiv)o(es.)19 b(It)11 b(is)f(exp)q(ected)i(that)e(man)o(y)f(MPI)i(implemen)o(tations)g (will)75 1764 y(follo)o(w)j(this)g(practice,)g(and)f(pro)o(vide)h(the)g(same) f(lev)o(el)i(of)e(bu\013ering)h(that)f(is)h(a)o(v)m(ailable)h(on)f(the)f (libraries)75 1820 y(they)21 b(replace.)37 b(Ho)o(w)o(ev)o(er,)20 b(message)g(bu\013ering)h(is)h(not)e(a)g(univ)o(ersal)i(practice.)36 b(Ev)o(en)21 b(on)f(systems)75 1877 y(where)e(bu\013ering)g(o)q(ccur,)g(the)f (amoun)o(t)g(of)g(bu\013er)h(space)f(a)o(v)m(ailable)j(and)d(the)h(w)o(a)o(y) e(it)i(is)g(allo)q(cated)g(is)75 1933 y(b)q(ound)e(to)f(b)q(e)h(implemen)o (tation)g(dep)q(enden)o(t.)166 1990 y(Therefore,)d(message)h(bu\013ering)g (is)g(not)f(mandated)h(b)o(y)f(MPI.)h(A)f(v)m(alid)j(MPI)d(implemen)o(tation) i(of)75 2046 y Ft(MPI)p 160 2046 V 16 w(SEND)j Fy(is)g(to)f(blo)q(c)o(k)h (the)f(sender)i(un)o(til)f(a)f(matc)o(hing)h(receiv)o(e)g(has)g(b)q(een)g (initiated.)29 b(In)18 b(general,)75 2103 y(the)d(programmer)e(can)i(mak)o(e) g(no)f(assumptions)h(on)g(the)g(a)o(v)m(ailabilit)o(y)i(of)d(bu\013er)h (space,)f(and)h(ho)o(w)g(this)75 2159 y(space)g(is)g(allo)q(cated.)21 b(Th)o(us,)14 b(p)q(ortable)h(\(safe\))f(MPI)g(co)q(de)i(should)f(w)o(ork)f (under)h(the)g(assumption)g(that)75 2215 y(an)g(arbitrary)g(subset)h(of)f (the)h(send)g(op)q(erations)f(are)h(going)f(to)g(return)g(b)q(efore)h(a)f (matc)o(hing)h(receiv)o(e)g(is)75 2272 y(p)q(osted,)f(and)g(the)h(rest)f (will)h(blo)q(c)o(k)g(un)o(til)h(a)e(matc)o(hing)g(receiv)o(e)h(is)g(p)q (osted.)166 2328 y(An)g(MPI)g(implemen)o(tation)h(will)g(pro)o(vide)f (information)g(and)g(con)o(trol)f(on)h(the)g(amoun)o(t)f(of)g(a)o(v)m(ail-)75 2385 y(able)j(bu\013er)f(space)h(and)f(on)g(the)g(bu\013ering)h(p)q(olicy)h (via)e(the)h(en)o(vironmen)o(tal)g(inquiries)h(describ)q(ed)g(in)75 2441 y(Chapter)c Fx(??)p Fy(.)166 2498 y(Examples)h(\(in)o(v)o(olving)g(t)o (w)o(o)e(pro)q(cesses)h(with)h(ranks)f(0)f(and)i(1\))166 2554 y(The)f(follo)o(wing)h(program)e(is)i(safe,)f(and)g(should)h(alw)o(a)o(ys)f (succeed.)75 2704 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 9 bop 75 -100 a Fu(1.5.)34 b(D)o(A)l(T)l(A)15 b(TYPE)g(MA)l(TCHING)1127 b Fy(9)75 49 y Fr(IF)24 b(\(rank.EQ.0\))123 106 y(THEN)170 162 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g (ierr\))170 219 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,) f(comm,)g(status,)g(ierr\))123 275 y(ELSE)95 b(!)23 b(rank.EQ.1)170 332 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))170 388 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g (0,)h(tag,)f(comm,)g(ierr\))75 444 y(END)g(IF)166 538 y Fy(The)15 b(follo)o(wing)h(program)e(is)i(erroneous,)f(and)g(should)h(alw)o(a)o(ys)f (fail.)75 687 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 744 y(IF)h(\(rank.EQ.0\))123 800 y(THEN)170 857 y(CALL)g(MPI_RECV\(recvbuf,)d (count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g(ierr\))170 913 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g (ierr\))123 970 y(ELSE)95 b(!)23 b(rank.EQ.1)170 1026 y(CALL)h (MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g(status,)g (ierr\))170 1082 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(0,)h (tag,)f(comm,)g(ierr\))75 1139 y(END)g(IF)166 1232 y Fy(The)15 b(receiv)o(e)g(op)q(eration)g(of)f(the)h(\014rst)f(pro)q(cess)h(m)o(ust)f (complete)i(b)q(efore)f(its)f(send,)h(and)g(can)g(com-)75 1289 y(plete)j(only)g(if)g(the)f(matc)o(hing)g(send)h(of)f(the)g(second)h(pro)q (cessor)f(is)h(executed;)g(the)g(receiv)o(e)g(op)q(eration)75 1345 y(of)c(the)h(second)g(pro)q(cess)g(m)o(ust)f(complete)h(b)q(efore)g(its) f(send)i(and)e(can)h(complete)g(only)g(if)g(the)g(matc)o(hing)75 1401 y(send)h(of)f(the)g(\014rst)g(pro)q(cess)g(is)h(executed.)21 b(This)15 b(program)f(will)j(deadlo)q(c)o(k.)166 1458 y(The)10 b(follo)o(wing)h(program)f(is)g(unsafe,)h(and)g(ma)o(y)e(succeed)j(or)e (fail,)h(dep)q(ending)i(on)d(implemen)o(tation.)75 1620 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1676 y(IF)h(\(rank.EQ.0\))123 1733 y(THEN)170 1789 y(CALL)g(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h (tag,)f(comm,)g(ierr\))170 1846 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i (MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g(ierr\))123 1902 y(ELSE)95 b(!)23 b(rank.EQ.1)170 1958 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i (MPI_REAL,)g(0,)h(tag,)f(comm,)g(ierr\))170 2015 y(CALL)h(MPI_RECV\(recvbuf,) d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g(status,)g(ierr\))75 2071 y(END)g(IF)166 2233 y Fy(The)11 b(message)g(sen)o(t)g(b)o(y)g(eac)o(h)g (pro)q(cess)g(has)g(to)f(b)q(e)i(copied)g(out)f(b)q(efore)g(the)g(send)h(op)q (eration)f(returns)75 2290 y(and)k(the)g(receiv)o(e)h(op)q(eration)f(starts.) k(F)l(or)14 b(the)h(program)f(to)g(complete,)h(it)h(is)f(necessary)g(that)g (at)f(least)75 2346 y(one)i(of)g(the)h(t)o(w)o(o)d(messages)i(sen)o(t)g(is)h (bu\013ered.)23 b(Th)o(us,)16 b(this)h(program)e(can)i(succeed)g(only)g(if)g (the)f(MPI)75 2403 y(implemen)o(tation)e(bu\013ers)f(messages,)g(and)g(the)g (comm)o(unication)h(system)f(has)g(su\016cien)o(t)h(bu\013er)f(space)75 2459 y(to)i(bu\013er)g Ft(count)h Fy(w)o(ords)f(of)f(data.)75 2602 y Fz(1.5)59 b(Data)20 b(T)-5 b(yp)r(e)19 b(Matching)75 2704 y Fy(One)d(can)f(think)h(of)f(message)g(transmission)g(as)g(consisting)h (of)f(three)g(phases:)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 10 bop 75 -100 a Fy(10)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)131 49 y Fy(1.)22 b(Data)14 b(is)h(pulled)j(out)d(of)f(the)i(send)f(bu\013er)h(and)f(a)g(message)g(is)g (assem)o(bled)131 142 y(2.)22 b(A)15 b(message)g(is)g(transferred)g(from)f (sender)i(to)f(receiv)o(er)131 235 y(3.)22 b(Data)14 b(is)h(pulled)j(from)c (the)h(incoming)i(message)e(and)g(disassem)o(bled)i(in)o(to)e(the)g(receiv)o (e)h(bu\013er)166 326 y(T)o(yp)q(e)k(matc)o(hing)f(has)h(to)e(b)q(e)j(observ) o(ed)e(at)g(eac)o(h)g(of)h(these)f(three)h(phases:)29 b(The)19 b(t)o(yp)q(e)h(of)f(eac)o(h)75 383 y(v)m(ariable)i(in)f(the)g(sender)g (bu\013er)g(has)g(to)f(matc)o(h)g(the)g(t)o(yp)q(e)h(sp)q(eci\014ed)i(for)d (that)g(en)o(try)g(b)o(y)h(the)f(send)75 439 y(op)q(eration;)d(the)f(t)o(yp)q (e)h(sp)q(eci\014ed)i(b)o(y)e(the)f(send)i(op)q(eration)e(has)h(to)f(matc)o (h)g(the)h(t)o(yp)q(e)f(sp)q(eci\014ed)j(b)o(y)e(the)75 496 y(receiv)o(e)g(op)q(eration;)e(and)h(the)g(t)o(yp)q(e)f(of)h(eac)o(h)f(v)m (ariable)i(in)g(the)e(receiv)o(e)i(bu\013er)f(has)f(to)g(matc)o(h)g(the)h(t)o (yp)q(e)75 552 y(sp)q(eci\014ed)f(for)e(that)g(en)o(try)g(b)o(y)g(the)g (receiv)o(e)h(op)q(eration.)20 b(A)12 b(program)f(that)h(fails)h(to)e(observ) o(e)h(these)h(three)75 608 y(rules)j(is)g(erroneous.)166 665 y(T)l(o)f(de\014ne)i(t)o(yp)q(e)f(matc)o(hing)g(more)f(precisely)l(,)j(w)o(e) d(need)i(to)e(deal)h(with)g(t)o(w)o(o)f(issues:)22 b(matc)o(hing)15 b(of)75 721 y(t)o(yp)q(es)e(of)g(the)h(host)f(language)g(with)h(t)o(yp)q(es)g (sp)q(eci\014ed)h(in)f(comm)o(unication)h(op)q(erations;)e(and)h(matc)o(hing) 75 778 y(of)h(t)o(yp)q(es)g(at)g(sender)g(and)h(receiv)o(er.)166 834 y(A)h(t)o(yp)q(e)f(sp)q(eci\014ed)j(for)d(an)h(en)o(try)f(b)o(y)h(a)f (send)h(op)q(eration)g(matc)o(hes)f(the)h(t)o(yp)q(e)g(sp)q(eci\014ed)i(for)d (that)75 891 y(en)o(try)11 b(b)o(y)h(a)f(receiv)o(e)h(op)q(eration)g(if)g(b)q (oth)f(op)q(erations)h(use)g(iden)o(tical)h(names:)36 b Ft(MPI)p 1500 891 14 2 v 16 w(INTEGER)12 b Fy(matc)o(hes)75 947 y Ft(MPI)p 160 947 V 16 w(INTEGER)p Fy(,)j Ft(MPI)p 475 947 V 16 w(REAL)h Fy(matc)o(hes)30 b Ft(MPI)p 891 947 V 15 w(REAL)p Fy(,)16 b(and)f(so)g(on.) 166 1004 y(The)k(t)o(yp)q(e)g(of)f(a)g(v)m(ariable)i(in)g(a)e(host)h(program) e(matc)o(hes)h(the)h(t)o(yp)q(e)g(sp)q(eci\014ed)i(in)e(the)g(comm)o(u-)75 1060 y(nication)g(op)q(eration)g(if)f(the)h(datat)o(yp)q(e)f(name)g(used)h(b) o(y)f(that)f(op)q(eration)i(corresp)q(onds)f(to)g(the)g(basic)75 1117 y(t)o(yp)q(e)i(of)g(the)g(host)g(program)f(v)m(ariable:)31 b(an)20 b(en)o(try)f(with)i(t)o(yp)q(e)f(name)40 b Ft(MPI)p 1449 1117 V 16 w(INTEGER)21 b Fy(matc)o(hes)e(a)75 1173 y(F)l(ortran)c(v)m (ariable)j(of)f(t)o(yp)q(e)f Fr(INTEGER)p Fy(,)g(an)g(en)o(try)g(with)h(t)o (yp)q(e)g(name)g Ft(MPI)p 1368 1173 V 16 w(REAL)g Fy(matc)o(hes)f(a)g(F)l (ortran)75 1229 y(v)m(ariable)j(of)e(t)o(yp)q(e)g Fr(REAL)p Fy(,)f(and)i(so)f(on.)26 b(There)18 b(is)f(one)h(exception)g(to)f(this)h (last)f(rule:)25 b(An)18 b(en)o(try)f(with)75 1286 y(t)o(yp)q(e)h(name)37 b Ft(MPI)p 408 1286 V 15 w(BYTE)19 b Fy(can)f(b)q(e)h(used)g(to)e(matc)o(h)h (an)o(y)g(b)o(yte)g(of)g(storage)e(\(on)i(a)g(b)o(yte-addressable)75 1342 y(mac)o(hine\),)d(irresp)q(ectiv)o(e)i(of)d(the)h(datat)o(yp)q(e)g(of)f (the)h(v)m(ariable)i(that)d(con)o(tains)h(this)g(b)o(yte.)20 b(The)15 b(v)m(alue)h(of)75 1399 y(the)f(message)g(en)o(try)g(will)i(b)q(e)f (the)f(binary)h(v)m(alue)g(of)f(the)g(corresp)q(onding)h(b)o(yte)f(in)h (memory)l(.)166 1455 y(W)l(e)f(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)143 1547 y Fn(\017)23 b Fy(Comm)o(unication)11 b(of)f(t)o(yp)q(ed)i(v)m(alues)g (\(e.g.,)e(with)h(datat)o(yp)q(e)g(di\013eren)o(t)g(from)21 b Ft(MPI)p 1586 1547 V 16 w(BYTE)p Fy(\),)11 b(where)189 1603 y(the)k(datat)o(yp)q(es)f(of)g(the)h(corresp)q(onding)h(en)o(tries)f(in)h (the)f(sender)g(program,)e(in)j(the)f(send)g(call,)h(in)189 1659 y(the)f(receiv)o(e)h(call)g(and)g(in)g(the)f(receiv)o(er)h(program)e (should)i(all)g(matc)o(h.)143 1752 y Fn(\017)23 b Fy(Comm)o(unication)16 b(of)g(un)o(t)o(yp)q(ed)h(v)m(alues)g(\(e.g.,)e(of)g(datat)o(yp)q(e)h Ft(MPI)p 1330 1752 V 16 w(BYTE)p Fy(\),)g(where)g(b)q(oth)g(sender)189 1809 y(and)g(receiv)o(er)h(use)f(the)g(datat)o(yp)q(e)g Ft(MPI)p 881 1809 V 15 w(BYTE)p Fy(.)h(In)f(this)h(case,)f(there)g(are)g(no)f (requiremen)o(ts)i(on)189 1865 y(the)d(t)o(yp)q(es)h(of)f(the)g(corresp)q (onding)h(en)o(tries)g(in)h(the)e(sender)h(and)g(the)f(receiv)o(er)h (programs,)e(nor)h(is)189 1922 y(it)h(required)h(that)f(they)g(b)q(e)h(the)f (same.)166 2013 y(The)g(follo)o(wing)h(examples)g(illustrate)h(these)e(t)o(w) o(o)f(cases.)166 2069 y(First)h(program:)75 2161 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2217 y(IF\(rank.EQ.0\))123 2274 y(THEN)170 2330 y(CALL)h(MPI_SEND\(a\(1\),)e(10,)h(MPI_REAL,)g(1,)g (tag,)h(comm,)f(ierr\))123 2386 y(ELSE)170 2443 y(CALL)h(MPI_RECV\(a\(1\),)e (15,)h(MPI_REAL,)g(0,)g(tag,)h(comm,)f(status,)g(ierr\))75 2499 y(END)g(IF)166 2591 y Fy(This)c(co)q(de)f(is)h(correct)e(if)i(b)q(oth)f (sender)h(and)f(receiv)o(er)h(programs)d(ha)o(v)o(e)i(allo)q(cated)h (consecutiv)o(e)75 2647 y(storage)14 b(for)h(ten)g(real)g(n)o(um)o(b)q(ers,)g (starting)g(from)f Ft(a\(1\))p Fy(.)166 2704 y(Second)i(program:)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 11 bop 75 -100 a Fu(1.5.)34 b(D)o(A)l(T)l(A)15 b(TYPE)g(MA)l(TCHING)1105 b Fy(11)75 49 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 106 y(IF\(rank.EQ.0\))123 162 y(THEN)170 219 y(CALL)h(MPI_SEND\(a\(1\),)e (10,)h(MPI_REAL,)g(1,)g(tag,)h(comm,)f(ierr\))123 275 y(ELSE)170 332 y(CALL)h(MPI_RECV\(a\(1\),)e(40,)h(MPI_BYTE,)g(0,)g(tag,)h(comm,)f (status,)g(ierr\))75 388 y(END)g(IF)166 486 y Fy(This)d(co)q(de)g(is)f (erroneous,)h(since)g(sender)g(and)g(receiv)o(er)g(do)f(not)f(pro)o(vide)i (matc)o(hing)f(datat)o(yp)q(e)75 542 y(argumen)o(ts.)166 600 y(Third)d(program:)75 699 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h (ierr\))75 755 y(IF\(rank.EQ.0\))123 812 y(THEN)170 868 y(CALL)h (MPI_SEND\(a\(1\),)e(40,)h(MPI_BYTE,)g(1,)g(tag,)h(comm,)f(ierr\))123 925 y(ELSE)170 981 y(CALL)h(MPI_RECV\(a\(1\),)e(60,)h(MPI_BYTE,)g(0,)g(tag,)h (comm,)f(status,)g(ierr\))75 1037 y(END)g(IF)166 1135 y Fy(This)16 b(co)q(de)f(is)h(correct,)f(irresp)q(ectiv)o(e)h(of)f(the)g(t)o(yp)q(e)h(of)e Ft(a\(1\))g Fy(and)i(follo)o(wing)g(v)m(ariables)g(in)g(store.)189 1247 y Fp(A)n(dvic)n(e)11 b(to)h(users.)56 b Fy(If)11 b(a)g(bu\013er)f(of)h (t)o(yp)q(e)f Fo(MPI)p 974 1247 13 2 v 15 w(BYTE)g Fy(is)h(passed)g(as)f(an)h (argumen)o(t)f(to)g Ft(MPI)p 1732 1247 14 2 v 16 w(SEND)p Fy(,)69 b Fm(\()189 1303 y Fy(then)14 b(MPI)g(will)h(send)g(the)f(data)f(stored)g(at) g(con)o(tiguous)h(lo)q(cations,)h(starting)e(from)g(the)h(address)189 1360 y(indicated)22 b(b)o(y)e(the)h Ft(buf)g Fy(argumen)o(t.)35 b(This)21 b(ma)o(y)f(ha)o(v)o(e)g(unexp)q(ected)i(results)f(when)g(the)g (data)189 1416 y(la)o(y)o(out)12 b(is)i(not)e(as)h(a)f(casual)i(user)f(w)o (ould)g(exp)q(ect)h(it)f(to)g(b)q(e:)19 b(F)l(or)12 b(example,)i(some)f (compilers)h(pass)189 1473 y(a)e(F)l(ortran)f Fo(CHARA)o(CTER)g Fy(argumen)o(t)h(as)g(a)g(structure)g(that)g(con)o(tains)g(the)h(c)o (haracter)f(length)h(and)189 1529 y(a)k(p)q(oin)o(ter)i(to)e(the)h(actual)g (string.)29 b(In)18 b(suc)o(h)h(en)o(vironmen)o(t,)f(sending)h(or)f (receiving)h(a)f(F)l(ortran)189 1585 y Fo(CHARA)o(CTER)f Fy(v)m(ariable)j (using)g(the)f Fo(MPI)p 907 1585 13 2 v 15 w(BYTE)f Fy(will)j(not)e(ha)o(v)o (e)f(the)i(an)o(ticipated)g(result.)32 b(F)l(or)189 1642 y(this)15 b(reason,)f(the)h(user)g(is)g(advised)h(to)e(use)h(t)o(yp)q(ed)g(comm)o (unication)h(whenev)o(er)f(p)q(ossible.)22 b(\()p Fp(End)189 1698 y(of)16 b(advic)n(e)g(to)h(users.)p Fy(\))75 1826 y Fq(1.5.1)49 b(T)l(yp)q(e)17 b Fo(MPI)p 417 1826 V 14 w(CHARA)o(CTER)75 1914 y Fy(The)23 b(t)o(yp)q(e)11 b Fo(MPI)p 351 1914 V 14 w(CHARA)o(CTER)f Fy(matc)o(hes)h(one)g(c)o(haracter)g(of)f(a)h(F)l(ortran)f(v)m(ariable)j(of)e (t)o(yp)q(e)g Fo(CHARA)o(CTER)p Fy(.)67 b Fm(\()75 1970 y Fy(F)l(ortran)12 b(v)m(ariables)j(of)e(t)o(yp)q(e)h Fo(CHARA)o(CTER)e Fy(or)h(substrings)h (are)f(transferred)g(as)g(if)h(they)g(w)o(ere)f(arra)o(ys)f(of)75 2027 y(c)o(haracters.)18 b(The)c Ft(buf)f Fy(argumen)o(t)g(should)h(b)q(e)f (a)g(reference)h(to)e(a)h(string)g(or)f(substring)i(that)e(starts)g(with)75 2083 y(the)j(\014rst)g(c)o(haracter)g(of)f(the)i(comm)o(unication)g (bu\013er.)j(This)d(is)g(illustrated)g(in)g(the)g(example)g(b)q(elo)o(w.)75 2195 y Fr(CHARACTER*10)22 b(a)75 2252 y(CHARACTER*10)g(b)75 2365 y(CALL)h(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2421 y(IF\(rank.EQ.0\))123 2478 y(THEN)170 2534 y(CALL)h(MPI_SEND\(a,)e(5,)i (MPI_CHARACTER,)e(1,)h(tag,)h(comm,)f(ierr\))123 2591 y(ELSE)170 2647 y(CALL)h(MPI_RECV\(b\(6:10\),)d(5,)j(MPI_CHARACTER,)e(0,)h(tag,)h(comm,) f(status,)g(ierr\))75 2704 y(END)g(IF)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 12 bop 75 -100 a Fy(12)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fy(The)e(last)h(\014v)o(e)f(c)o (haracters)g(of)f(string)i Fo(b)f Fy(at)g(pro)q(cess)g(1)g(are)g(replaced)i (b)o(y)e(the)g(\014rst)g(\014v)o(e)h(c)o(haracters)75 106 y(of)g(string)g Fo(a)g Fy(at)g(pro)q(cess)g(0.)166 245 y Fl(Implemen)o(tati)o(on)10 b(note:)33 b Fs(Some)10 b(compilers)h(pass)h(F)m(ortran)f Fo(CHARA)o(CTER)f Fs(argumen)o(ts)h(as)h(a)f(structure)75 301 y(with)16 b(a)f(\014eld)h(length) g(and)g(a)g(p)q(oin)o(ter)g(to)g(the)g(actual)g(string.)24 b(In)16 b(suc)o(h)h(en)o(vironmen)o(t,)e(the)i(MPI)f(call)f(need)i(to)75 358 y(dereference)g(the)d(p)q(oin)o(ter)g(in)g(order)g(to)g(reac)o(h)h(the)f (string.)75 558 y Ft(Alternative)i(De\014nition)75 644 y Fy(The)h(t)o(yp)q(e) g Fo(MPI)p 351 644 13 2 v 14 w(CHARA)o(CTER)e Fy(matc)o(hes)h(a)h(F)l(ortran) f(v)m(ariable)i(of)e(t)o(yp)q(e)h Fo(CHARA)o(CTER)e Fy(or)h(a)h(substring)75 700 y(of)h(arbitrary)f(length.)30 b(Th)o(us,)19 b(suc)o(h)f(c)o(haracter)g (is)h(sen)o(t)f(b)o(y)g(sp)q(ecifying)i Ft(t)o(yp)q(e)f(=)g(MPI)p 1596 700 14 2 v 16 w(CHARA)o(CTER)75 757 y Fy(and)c Ft(count)i(=)e(1)p Fy(.)20 b(This)c(is)f(illustrated)i(in)f(the)f(example)h(b)q(elo)o(w.)75 846 y Fr(CHARACTER*10)22 b(a)75 902 y(CHARACTER*10)g(b)75 1015 y(CALL)h(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1072 y(IF\(rank.EQ.0\))123 1128 y(THEN)170 1184 y(CALL)h(MPI_SEND\(a\(1:5\),)d(1,)j(MPI_CHARACTER,)e(1,) i(tag,)f(comm,)g(ierr\))123 1241 y(ELSE)170 1297 y(CALL)h (MPI_RECV\(b\(6:10\),)d(1,)j(MPI_CHARACTER,)e(0,)h(tag,)h(comm,)f(status,)g (ierr\))75 1354 y(END)g(IF)166 1443 y Fy(The)14 b(last)h(\014v)o(e)f(c)o (haracters)g(of)f(string)i Fo(b)f Fy(at)g(pro)q(cess)g(1)g(are)g(replaced)i (b)o(y)e(the)g(\014rst)g(\014v)o(e)h(c)o(haracters)75 1499 y(of)g(string)g Fo(a)g Fy(at)g(pro)q(cess)g(0.)166 1556 y(Mismatc)o(hes)f(b)q (et)o(w)o(een)g(the)g(length)g(of)f(an)h(incoming)h(c)o(haracter)e(and)h(the) g(length)h(of)e(a)h(c)o(haracter)75 1612 y(in)21 b(the)f(receiv)o(e)g (bu\013er)g(are)g(handled)h(as)e(in)i(F)l(ortran)e(c)o(haracter)g(assignmen)o (ts)g(\(blank)i(padding)f(or)75 1669 y(truncation\).)166 1801 y Fl(Implemen)o(tati)o(on)13 b(note:)37 b Fs(F)m(ortran)14 b(passes)i(information)c(on)i(c)o(haracter)h(length)g(together)g(with)f(c)o (har-)75 1851 y(acter)j(argumen)o(ts.)24 b(Some)14 b(compilers)h(pass)i(the)f (c)o(haracter)i(length)e(via)f(an)h(additional)e(\\hidden")i(argumen)o(t;)75 1901 y(other)d(pass)f(a)g(c)o(haracter)i(argumen)o(t)d(as)h(a)g(structure)i (with)e(a)g(length)g(\014eld)g(and)g(a)g(p)q(oin)o(ter)h(to)f(the)h(actual)e (string.)75 1950 y(The)k(MPI)f(send)h(call)f(needs)h(to)f(access)i(the)f (length)f(information)e(for)i(the)h Fo(buf)f Fs(c)o(haracter)i(argumen)o(t)d (and)h(send)75 2000 y(it)f(together)h(with)e(the)i(actual)e(string;)h(the)h (MPI)f(receiv)o(e)h(call)e(needs)j(to)e(compare)f(the)h(length)g(of)g(the)g (incoming)75 2050 y(c)o(haracter)i(with)f(the)g(length)g(of)f(the)i Fo(buf)f Fs(c)o(haracter)i(argumen)o(t)c(and)i(pad)g(or)g(truncate,)g(as)g (is)g(the)h(case.)166 2107 y(F)m(or)e(non)o(blo)q(c)o(king)f(comm)o (unication)e(or)k(p)q(ersisten)o(t)h(handles,)e(the)h(comm)o(unication)c (initiating)i(call)h(needs)75 2163 y(to)e(retriev)o(e)i(the)f(c)o(haracter)h (length)f(information)c(and)k(bu\013er)g(it)f(in)o(ternally)m(.)16 b(\(Since)d(the)f(storage)g(for)f(the)h(length)75 2219 y(\014eld)j(in)f(a)h (c)o(haracter)h(argumen)o(t)e(is)g(not)h(explicitly)f(allo)q(cated)g(b)o(y)h (the)g(F)m(ortran)g(program,)e(the)i(compiler)f(ma)o(y)75 2276 y(reuse)j(this)f(storage)g(ev)o(en)h(if)e(the)h(user)h(do)q(es)g(not)e(mo)q (dify)f(the)i(corresp)q(onding)h(actual)e(c)o(haracter)i(argumen)o(t.\))75 2332 y(E\013ectiv)o(ely)m(,)10 b(the)h(MPI)f(call)f(has)i(to)e(compute)h(and) g(bu\013er)h(a)e(\\deriv)o(ed)h(datat)o(yp)q(e")g(that)g(describ)q(es)j(this) d(c)o(haracter)75 2389 y(v)n(ariable.)166 2604 y Fl(Discussion:)55 b Fs(Please)19 b(advise)h(on)e(a)h(c)o(hoice)h(b)q(et)o(w)o(een)g(the)g(t)o (w)o(o)f(de\014nitions.)33 b(The)20 b(second)g(is)f(more)75 2654 y(elegan)o(t,)12 b(and)f(more)g(in)g(tune)h(with)g(F)m(ortran)f(seman)o (tics)h(for)f(c)o(haracter)i(strings)f(\(they)g(are)h(scalars...\).)j(The)c (\014rst)75 2704 y(is)i(closer)g(to)g(the)h(C)e(de\014nition)h(and)g(easier)g (to)g(implemen)o(t.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 13 bop 75 -100 a Fu(1.6.)34 b(D)o(A)l(T)l(A)15 b(CONVERSION)1197 b Fy(13)166 49 y Fs(Please)17 b(also)e(c)o(hec)o(k)i(for)e(hidden)h(pitfalls) f(with)g(y)o(our)h(compiler)e(of)i(c)o(hoice.)24 b(Do)q(es)17 b(ev)o(erything)f(w)o(ork)f(OK)75 99 y(with)f(substrings?)19 b(with)13 b(arra)o(ys)h(of)f(c)o(haracters?)166 156 y(The)i(de\014nitions)g (also)f(extend)i(to)f(the)h(construction)f(of)g(deriv)o(ed)g(datat)o(yp)q (es.)22 b(One)15 b(needs)i(to)d(de\014ne)i(the)75 213 y(exten)o(t)d(of)e(a)g (c)o(haracter)i(correctly)g(\(one)f(b)o(yte,)g(in)f(the)h(\014rst)h(in)o (terpretation,)f(string-length)f(b)o(ytes)h(in)g(the)g(second)75 269 y(one\);)j(one)g(needs)h(to)f(retriev)o(e)h(the)f(string)g(lo)q(cation)f (and)g(\(in)h(the)g(second)h(in)o(terpretation\))f(the)h(string)e(length,)75 326 y(in)f(order)i(to)f(build)f(the)h(righ)o(t)g(deriv)o(ed)g(datat)o(yp)q(e) g(represen)o(tation.)75 556 y Fz(1.6)59 b(Data)20 b(conversion)75 659 y Fy(One)c(of)f(the)g(goals)g(of)g(MPI)g(is)h(to)e(supp)q(ort)i(parallel) g(computations)f(across)g(heterogeneous)g(en)o(viron-)75 715 y(men)o(ts.)20 b(Comm)o(unication)15 b(in)h(a)f(heterogeneous)g(en)o (vironmen)o(t)h(ma)o(y)e(require)i(data)f(con)o(v)o(ersions.)166 772 y(W)l(e)g(use)h(the)f(follo)o(wing)h(terminology:)75 869 y Fx(t)o(yp)q(e)h(con)o(v)o(ersion)22 b Fy(c)o(hanges)16 b(the)h(datat)o(yp)q (e)f(of)g(a)g(v)m(alue,)h(e.g.)23 b(b)o(y)17 b(rounding)g(a)f Ft(REAL)h Fy(to)f(an)g Ft(INTE-)189 926 y(GER)p Fy(.)75 1022 y Fx(represen)o(tation)h(con)o(v)o(ersion)22 b Fy(c)o(hanges)16 b(the)h(binary)g(represen)o(tation)f(of)g(a)g(v)m(alue,)h(e.g.)23 b(from)16 b(Hex)189 1079 y(\015oating)f(p)q(oin)o(t)g(to)g(IEEE)g(\015oating) h(p)q(oin)o(t.)166 1176 y(The)d(t)o(yp)q(e)g(matc)o(hing)g(rules)g(imply)h (that)e(MPI)h(comm)o(unication)h(nev)o(er)f(en)o(tails)g(t)o(yp)q(e)g(con)o (v)o(ersion.)75 1232 y(On)20 b(the)f(other)f(hand,)i(MPI)f(requires)h(that)e (a)h(represen)o(tation)g(con)o(v)o(ersion)g(b)q(e)h(p)q(erformed)f(when)g(a) 75 1289 y(t)o(yp)q(ed)i(v)m(alue)h(is)f(transferred)f(across)g(en)o(vironmen) o(ts)h(that)f(use)h(di\013eren)o(t)f(represen)o(tations)h(for)f(the)75 1345 y(datat)o(yp)q(e)g(of)g(this)h(v)m(alue.)38 b(MPI)20 b(do)q(es)h(not)f (sp)q(ecify)i(rules)g(for)d(represen)o(tation)i(con)o(v)o(ersion.)36 b(Suc)o(h)75 1401 y(con)o(v)o(ersion)21 b(is)g(exp)q(ected)i(to)d(preserv)o (e)h(in)o(teger,)h(logical)g(or)e(c)o(haracter)h(v)m(alues,)i(and)e(to)f(con) o(v)o(ert)g(a)75 1458 y(\015oating)15 b(p)q(oin)o(t)h(v)m(alue)g(to)f(the)g (nearest)g(v)m(alue)h(that)f(can)g(represen)o(ted)h(on)f(the)g(target)f (system.)166 1515 y(Ov)o(er\015o)o(w)c(and)h(under\015o)o(w)g(exceptions)h (ma)o(y)e(o)q(ccur)i(during)f(\015oating)g(p)q(oin)o(t)g(con)o(v)o(ersions;)h (con)o(v)o(er-)75 1572 y(sion)i(of)e(in)o(tegers)i(or)e(c)o(haracters)h(ma)o (y)f(also)h(lead)h(to)f(exceptions)h(when)g(a)e(v)m(alue)j(that)d(can)h (represen)o(ted)75 1628 y(in)18 b(one)g(system)f(cannot)g(b)q(e)h(represen)o (ted)g(in)h(the)e(other)g(system.)26 b(An)18 b(exception)h(o)q(ccurring)f (during)75 1684 y(represen)o(tation)f(con)o(v)o(ersion)f(results)h(in)h(a)e (failure)i(of)e(the)g(comm)o(unication;)i(an)e(error)g(o)q(ccurs)h(either)75 1741 y(in)f(the)f(send)h(op)q(eration,)f(or)g(the)g(receiv)o(e)h(op)q (eration,)f(or)g(b)q(oth.)166 1798 y(If)22 b(a)f(v)m(alue)i(sen)o(t)e(in)h(a) f(message)g(is)h(un)o(t)o(yp)q(ed)h(\(i.e.,)f(of)f(t)o(yp)q(e)h Ft(MPI)p 1370 1798 14 2 v 15 w(BYTE)p Fy(\),)g(then)g(the)f(binary)75 1855 y(represen)o(tation)15 b(of)f(the)h(b)o(yte)g(stored)f(at)g(the)h (receiv)o(er)h(is)f(iden)o(tical)i(to)d(the)h(binary)h(represen)o(tation)e (of)75 1911 y(the)f(b)o(yte)g(loaded)h(at)e(the)h(sender.)20 b(This)14 b(holds)f(true,)g(whether)g(sender)h(and)f(receiv)o(er)h(run)f(in)h (the)f(same)75 1967 y(or)i(in)h(distinct)g(en)o(vironmen)o(ts;)f(no)g (represen)o(tation)h(con)o(v)o(ersion)f(is)h(required.)166 2025 y(Note)d(that)g(no)g(con)o(v)o(ersion)h(ev)o(er)f(o)q(ccurs)h(when)g(an) f(MPI)g(program)g(executes)h(in)g(a)f(homogeneous)75 2081 y(system,)i(where)h (all)g(pro)q(cesses)g(run)g(in)g(the)g(same)f(en)o(vironmen)o(t.)22 b(Also)16 b(note)f(the)h(di\013eren)o(t)f(b)q(eha)o(vior)75 2138 y(of)e Ft(MPI)p 210 2138 V 15 w(BYTE)h Fy(and)f(of)g Ft(MPI)p 575 2138 V 15 w(CHAR)p Fy(.)h(A)f(bu\013er)g(descriptor)g(en)o(try)g(with)g (datat)o(yp)q(e)g(of)f Ft(MPI)p 1660 2138 V 16 w(CHAR)i Fy(can)75 2194 y(only)f(matc)o(h)f(a)g(C)g(v)m(ariable)i(of)e(t)o(yp)q(e)h Fr(CHAR)p Fy(;)e(and)i(represen)o(tation)g(con)o(v)o(ersion)f(ma)o(y)g(o)q (ccur)h(when)g(v)m(alues)75 2250 y(of)g(t)o(yp)q(e)h Ft(MPI)p 311 2250 V 15 w(CHAR)h Fy(are)e(transferred.,)g(e.g.,)f(from)h(an)g(EBCDIC)h (enco)q(ding)h(to)e(an)g(ASCI)q(I)i(enco)q(ding.)166 2308 y(Consider)h(the)f (previous)h(three)f(examples)h(on)f(page)g(10.)166 2365 y(The)c(\014rst)f (program)f(is)i(correct,)g(assuming)g(b)q(oth)f(sender)i(and)e(receiv)o(er)i (declared)f(ten)g(consecutiv)o(e)75 2421 y(real)20 b(v)m(ariables)h(in)g (storage)e(starting)g(at)g Ft(a\(1\))p Fy(.)33 b(If)20 b(the)g(sender)g(and)g (receiv)o(er)h(execute)f(in)h(di\013eren)o(t)75 2478 y(en)o(vironmen)o(ts,)12 b(then)f(the)g(ten)f(real)i(v)m(alues)f(that)g(are)f(fetc)o(hed)h(from)f(the) h(send)h(bu\013er)e(will)j(b)q(e)e(con)o(v)o(erted)75 2534 y(to)g(the)h(represen)o(tation)f(for)g(reals)h(on)f(the)h(receiv)o(er)g(site) g(b)q(efore)g(they)g(are)f(stored)g(in)h(the)g(receiv)o(e)g(bu\013er.)75 2591 y(While)19 b(the)e(n)o(um)o(b)q(er)h(of)f(real)g(elemen)o(ts)h(fetc)o (hed)g(from)f(the)g(send)h(bu\013er)f(equal)i(the)e(n)o(um)o(b)q(er)h(of)f (real)75 2647 y(elemen)o(ts)d(stored)e(in)i(the)f(receiv)o(e)h(bu\013er,)f (the)g(n)o(um)o(b)q(er)g(of)g(b)o(ytes)g(stored)f(need)i(not)f(equal)h(the)f (n)o(um)o(b)q(er)75 2704 y(of)f(b)o(ytes)g(loaded:)19 b(e.g.)g(the)12 b(sender)h(ma)o(y)e(use)i(a)f(four)g(b)o(yte)g(represen)o(tation)h(and)f(the) g(receiv)o(er)i(an)e(eigh)o(t)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 14 bop 75 -100 a Fy(14)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fy(b)o(yte)h(represen)o(tation)h(for) f(reals.)27 b(If)18 b(the)g(send)g(or)f(receiv)o(e)h(bu\013er)g(do)f(not)h (con)o(tain)f(ten)h(consecutiv)o(e)75 106 y(real)e(v)m(ariables,)g(then)f (the)g(program)g(is)g(erroneous,)g(and)g(its)h(b)q(eha)o(vior)f(is)h (unde\014ned.)166 162 y(The)f(second)h(program)e(is)i(erroneous,)f(and)g(its) g(b)q(eha)o(vior)h(is)g(unde\014ned.)166 219 y(The)e(third)g(program)e(is)i (correct.)19 b(The)13 b(exact)g(same)g(sequence)i(of)e(fort)o(y)f(b)o(ytes)h (that)g(w)o(ere)g(loaded)75 275 y(from)18 b(the)h(send)h(bu\013er)f(will)i(b) q(e)e(stored)g(in)h(the)f(receiv)o(e)h(bu\013er,)f(ev)o(en)h(if)f(sender)h (and)f(receiv)o(er)h(run)75 332 y(in)e(a)g(di\013eren)o(t)f(en)o(vironmen)o (t.)28 b(The)18 b(message)f(sen)o(t)g(has)h(exactly)f(the)h(same)f(length)i (\(in)f(b)o(ytes\))f(and)75 388 y(the)f(same)f(binary)h(represen)o(tation)f (as)g(the)h(message)f(receiv)o(ed.)22 b(If)16 b(the)f(v)m(ariables)i(in)f (the)g(send)g(bu\013er)75 444 y(are)f(of)h(di\013eren)o(t)g(t)o(yp)q(es)g (from)f(the)h(v)m(ariables)h(in)f(the)g(receiv)o(e)h(bu\013er,)e(or)h(they)g (are)f(of)h(the)f(same)h(t)o(yp)q(e)75 501 y(but)h(di\013eren)o(t)g(data)g (represen)o(tations)f(are)h(used,)h(then)f(the)g(bits)h(stored)e(in)i(the)f (receiv)o(e)h(bu\013er)f(ma)o(y)75 557 y(enco)q(de)f(v)m(alues)h(that)d(are)h (di\013eren)o(t)g(from)g(the)g(v)m(alues)h(they)g(enco)q(ded)g(in)g(the)g (send)f(bu\013er.)166 614 y(Data)f(represen)o(tation)i(con)o(v)o(ersion)f (also)h(applies)h(to)e(the)g(en)o(v)o(elop)q(e)i(of)e(a)g(message:)20 b(source,)15 b(des-)75 670 y(tination)h(and)f(tag)f(are)h(all)h(in)o(tegers)g (that)e(ma)o(y)h(need)h(to)e(b)q(e)i(con)o(v)o(erted.)166 803 y Fl(Implemen)o(tati)o(on)11 b(note:)36 b Fs(The)13 b(curren)o(t)i (de\014nition)e(do)q(es)h(not)f(require)h(messages)g(to)f(carry)h(data)e(t)o (yp)q(e)75 853 y(information.)j(A)e(message)g(can)g(b)q(e)g(comp)q(osed)g (and)g(sen)o(t)g(using)g(only)f(the)i(information)c(pro)o(vided)j(b)o(y)f (the)i(send)75 902 y(call,)k(and)g(can)h(b)q(e)g(receiv)o(ed)h(and)e(stored)h (using)f(only)g(the)h(information)d(pro)o(vided)i(b)o(y)g(the)h(receiv)o(e)h (call.)30 b(If)75 952 y(messages)16 b(are)f(sen)o(t)i(b)q(et)o(w)o(een)g (di\013eren)o(t)f(mac)o(hines)f(then)h(one)f(can)h(either)g(use)h(a)e(\\univ) o(ersal")g(data)g(enco)q(ding)75 1002 y(for)g(messages,)f(use)i(kno)o(wledge) f(of)f(the)i(receiv)o(er)g(en)o(vironmen)o(t)e(in)h(order)g(to)g(con)o(v)o (ert)h(data)e(at)h(the)h(sender,)g(or)75 1052 y(use)e(kno)o(wledge)f(of)f (the)i(sender)g(en)o(vironmen)o(t)e(in)h(order)g(to)g(con)o(v)o(ert)h(data)f (at)g(the)g(receiv)o(er.)19 b(In)13 b(either)h(case)g(the)75 1102 y(datat)o(yp)q(e)g(argumen)o(t)f(in)g(the)i(lo)q(cal)e(call)g(can)h(b)q (e)h(used)f(to)g(deriv)o(e)g(the)h(t)o(yp)q(es)g(of)e(the)h(v)n(alues)g (transferred.)166 1158 y(Additional)e(t)o(yp)q(e)j(information)c(can)j(b)q(e) g(carried)h(in)e(messages)h(in)f(order)i(to)f(pro)o(vide)f(b)q(etter)j(error) f(detec-)75 1215 y(tion.)166 1354 y Fy(The)20 b(curren)o(t)g(draft)f(do)q(es) h(not)g(pro)o(vide)g(a)g(mec)o(hanism)h(for)e(in)o(ter-language)i(comm)o (unication:)75 1410 y(messages)d(sen)o(t)h(b)o(y)f(F)l(ortran)g(calls)i (should)f(b)q(e)h(receiv)o(ed)g(b)o(y)e(F)l(ortran)g(calls)h(and)g(messages)g (sen)o(t)f(b)o(y)75 1467 y(C)e(calls)g(should)h(b)q(e)g(receiv)o(ed)g(b)o(y)e (C)h(calls)h(\(this)f(follo)o(ws)f(from)h(the)f(requiremen)o(ts)i(of)e(t)o (yp)q(e)h(matc)o(hing)75 1523 y(and)g(the)h(fact)f(that)f(F)l(ortran)g(and)i (C)f(datat)o(yp)q(es)g(are)g(distinct\).)24 b(If)16 b(in)o(ter-language)h (comm)o(unication)75 1579 y(is)g(needed)i(then)e(one)g(needs)h(to)e(in)o(v)o (ok)o(e)h(C)f(comm)o(unication)i(routines)f(from)f(a)h(F)l(ortran)f(program)f (or)75 1636 y(F)l(ortran)f(comm)o(unication)i(routines)g(from)e(a)h(C)g (program.)189 1739 y Fp(R)n(ationale.)46 b Fy(MPI)17 b(do)q(es)f(not)g (handle)i(in)o(ter-language)f(comm)o(unication)g(b)q(ecause)g(there)g(seem) 189 1796 y(to)h(b)q(e)i(no)f(agreed)h(standards)e(for)h(in)o(ter-language)h (t)o(yp)q(e)f(con)o(v)o(ersions.)32 b(The)20 b(curren)o(t)f(design)189 1852 y(means)13 b(that)g(in)o(ter-language)i(comm)o(unication)f(relies)h(on)f (the)g(v)o(endor)f(sp)q(eci\014c)j(rules)e(for)g(in)o(ter-)189 1909 y(language)h(calls)h(at)f(eac)o(h)g(site.)20 b(\()p Fp(End)c(of)g(r)n (ationale.)p Fy(\))166 2094 y Fl(Discussion:)53 b Fs(Do)13 b(w)o(e)h(w)o(an)o(t)g(to)g(sp)q(ecify)g(\(at)g(least\))h(that)f(messages)g (of)f(t)o(yp)q(e)i Fo(MPI)p 1535 2094 13 2 v 14 w(BYTE)e Fs(can)i(b)q(e)f (sen)o(t)-1903 b Fk(\))75 2151 y Fs(across)14 b(languages?)k(This)13 b(still)g(requires)h(to)f(agree)h(on)g(the)g(corresp)q(ondence)i(b)q(et)o(w)o (een)f(the)f(in)o(teger)g(t)o(yp)q(es)g(used)75 2207 y(for)g(tag)f(and)h (source/destination,)g(and)g(on)f(the)i(represen)o(tation)g(of)f(con)o(text)g (information.)75 2433 y Fz(1.7)59 b(Communication)17 b(Mo)r(des)75 2534 y Fy(The)c(basic)h(send)f(op)q(eration)g(describ)q(ed)i(in)f(section)f (1.2)f(used)h(the)g Fx(standard)g Fy(comm)o(unication)h(mo)q(de.)75 2591 y(In)g(suc)o(h)g(a)f(comm)o(unication)h(mo)q(de,)g(a)f(send)h(op)q (eration)g(can)f(b)q(e)h(started)f(whether)h(or)f(not)g(a)g(matc)o(hing)75 2647 y(receiv)o(e)18 b(w)o(as)e(p)q(osted.)24 b(The)17 b(completion)h(of)e (the)h(send)h(op)q(eration)e(indicates)j(that)d(the)g(message)h(and)75 2704 y(its)f(en)o(v)o(elop)q(e)h(ha)o(v)o(e)e(b)q(een)i(safely)f(stored)f(a)o (w)o(a)o(y)f(and)i(that)f(the)h(sender)g(is)g(free)g(to)f(access)h(and)f(mo)q (dify)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 15 bop 75 -100 a Fu(1.7.)34 b(COMMUNICA)l(TION)16 b(MODES)1048 b Fy(15)75 49 y(the)18 b(sender)h(bu\013er.)30 b(Th)o(us,)18 b(the)h(op)q(eration)f(is)h Fx(lo)q(cally-blo)r(c)o(king)p Fy(:)29 b(it)19 b(do)q(es)f(not)g(return)g(un)o(til)i(the)75 106 y(send)c(op)q(eration)f(has)g(completed)h Fx(lo)q(cally)p Fy(,)i(on)d(the)g(sender)h(side.)166 163 y(The)f(completion)i(of)e(a)g(send)h (op)q(eration)f(giv)o(es)h(no)f(indication)i(that)e(the)g(message)g(w)o(as)g (receiv)o(ed)75 220 y(on)i(the)g(receiv)o(er)h(side.)26 b(A)17 b(blo)q(c)o(king)i(send)e(ma)o(y)g(b)q(e)g(implemen)o(ted)i(so)e(that)f(it)h (returns)g(only)h(after)e(a)75 276 y(matc)o(hing)j(receiv)o(e)h(has)f(b)q (een)h(executed)g(on)f(the)h(receiv)o(er)f(side.)33 b(This)20 b(a)o(v)o(oids)e(the)i(need)g(to)e(bu\013er)75 333 y(message)e(data)f(out)h (of)f(sender)i(or)e(receiv)o(er)i(memory)l(.)22 b(In)17 b(this)f(case)g(the)g (send)h(op)q(eration)f(completes)75 389 y(only)e(after)e(the)i(matc)o(hing)f (receiv)o(e)h(has)g(started)e(executing.)21 b(On)13 b(the)h(other)f(hand,)g (it)h(is)g(also)f(p)q(ossible)75 446 y(for)j(MPI)g(to)f(bu\013er)h(messages,) g(so)g(as)g(to)f(allo)o(w)i(the)f(sender)h(to)e(pro)q(ceed)i(ahead)g(of)e (the)i(receiv)o(er.)23 b(In)75 502 y(suc)o(h)15 b(a)g(case)g(the)g(send)g(op) q(eration)g(ma)o(y)f(complete)i(successfully)h(b)q(efore)e(the)g(message)f(w) o(as)g(receiv)o(ed.)166 560 y(There)h(are)g(t)o(w)o(o)f(additional)j(comm)o (unication)f(mo)q(des:)166 617 y(A)e(send)h(that)e(uses)h(the)g Fx(ready)g Fy(comm)o(unication)g(mo)q(de)h(ma)o(y)e(b)q(e)i(started)e(only)h (if)h(the)f(matc)o(hing)75 674 y(receiv)o(e)g(is)f(already)g(p)q(osted;)g (otherwise)g(the)g(op)q(eration)g(is)g(erroneous)f(and)h(its)g(outcome)f(is)h (unde\014ned.)75 730 y(In)19 b(some)f(systems,)h(this)f(allo)o(ws)h(the)f (remo)o(v)m(al)h(of)f(a)g(hand-shak)o(e)h(op)q(eration)g(that)e(is)i (otherwise)g(re-)75 787 y(quired,)g(and)f(results)g(in)g(impro)o(v)o(ed)g(p)q (erformance.)27 b(The)18 b(completion)h(of)e(the)h(send)g(op)q(eration)g(do)q (es)75 843 y(not)e(dep)q(end)i(on)e(the)g(status)f(of)h(a)g(matc)o(hing)g (receiv)o(e,)h(and)f(merely)h(indicates)h(that)d(the)h(send)h(bu\013er)75 900 y(can)i(b)q(e)g(reused.)30 b(A)19 b(send)g(op)q(eration)g(that)f(uses)g (the)h(ready)f(mo)q(de)h(has)g(the)f(same)g(seman)o(tics)h(as)f(a)75 956 y(standard)h(send)h(op)q(eration;)h(it)f(is)g(merely)g(that)f(it)g(can)h (b)q(e)g(used)g(only)g(in)g(some)f(of)g(the)h(situations)75 1013 y(where)15 b(a)g(regular)h(send)f(can)h(b)q(e)g(used.)166 1070 y(A)i(send)g(that)f(uses)g(the)h Fx(sync)o(hronous)e Fy(mo)q(de)i(can)g (b)q(e)g(started)f(whether)g(or)g(not)h(a)f(matc)o(hing)75 1127 y(receiv)o(e)12 b(w)o(as)e(p)q(osted.)19 b(Ho)o(w)o(ev)o(er,)10 b(the)h(send)h(will)g(complete)g(successfully)h(only)f(if)f(a)g(matc)o(hing)g (receiv)o(e)g(is)75 1183 y(p)q(osted,)h(and)g(the)g(receiv)o(e)g(op)q (eration)g(has)f(started)g(to)g(receiv)o(e)i(the)e(message)g(sen)o(t)h(b)o(y) f(the)h(sync)o(hronous)75 1240 y(send.)29 b(\(I.e.,)19 b(the)f(receiv)o(e)h (has)f(b)q(een)h(p)q(osted,)g(and)g(the)f(incoming)h(message)f(has)g(b)q(een) i(matc)o(hed)e(to)75 1296 y(this)g(p)q(osted)g(receiv)o(e.\))28 b(Th)o(us,)18 b(the)g(completion)g(of)g(a)f(sync)o(hronous)h(send)g(not)f (only)i(indicates)g(that)75 1352 y(the)g(send)h(bu\013er)f(can)g(b)q(e)h (reused,)g(but)f(also)g(indicates)i(that)d(the)i(receiv)o(er)f(has)g(reac)o (hed)h(a)f(certain)75 1409 y(p)q(oin)o(t)e(in)h(its)f(execution,)h(namely)g (that)e(it)h(has)g(started)f(executing)i(the)f(matc)o(hing)g(receiv)o(e.)26 b(If)17 b(b)q(oth)75 1465 y(sends)f(and)h(receiv)o(es)f(are)g(blo)q(c)o(king) h(op)q(erations)f(then)h(the)f(use)g(of)f(the)h(sync)o(hronous)g(mo)q(de)g (pro)o(vides)75 1522 y(sync)o(hronous)h(comm)o(unication)g(seman)o(tics:)23 b(a)16 b(comm)o(unication)h(do)q(es)g(not)g(complete)g(at)f(either)h(end)75 1578 y(b)q(efore)12 b(b)q(oth)g(pro)q(cesses)g(\\attend")f(to)g(the)h(comm)o (unication;)h(the)f(completion)h(of)e(a)h(sync)o(hronous)f(send)75 1635 y(is)16 b(a)f Fx(global)i Fy(ev)o(en)o(t,)d(and)h(the)h(op)q(eration)f (is)h Fx(globally-blo)q(cking)p Fy(.)166 1692 y(Tw)o(o)22 b(additional)j (send)e(functions)h(are)f(pro)o(vided)h(for)f(the)g(t)o(w)o(o)f(additional)i (comm)o(unication)75 1749 y(mo)q(des.)38 b(The)22 b(comm)o(unication)g(mo)q (de)f(is)h(indicated)h(b)o(y)e(a)g(one)h(letter)f(pre\014x:)32 b Ft(R)22 b Fy(for)f(ready)g(and)75 1805 y Ft(S)15 b Fy(for)g(sync)o (hronous.)166 1863 y(Send)h(in)g(ready)f(mo)q(de)75 1968 y Ft(MPI)p 160 1968 14 2 v 16 w(RSEND)h(\(buf,)f(count,)h(datat)o(yp)q(e,)h (dest,)f(tag,)f(comm)m(\))117 2046 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 2124 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in) h(send)h(bu\013er)f(\(in)o(teger\))117 2201 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 2279 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h (\(in)o(teger\))117 2356 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2434 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator) 11 b(\(handle\))75 2559 y Fr(int)23 b(MPI)p 245 2559 15 2 v 17 w(Rsend\(void*)g(buf,)g(int)g(count,)g(MPI)p 1002 2559 V 17 w(Datatype)g(datatype,)g(int)g(dest,)393 2616 y(int)h(tag,)f(MPI)p 683 2616 V 17 w(Comm)g(comm\))75 2704 y(MPI)p 150 2704 V 17 w(RSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)h(IERROR\))-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 16 bop 75 -100 a Fy(16)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)170 49 y Fr()23 b(BUF\(*\))170 106 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(IERROR)166 192 y Fy(Send)16 b(in)g(sync)o(hronous)f(mo)q(de)75 296 y Ft(MPI)p 160 296 14 2 v 16 w(SSEND)h(\(buf,)f(count,)h(datat)o(yp)q(e,)g(dest,)h(tag,) e(comm)m(\))117 373 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 446 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h (bu\013er)f(\(in)o(teger\))117 520 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 593 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h (\(in)o(teger\))117 666 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 739 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))75 864 y Fr(int)23 b(MPI)p 245 864 15 2 v 17 w(Ssend\(void*)g (buf,)g(int)g(count,)g(MPI)p 1002 864 V 17 w(Datatype)g(datatype,)g(int)g (dest,)393 920 y(int)h(tag,)f(MPI)p 683 920 V 17 w(Comm)g(comm\))75 1007 y(MPI)p 150 1007 V 17 w(SSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,) g(COMM,)h(IERROR\))170 1063 y()f(BUF\(*\))170 1120 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(IERROR)166 1206 y Fy(There)16 b(is)g(only)g(one)g(receiv)o(e)h(mo)q(de,)e(whic)o(h)i(can)f(matc)o(h)f(an)o (y)g(of)h(the)f(send)i(mo)q(des.)k(The)16 b(receiv)o(e)75 1263 y(op)q(eration)g(describ)q(ed)h(in)g(the)e(last)h(section)g(is)g Fx(blo)q(c)o(king)p Fy(:)22 b(it)16 b(returns)f(only)h(after)f(the)h(receiv)o (e)g(bu\013er)75 1319 y(con)o(tains)h(the)h(newly)g(receiv)o(ed)g(message.)26 b(It)17 b(is)h Fx(async)o(hronous)p Fy(:)23 b(the)17 b(completion)i(of)d(a)h (receiv)o(e)h(is)75 1375 y(a)e(lo)q(cal)i(op)q(eration,)f(and)f(a)g(receiv)o (e)i(can)f(complete)g(b)q(efore)g(the)f(matc)o(hing)h(send)g(has)f(completed) i(\(of)75 1432 y(course,)d(it)g(can)h(complete)g(only)f(after)g(the)g(matc)o (hing)g(send)h(has)f(started\).)166 1488 y(Comm)o(unication)e(imp)q(oses)g (an)f(order)h(on)f(the)h(ev)o(en)o(ts)f(o)q(ccurring)h(at)f(the)h(comm)o (unicating)g(no)q(des.)75 1545 y(It)i(is)g(alw)o(a)o(ys)e(the)i(case)g(that)e (the)i(completion)h(of)e(a)g(receiv)o(e)i(o)q(ccurs)e(after)g(the)h(start)e (of)h(the)h(matc)o(hing)75 1601 y(send.)29 b(If)19 b(the)f(sync)o(hronous)g (send)h(mo)q(de)f(is)h(used,)g(then)g(it)f(is)h(also)f(the)g(case)g(that)g (the)g(completion)75 1658 y(of)i(a)h(send)g(o)q(ccurs)g(after)f(the)g(start)g (of)g(the)h(matc)o(hing)f(receiv)o(e.)38 b(Of)20 b(course,)i(on)f(eac)o(h)f (pro)q(cess,)i(a)75 1714 y(comm)o(unication)14 b(completes)h(after)d(it)i(is) g(started.)19 b(No)13 b(other)h(order)f(is)h(imp)q(osed)h(b)o(y)e(MPI.)g (E.g.,)g(if)h(the)75 1771 y(standard)h(send)g(mo)q(de)h(is)f(used,)h(the)f (send)h(op)q(eration)f(ma)o(y)f(complete)i(b)q(efore)g(the)f(matc)o(hing)g (receiv)o(e)75 1827 y(w)o(as)f(started.)166 1884 y(In)h(a)f(m)o (ulti-threaded)i(implemen)o(tation)g(of)e(MPI,)h(the)f(system)g(ma)o(y)g (desc)o(hedule)j(a)d(thread)h(that)75 1940 y(is)21 b(blo)q(c)o(k)o(ed)g(on)g (a)f(send)h(or)e(receiv)o(e)j(op)q(eration,)f(and)g(sc)o(hedule)h(another)e (thread)g(for)g(execution)h(in)75 1996 y(the)d(same)g(address)g(space.)28 b(In)19 b(suc)o(h)f(a)f(case)h(it)h(the)f(user)g(resp)q(onsibilit)o(y)i(not)e (to)f(access)h(or)f(mo)q(dify)75 2053 y(a)f(comm)o(unication)h(bu\013er)f(un) o(til)i(the)e(comm)o(unication)h(completes;)g(otherwise)g(the)f(outcome)g(of) g(the)75 2109 y(computation)f(is)h(unde\014ned.)189 2202 y Fp(R)n(ationale.)61 b Fy(W)l(e)19 b(prohibit)h(read)f(accesses)g(to)f(a)g (send)i(bu\013er)e(while)j(it)e(is)g(b)q(een)h(used,)g(ev)o(en)189 2259 y(though)e(the)g(send)g(op)q(eration)h(is)f(not)g(supp)q(osed)h(to)e (alter)h(the)g(con)o(ten)o(t)g(of)f(this)i(bu\013er.)28 b(This)189 2315 y(ma)o(y)17 b(seem)h(more)f(stringen)o(t)g(than)h(necessary)l(,)g(but)g (the)g(additional)h(restriction)g(causes)f(little)189 2372 y(loss)g(of)g(functionalit)o(y)i(and)e(allo)o(ws)h(b)q(etter)f(p)q (erformance)h(on)f(some)g(systems)g({)h(consider)g(the)189 2428 y(case)d(where)g(data)g(transfer)f(is)i(done)f(b)o(y)h(a)e(DMA)h(engine) h(whic)o(h)g(is)g(not)f(cac)o(he-coheren)o(t)h(with)189 2485 y(the)e(main)h(pro)q(cessor.)j(\()p Fp(End)d(of)g(r)n(ationale.)p Fy(\))166 2654 y Fl(Implemen)o(tati)o(on)10 b(note:)34 b Fs(A)13 b(ready)f(send)h(can)f(b)q(e)h(implemen)o(ted)d(as)i(a)g(standard)h(send;)g (in)e(suc)o(h)i(a)f(case)75 2704 y(there)j(will)e(b)q(e)h(no)g(p)q (erformance)g(adv)n(an)o(tage)f(\(or)h(disadv)n(an)o(tage\))f(for)g(the)i (use)f(of)g(ready)g(send.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 17 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(17)166 49 y Fs(A)13 b(standard)h(send)h(can)e(b)q(e)h(implemen)o(ted)e (as)h(a)g(sync)o(hronous)i(send.)j(In)c(suc)o(h)g(a)f(case,)h(no)f(data)h (bu\013ering)75 99 y(is)g(needed.)166 158 y(A)g(p)q(ossible)g(comm)o (unication)c(proto)q(col)k(for)g(the)g(v)n(arious)f(comm)o(unication)e(mo)q (des)i(is)h(outlined)f(b)q(elo)o(w:)166 216 y Fo(ready)i(send)p Fs(:)k(The)14 b(message)g(is)g(sen)o(t)h(as)e(so)q(on)h(as)g(p)q(ossible.)166 274 y Fo(synchronous)f(send:)18 b Fs(The)12 b(sender)h(sends)f(a)f (request-to-send)i(message.)k(The)12 b(receiv)o(er)h(stores)f(this)g (request.)75 324 y(When)j(a)f(matc)o(hing)f(receiv)o(e)j(is)f(p)q(osted,)g (the)h(receiv)o(er)g(sends)g(bac)o(k)f(a)f(p)q(ermission-to-send)h(message,)f (and)g(the)75 374 y(sender)h(no)o(w)f(sends)h(the)g(message.)166 432 y Fo(standa)o(rd)k(send:)27 b Fs(First)18 b(proto)q(col)g(ma)o(y)e(b)q(e) i(used)h(for)e(short)i(messages,)f(and)g(second)h(proto)q(col)e(for)h(long)75 482 y(messages.)166 541 y(Additional)d(con)o(trol)h(messages)h(migh)o(t)e(b)q (e)i(needed)h(for)e(\015o)o(w)g(con)o(trol)h(and)f(error)i(reco)o(v)o(ery)m (.)27 b(Of)16 b(course,)75 591 y(there)f(are)f(man)o(y)f(other)h(p)q(ossible) g(proto)q(cols.)75 866 y Fz(1.8)59 b(Nonblo)r(cking)20 b(communication)75 984 y Fy(One)e(can)g(impro)o(v)o(e)f(p)q(erformance)g(on)h(man)o(y)e(systems) h(b)o(y)h(o)o(v)o(erlapping)f(comm)o(unication)i(and)e(com-)75 1040 y(putation.)35 b(This)21 b(is)g(esp)q(ecially)i(true)d(on)g(systems)g (where)g(comm)o(unication)h(can)g(b)q(e)g(executed)g(au-)75 1097 y(tonomously)12 b(b)o(y)h(an)g(in)o(telligen)o(t)h(comm)o(unication)g (con)o(troller.)19 b(Ligh)o(t-w)o(eigh)o(t)13 b(threads)f(are)h(one)g(mec)o (h-)75 1153 y(anism)21 b(for)g(ac)o(hieving)h(suc)o(h)g(o)o(v)o(erlap.)37 b(An)21 b(alternativ)o(e)h(mec)o(hanism)g(that)e(often)h(leads)g(to)g(b)q (etter)75 1210 y(p)q(erformance)e(is)g(to)f(use)h Fx(non)o(blo)q(c)o(king)j (comm)o(unication)p Fy(.)32 b(A)19 b(non)o(blo)q(c)o(king)h(send)f(call)h (initiates)75 1266 y(the)15 b(send)h(op)q(eration,)f(but)h(do)q(es)f(not)g (complete)h(it.)k(The)15 b(send)h(call)h(will)f(return)g(b)q(efore)f(the)g (message)75 1323 y(w)o(as)d(copied)h(out)f(of)g(the)g(send)h(bu\013er.)19 b(A)13 b(separate)e(call)j(is)f(needed)g(to)f(complete)h(the)g(comm)o (unication,)75 1379 y(i.e.)20 b(to)13 b(v)o(erify)i(that)e(the)h(data)g(has)g (b)q(een)h(copied)g(out)f(of)g(the)g(send)h(bu\013er.)k(With)c(suitable)g (hardw)o(are,)75 1436 y(the)k(transfer)f(of)h(data)f(out)h(of)g(the)g(sender) g(memory)g(ma)o(y)f(pro)q(ceed)i(concurren)o(tly)g(with)f(computa-)75 1492 y(tions)g(done)g(at)f(the)g(sender)h(after)f(the)h(send)g(w)o(as)f (initiated)i(and)e(b)q(efore)h(it)g(completed.)31 b(Similarly)l(,)75 1549 y(a)19 b(non)o(blo)q(c)o(king)h(receiv)o(e)g(initiates)h(the)e(receiv)o (e)h(op)q(eration,)g(but)f(do)q(es)g(not)g(complete)h(it.)32 b(The)19 b(call)75 1605 y(will)f(return)f(b)q(efore)g(a)f(message)g(is)i (stored)e(in)o(to)g(the)h(receiv)o(e)h(bu\013er.)24 b(A)17 b(separate)f(call)h(is)h(needed)g(to)75 1661 y(complete)f(the)f(receiv)o(e)i (op)q(eration)e(and)g(v)o(erify)h(that)e(the)i(data)e(has)i(b)q(een)g(receiv) o(ed)g(in)o(to)g(the)f(receiv)o(e)75 1718 y(bu\013er.)k(With)14 b(suitable)h(hardw)o(are,)f(the)g(transfer)f(of)h(data)g(in)o(to)g(the)g (receiv)o(er)h(memory)f(ma)o(y)f(pro)q(ceed)75 1774 y(concurren)o(tly)f(with) g(computations)f(done)g(after)g(the)g(receiv)o(e)h(w)o(as)f(initiated)i(and)e (b)q(efore)h(it)f(completed.)75 1831 y(The)17 b(use)h(of)f(non)o(blo)q(c)o (king)h(receiv)o(es)g(ma)o(y)f(also)g(a)o(v)o(oid)g(system)f(bu\013ering)i (and)f(memory)g(to)g(memory)75 1887 y(cop)o(ying,)e(as)g(information)g(is)h (pro)o(vided)g(early)g(on)f(the)g(lo)q(cation)h(of)f(the)g(receiv)o(e)h (bu\013er.)166 1952 y(Non)o(blo)q(c)o(king)d(sends)f(can)g(use)h(the)f(same)f (three)h(mo)q(des)g(as)g(blo)q(c)o(king)h(sends:)19 b Ft(standa)o(rd)p Fy(,)13 b Ft(ready)f Fy(and)75 2009 y Ft(synchronous)p Fy(.)26 b(These)16 b(carry)g(the)h(same)f(meaning.)24 b(The)17 b(initiation)h(and)f (completion)g(of)f(a)g(standard)75 2065 y(send)k(do)f(not)g(dep)q(end)i(on)e (the)h(status)e(of)h(a)g(matc)o(hing)g(receiv)o(e.)33 b(A)20 b(ready)f(send)h(can)f(b)q(e)h(initiated)75 2122 y(only)f(if)h(a)e(matc)o (hing)h(receiv)o(e)h(has)f(already)g(b)q(een)h(initiated,)h(otherwise)e(the)g (call)h(is)g(erroneous;)g(its)75 2178 y(completion)d(do)q(es)g(not)f(dep)q (end)i(on)e(the)g(status)g(of)f(a)h(matc)o(hing)h(receiv)o(e.)23 b(A)17 b(sync)o(hronous)f(send)h(can)75 2235 y(b)q(e)j(initiated)h(b)q(efore) f(a)f(matc)o(hing)h(receiv)o(e)g(has)f(b)q(een)i(initiated,)h(but)d(will)i (complete)g(only)f(after)e(a)75 2291 y(matc)o(hing)13 b(receiv)o(e)g(has)g(b) q(een)h(initiated,)g(and)f(has)g(started)f(receiving)i(the)f(message)f (generated)h(b)o(y)g(the)75 2348 y(send)j(op)q(eration.)166 2413 y(In)f(all)h(cases)e(the)h(op)q(eration)g(that)f(initiates)i(the)f(comm) o(unication)g(\(send)g(or)f(receiv)o(e\))i(is)f(a)f(lo)q(cal,)75 2469 y(non)o(blo)q(c)o(king)22 b(call.)39 b(If)21 b(the)g(call)h(causes)g (some)e(system)h(resource)g(to)f(b)q(e)i(exhausted,)h(then)e(it)g(will)75 2526 y(fail)f(and)f(return)f(an)h(error)f(co)q(de.)31 b(Go)q(o)q(d)19 b(qualit)o(y)g(implemen)o(tations)h(of)f(MPI)g(should)g(ensure)h(that)75 2582 y(this)d(happ)q(ens)h(only)f(in)h(\\pathological")f(cases.)25 b(I.e.,)16 b(an)h(MPI)g(implemen)o(tation)h(should)g(b)q(e)f(able)h(to)75 2639 y(supp)q(ort)d(a)g(large)g(n)o(um)o(b)q(er)h(of)f(p)q(ending)i(non)o (blo)q(c)o(king)f(op)q(erations.)166 2704 y(Non)o(blo)q(c)o(king)g(sends)g (can)f(b)q(e)h(matc)o(hed)f(with)h(blo)q(c)o(king)g(receiv)o(es,)g(and)f (vice-v)o(ersa.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 18 bop 75 -100 a Fy(18)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fq(1.8.1)49 b(Comm)o(unication)14 b(Objects)75 184 y Fy(Non)o(blo)q(c)o(king)k(comm)o(unications)g(use)f (opaque)g(request)g(ob)s(jects)f(to)h(iden)o(tify)h(comm)o(unication)g(op)q (er-)75 241 y(ations)f(and)g(matc)o(h)f(the)i(op)q(eration)f(that)f (initiates)i(the)f(comm)o(unication)h(with)f(the)g(op)q(eration)h(that)75 297 y(terminates)f(it.)27 b(These)17 b(are)g(system)g(ob)s(jects)g(that)f (are)h(accessed)h(via)g(a)f(handle.)27 b(A)17 b(request)h(ob)s(ject)75 354 y(iden)o(ti\014es)j(v)m(arious)f(prop)q(erties)g(of)f(a)g(comm)o (unication)h(op)q(eration,)h(suc)o(h)e(as)g(the)h(\(send)f(or)g(receiv)o(e\)) 75 410 y(bu\013er)c(that)g(is)h(asso)q(ciated)f(with)h(it,)f(its)h(con)o (text,)e(the)i(tag)e(and)i(destination)g(argumen)o(ts)f(to)f(b)q(e)i(used)75 466 y(for)k(a)f(send,)j(or)e(the)g(tag)f(and)i(source)f(argumen)o(ts)f(to)h (b)q(e)h(used)f(for)g(a)g(receiv)o(e.)36 b(In)20 b(addition,)j(this)75 523 y(ob)s(ject)14 b(stores)g(information)g(ab)q(out)h(the)f(status)g(of)g (the)g(p)q(ending)j(comm)o(unication)e(op)q(eration)g(that)e(is)75 579 y(p)q(erformed)i(with)h(this)g(ob)s(ject.)75 846 y Fq(1.8.2)49 b(Comm)o(unication)14 b(initiation)75 981 y Fy(W)l(e)21 b(use)g(the)f(same)h (naming)g(con)o(v)o(en)o(tions)f(as)h(for)f(blo)q(c)o(king)h(comm)o (unication:)32 b(a)20 b(pre\014x)h(of)f Ft(R)h Fy(\()p Ft(S)p Fy(\))75 1037 y(is)f(used)f(for)f Ft(READ)o(Y)i Fy(\()p Ft(SYNCHRONOUS)p Fy(\))h(mo)q(de.)32 b(In)20 b(addition)g(a)e(pre\014x)i(of)e Ft(I)h Fy(\(for)f Ft(IMMEDIA)l(TE)p Fy(\))75 1093 y(indicates)f(that)d(the)h (call)i(is)e(non)o(blo)q(c)o(king.)166 1175 y(Initiate)h(a)f(standard)g(mo)q (de)g(non)o(blo)q(c)o(king)i(send.)75 1304 y Ft(MPI)p 160 1304 14 2 v 16 w(ISEND\(buf,)e(count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(,) d(request\))117 1407 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 1532 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h (bu\013er)f(\(in)o(teger\))117 1658 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 1784 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h (\(in)o(teger\))117 1910 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2035 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator) 11 b(\(handle\))117 2161 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 2311 y Fr(int)23 b(MPI)p 245 2311 15 2 v 17 w(Isend\(void*)g(buf,)g(int)g(count,)g(MPI)p 1002 2311 V 17 w(Datatype)g(datatype,)g(int)g(dest,)393 2367 y(int)h(tag,)f(MPI)p 683 2367 V 17 w(Comm)g(comm,)g(MPI)p 1034 2367 V 17 w(Comm)p 1147 2367 V 17 w(request)g(*request\))75 2479 y(MPI)p 150 2479 V 17 w(ISEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)h(REQUEST,)e (IERROR\))170 2535 y()h(BUF\(*\))170 2592 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 2704 y Fy(Initiate)16 b(a)f(ready)g(mo)q(de)h(non)o(blo)q(c)o(king)g(send.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 19 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(19)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(IRSEND\(buf,)16 b(count,)g(datat)o(yp)q(e,)g(dest,)g(tag,)f(comm)m(,)d(request\))117 143 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 253 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 362 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 471 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 580 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 689 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 798 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))75 940 y Fr(int)23 b(MPI)p 245 940 15 2 v 17 w(Irsend\(void*)f (buf,)i(int)f(count,)g(MPI)p 1026 940 V 17 w(Datatype)g(datatype,)f(int)i (dest,)393 996 y(int)g(tag,)f(MPI)p 683 996 V 17 w(Comm)g(comm,)g(MPI)p 1034 996 V 17 w(Comm)p 1147 996 V 17 w(request)g(*request\))75 1100 y(MPI)p 150 1100 V 17 w(IRSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g (TAG,)g(COMM,)g(REQUEST,)g(IERROR\))170 1156 y()g(BUF\(*\))170 1213 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f (IERROR)166 1316 y Fy(Initiate)16 b(a)f(sync)o(hronous)g(mo)q(de)h(non)o(blo) q(c)o(king)g(send.)75 1437 y Ft(MPI)p 160 1437 14 2 v 16 w(ISSEND\(buf,)f (count,)i(datat)o(yp)q(e,)f(dest,)g(tag,)f(comm)m(,)d(request\))117 1531 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 1640 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 1749 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 1858 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 1968 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2077 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2186 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))75 2327 y Fr(int)23 b(MPI)p 245 2327 15 2 v 17 w(Issend\(void*)f(buf,)i(int)f(count,)g(MPI)p 1026 2327 V 17 w(Datatype)g(datatype,)f(int)i(dest,)393 2384 y(int)g(tag,)f(MPI)p 683 2384 V 17 w(Comm)g(comm,)g(MPI)p 1034 2384 V 17 w(Comm)p 1147 2384 V 17 w(request)g(*request\))75 2487 y(MPI)p 150 2487 V 17 w(ISSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)g(REQUEST,)g (IERROR\))170 2544 y()g(BUF\(*\))170 2600 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 2704 y Fy(Initiate)16 b(a)f(non)o(blo)q(c)o(king)i(receiv)o(e.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 20 bop 75 -100 a Fy(20)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(IRECV)f(\(buf,)h(count,)g(datat)o(yp)q(e,)g(source,)g(tag,)e(comm)n (,)d(request\))117 127 y Fs(OUT)108 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 204 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h (receiv)o(e)h(bu\013er)g(\(in)o(teger\))117 281 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h (bu\013er)g(elemen)o(t)e(\(handle\))117 357 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o(teger\))117 434 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 511 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 587 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))75 713 y Fr(int)23 b(MPI)p 245 713 15 2 v 17 w(Irecv\(void*)g (buf,)g(int)g(count,)g(MPI)p 1002 713 V 17 w(Datatype)g(datatype,)g(int)g (source,)393 769 y(int)h(tag,)f(MPI)p 683 769 V 17 w(Comm)g(comm,)g(MPI)p 1034 769 V 17 w(Comm)p 1147 769 V 17 w(request)g(*request\))75 856 y(MPI)p 150 856 V 17 w(IRECV\(BUF,)f(COUNT,)h(DATATYPE,)g(SOURCE,)g(TAG,) g(COMM,)g(REQUEST,)g(IERROR\))170 913 y()g(BUF\(*\))170 969 y(INTEGER)g(COUNT,)g(DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(REQUEST,)g (IERROR)166 1056 y Fy(These)c(calls)g(allo)q(cate)h(a)e(comm)o(unication)h (request)f(and)h(asso)q(ciate)g(it)f(with)h(the)g(handle.)31 b(The)75 1113 y(request)11 b(can)f(b)q(e)i(used)f(later)g(to)f(query)g(the)h (status)f(of)g(the)h(comm)o(unication)g(or)f(w)o(ait)g(for)g(its)h (completion.)166 1170 y(A)17 b(non)o(blo)q(c)o(king)i(send)f(call)g (indicates)h(that)e(the)g(system)g(ma)o(y)g(start)f(cop)o(ying)h(data)g(out)g (of)g(the)75 1227 y(send)g(bu\013er.)24 b(The)17 b(sender)g(should)g(not)g (access)f(an)o(y)g(part)g(of)g(the)h(send)g(bu\013er)g(after)e(a)i(non)o(blo) q(c)o(king)75 1283 y(send)f(op)q(eration)f(is)h(called,)g(un)o(til)h(the)e (send)h(completes.)166 1340 y(A)d(non)o(blo)q(c)o(king)h(receiv)o(e)g(call)g (indicates)g(that)e(the)h(system)f(ma)o(y)g(start)g(writing)h(data)f(in)o(to) h(the)g(re-)75 1397 y(ceiv)o(e)f(bu\013er.)19 b(The)11 b(receiv)o(er)h (should)h(not)e(access)g(an)o(y)g(part)g(of)g(the)g(receiv)o(e)i(bu\013er)e (after)g(a)g(non)o(blo)q(c)o(king)75 1453 y(receiv)o(e)16 b(op)q(eration)g (is)f(called,)i(un)o(til)f(the)f(receiv)o(e)h(completes.)75 1579 y Fq(1.8.3)49 b(Comm)o(unication)14 b(Completion)75 1667 y Fy(The)j(functions)g Ft(MPI)p 452 1667 14 2 v 16 w(W)l(AIT)f Fy(and)g Ft(MPI)p 768 1667 V 16 w(TEST)h Fy(are)f(used)h(to)f(complete)h(a)f (non)o(blo)q(c)o(king)h(comm)o(unica-)75 1723 y(tion.)i(The)13 b(completion)h(of)e(a)g(send)h(op)q(eration)g(indicates)h(that)e(the)g (sender)h(is)g(no)o(w)f(free)h(to)f(up)q(date)h(the)75 1780 y(lo)q(cations)k(in)f(the)h(send)f(bu\013er)g(\(the)g(send)g(op)q(eration)g (itself)h(lea)o(v)o(es)f(the)g(con)o(ten)o(t)g(of)f(the)h(send)h(bu\013er)75 1836 y(unc)o(hanged\).)25 b(It)16 b(do)q(es)h(not)g(indicate)h(that)e(the)h (message)f(has)g(b)q(een)i(receiv)o(ed;)g(rather,)e(it)h(ma)o(y)f(ha)o(v)o(e) 75 1892 y(b)q(een)g(bu\013ered)g(b)o(y)f(the)h(comm)o(unication)g(subsystem.) k(Ho)o(w)o(ev)o(er,)14 b(if)i(a)f Ft(synchronous)i Fy(mo)q(de)f(send)g(w)o (as)75 1949 y(used,)f(the)g(completion)h(of)f(the)f(send)i(op)q(eration)f (indicates)h(that)e(a)h(matc)o(hing)g(receiv)o(e)h(w)o(as)e(initiated,)75 2005 y(and)h(that)g(the)g(message)g(will)i(ev)o(en)o(tually)f(b)q(e)g(receiv) o(ed)g(b)o(y)f(this)h(matc)o(hing)f(receiv)o(e.)166 2063 y(The)20 b(completion)h(of)e(a)g(receiv)o(e)i(op)q(eration)f(indicates)h(that)e(the)h (receiv)o(e)g(bu\013er)g(con)o(tains)g(the)75 2119 y(receiv)o(ed)f(message,)f (and)g(that)f(the)h(status)f(ob)s(ject)h(is)g(set;)h(the)f(receiv)o(er)h(is)f (no)o(w)f(free)h(to)g(access)g(the)75 2176 y(receiv)o(e)g(bu\013er.)27 b(It)17 b(do)q(es)h(not)f(indicate)i(that)d(the)i(matc)o(hing)f(send)h(op)q (eration)g(has)f(completed)h(\(but)75 2232 y(indicates,)e(of)f(course,)g (that)f(the)i(send)f(w)o(as)g(initiated\).)75 2336 y Ft(MPI)p 160 2336 V 16 w(W)l(AIT\(request,)h(status\))117 2414 y Fs(INOUT)62 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))117 2491 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 2616 y Fr(int)23 b(MPI)p 245 2616 15 2 v 17 w(Wait\(MPI)p 454 2616 V 16 w(Comm)p 566 2616 V 17 w(request)g(*request,)f(MPI)p 1084 2616 V 17 w(Status)h(*status\))75 2704 y(MPI)p 150 2704 V 17 w(WAIT\(REQUEST,)f(STATUS,)h(IERROR\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 21 21 bop 75 -100 a Fu(1.8.)29 b(NONBLOCKING)17 b(COMMUNICA)l(TION)868 b Fy(21)170 49 y Fr(INTEGER)23 b(REQUEST,)g(STATUS\(MPI)p 819 49 15 2 v 16 w(STATUS)p 979 49 V 16 w(SIZE\),)g(IERROR)166 136 y Fy(A)14 b(call)h(to)f Ft(MPI)p 436 136 14 2 v 15 w(W)l(AIT)h Fy(returns)e(when)i(the)f(send)h(op)q(eration)f(iden)o(ti\014ed)i(b)o(y)e Ft(request)i Fy(is)f(complete.)75 192 y(If)21 b(the)g(comm)o(unication)h(ob)s (ject)f(asso)q(ciated)g(with)g(this)h(request)f(w)o(as)f(created)h(b)o(y)g(a) g(non)o(blo)q(c)o(king)75 249 y(send)13 b(or)g(receiv)o(e)h(call,)g(then)f (the)g(ob)s(ject)f(is)h(deallo)q(cated)i(b)o(y)d(the)h(call)h(to)25 b Ft(MPI)p 1431 249 V 16 w(W)l(AIT)13 b Fy(and)g(the)g(request)75 305 y(b)q(ecomes)j(n)o(ull.)189 396 y Fp(R)n(ationale.)74 b Fy(MPI)17 b(alw)o(a)o(ys)f(n)o(ulli\014es)j(handles)f(to)f(ob)s(jects)f(that) g(ha)o(v)o(e)g(b)q(een)i(deallo)q(cated,)g(so)69 b Fm(\()189 452 y Fy(as)14 b(to)h(a)o(v)o(oid)g(dangling)h(references.)21 b(\()p Fp(End)15 b(of)i(r)n(ationale.)p Fy(\))166 543 y(The)g(call)g(returns) g(in)g Ft(status)h Fy(information)f(on)f(the)h(completed)g(op)q(eration.)24 b(The)17 b(status)f(ob)s(ject)75 599 y(for)f(a)h(receiv)o(e)g(op)q(eration)g (can)g(b)q(e)g(queried)i(using)e(the)g(functions)g(describ)q(ed)i(in)f (section)f(1.3.1.)42 b(The)69 b Fm(\()75 656 y Fy(status)12 b(ob)s(ject)f(for)h(a)g(send)h(op)q(eration)g(ma)o(y)e(b)q(e)i(up)q(dated)h (as)e(w)o(ell)h(b)o(y)f Ft(MPI)p 1362 656 V 16 w(CANCEL)p Fy(,)g(if)h(the)g (op)q(eration)75 712 y(is)i(canceled)i(\(Section)e(1.9,)f(b)q(elo)o(w\).)20 b(Th)o(us,)14 b(one)h(alw)o(a)o(ys)f(need)i(to)e(pro)o(vide)i(a)e(v)m(alid)j (status)d(argumen)o(t)75 769 y(to)h Ft(MPI)p 216 769 V 15 w(W)l(AIT)h Fy(and)f(the)g(other)g(related)h(functions.)166 825 y(It)21 b(is)11 b(allo)o(w)o(ed)g(to)f(call)i Ft(MPI)p 634 825 V 16 w(W)l(AIT)e Fy(with)h(a)g(n)o(ull)h Ft(request)g Fy(argumen)o(t,)e(in)i(whic) o(h)f(case)g(the)g(op)q(eration)69 b Fm(\()75 882 y Fy(has)15 b(no)g(e\013ect.)189 973 y Fp(R)n(ationale.)52 b Fy(This)18 b(mak)o(es)e Ft(MPI)p 772 973 V 16 w(W)l(AIT)h Fy(functionally)i(equiv)m (alen)o(t)g(to)e Ft(MPI)p 1528 973 V 16 w(W)l(AIT)l(ALL)g Fy(with)g(a)189 1029 y(list)f(of)e(length)i(one)g(and)f(adds)g(some)g(elegance.)21 b(\()p Fp(End)16 b(of)g(r)n(ationale.)p Fy(\))75 1167 y Ft(MPI)p 160 1167 V 16 w(TEST\(request,)g(\015ag,)f(status\))117 1244 y Fs(INOUT)62 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))117 1317 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))117 1390 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 1514 y Fr(int)23 b(MPI)p 245 1514 15 2 v 17 w(Test\(MPI)p 454 1514 V 16 w(Comm)p 566 1514 V 17 w(request)g(*request,)f(int)i(flag,)f(MPI)p 1323 1514 V 17 w(Status)g(*status\))75 1601 y(MPI)p 150 1601 V 17 w(TEST\(REQUEST,)f(FLAG,)h(STATUS,)g(IERROR\))170 1657 y(LOGICAL)g(FLAG)170 1714 y(INTEGER)g(REQUEST,)g(STATUS\(MPI)p 819 1714 V 16 w(STATUS)p 979 1714 V 16 w(SIZE\),)g(IERROR)166 1800 y Fy(A)17 b(call)i(to)e Ft(MPI)p 446 1800 14 2 v 15 w(TEST)h Fy(returns)f Ft(\015ag)h(=)f(true)h Fy(if)g(the)g(op)q(eration)f(iden)o (ti\014ed)j(b)o(y)d Ft(request)i Fy(is)f(com-)75 1857 y(plete.)37 b(In)21 b(suc)o(h)g(a)f(case,)i(the)e(status)g(ob)s(ject)g(is)h(set)f(to)g (con)o(tain)h(information)g(on)f(the)h(completed)75 1913 y(op)q(eration;)15 b(if)h(the)f(comm)o(unication)h(ob)s(ject)e(w)o(as)h(created)g(b)o(y)g(a)g (non)o(blo)q(c)o(king)h(send)g(or)f(receiv)o(e,)h(then)75 1970 y(it)i(is)g(deallo)q(cated)i(and)e(the)g(request)f(b)q(ecomes)i(n)o(ull.)29 b(The)18 b(call)h(returns)f Ft(\015ag=false)p Fy(,)g(otherwise.)28 b(In)75 2026 y(suc)o(h)16 b(a)e(case,)h(the)h(v)m(alue)g(of)f(the)g(status)f (ob)s(ject)h(is)h(unde\014ned.)166 2083 y Ft(MPI)p 251 2083 V 16 w(TEST)f Fy(is)h(a)f(lo)q(cal,)h(non)o(blo)q(c)o(king)g(op)q(eration.) 166 2139 y(The)11 b(return)g(status)f(ob)s(ject)h(for)f(a)h(receiv)o(e)h(op)q (eration)f(carries)g(information)g(that)g(can)g(b)q(e)g(accessed)75 2195 y(using)k(the)g(functions)g(describ)q(ed)i(in)e(section)g(1.3.1.)38 b(The)15 b(status)f(ob)s(ject)g(for)g(a)g(send)h(op)q(eration)g(ma)o(y)68 b Fm(\()75 2252 y Fy(b)q(e)16 b(up)q(dated)g(as)f(w)o(ell,)h(if)f(the)g(op)q (eration)h(w)o(as)e(canceled)j(\(see)e(Section)h(1.9,)e(b)q(elo)o(w\).)166 2308 y(It)21 b(is)11 b(allo)o(w)o(ed)g(to)f(call)i Ft(MPI)p 634 2308 V 15 w(TEST)f Fy(with)g(a)f(n)o(ull)i Ft(request)g Fy(argumen)o(t,)f(in)g(whic)o(h)h(case)e(the)h(op)q(eration)69 b Fm(\()75 2365 y Fy(returns)15 b Fo(\015ag=false)p Fy(.)166 2421 y(In)g(a)f(m)o(ulti-threaded)i(en)o(vironmen)o(t,)e(the)g(use)h(of)f(a)g (blo)q(c)o(king)i(receiv)o(e)f(op)q(eration)g(\()f Ft(MPI)p 1732 2421 V 15 w(W)l(AIT)p Fy(\))75 2478 y(ma)o(y)k(allo)o(w)i(the)f(op)q (erating)g(system)g(to)g(de-sc)o(hedule)i(the)e(blo)q(c)o(k)o(ed)h(thread)f (and)h(sc)o(hedule)g(another)75 2534 y(thread)j(for)g(execution,)j(if)e(suc)o (h)g(is)f(a)o(v)m(ailable.)47 b(The)23 b(use)h(of)f(a)g(non)o(blo)q(c)o(king) h(receiv)o(e)h(op)q(eration)75 2591 y(\()p Ft(MPI)p 178 2591 V 15 w(TEST)p Fy(\))e(allo)o(ws)f(the)g(user)g(to)g(sc)o(hedule)i(alternativ) o(e)e(activities)i(within)f(a)f(single)h(thread)f(of)75 2647 y(execution.)166 2704 y(Example:)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 22 22 bop 75 -100 a Fy(22)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f (rank,)h(ierr\))75 106 y(IF\(rank.EQ.0\))123 162 y(THEN)170 219 y(CALL)h(MPI_ISEND\(a\(1\),)e(10,)h(MPI_REAL,)g(1,)g(tag,)h(comm,)f (request,)g(ierr\))170 275 y(****)h(do)f(some)h(computation)e(to)i(mask)f (latency)g(****)170 332 y(CALL)h(MPI_WAIT\(request,)d(status,)i(ierr\))123 388 y(ELSE)170 444 y(CALL)h(MPI_IRECV\(a\(1\),)e(15,)h(MPI_REAL,)g(0,)g(tag,) h(comm,)f(request,)g(ierr\))170 501 y(****)h(do)f(some)h(computation)e(to)i (mask)f(latency)g(****)170 557 y(CALL)h(MPI_WAIT\(request,)d(status,)i (ierr\))75 614 y(END)g(IF)166 708 y Fy(The)f(functions)h Ft(MPI)p 554 708 14 2 v 15 w(W)l(AIT)f Fy(and)g Ft(MPI)p 881 708 V 16 w(TEST)g Fy(can)g(b)q(e)g(used)h(to)e(complete)h(b)q(oth)g(sends)h(and)75 764 y(receiv)o(es;)14 b(they)f(are)f(also)h(used)g(to)f(complete)h(an)o(y)g (other)f(non)o(blo)q(c)o(king)i(comm)o(unication)f(call)h(pro)o(vided)75 821 y(b)o(y)h(MPI.)75 943 y Fq(1.8.4)49 b(Semantics)13 b(of)k(Nonblo)q(cking) h(Comm)o(unications)75 1029 y Fy(The)11 b(seman)o(tics)f(of)h(non)o(blo)q(c)o (king)g(comm)o(unication)h(is)f(de\014ned)h(b)o(y)e(suitably)i(extending)g (the)e(de\014nitions)75 1085 y(in)16 b(section)g(1.4)e(for)h(order)g(and)g (progress.)166 1142 y(Non)o(blo)q(c)o(king)k(comm)o(unication)f(op)q (erations)g(are)g(ordered)g(according)g(to)f(the)h(execution)h(order)75 1198 y(of)h(the)g(calls)h(that)e(initiate)j(the)e(comm)o(unication;)i(this)f (de\014nes)g(the)f(meaning)h(of)e(non-o)o(v)o(ertaking)75 1255 y(comm)o(unication.)166 1311 y(Example:)75 1418 y Fr(CALL)k (MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1474 y(IF)h(\(RANK.EQ.0\))147 1531 y(THEN)218 1587 y(CALL)f(MPI_ISEND\(a,)g(1,)g(MPI_REAL,)g(1,)h(0,)f (comm,)g(h1,)h(ierr\))218 1644 y(CALL)f(MPI_ISEND\(b,)g(1,)g(MPI_REAL,)g(1,)h (0,)f(comm,)g(h2,)h(ierr\))147 1700 y(ELSE)94 b(!)24 b(rank.EQ.1)218 1756 y(CALL)f(MPI_IRECV\(a,)g(1,)g(MPI_REAL,)g(0,)h(MPI_ANY_TAG,)e(comm,)h (h1,)g(ierr\))218 1813 y(CALL)g(MPI_IRECV\(b,)g(1,)g(MPI_REAL,)g(0,)h(0,)f (comm,)g(h2,)h(ierr\))75 1869 y(END)f(IF)75 1926 y(CALL)g (MPI_WAIT\(h1,status\))75 1982 y(CALL)g(MPI_WAIT\(h2,status\))166 2089 y Fy(The)17 b(\014rst)f(send)h(of)f(pro)q(cess)h(zero)g(will)h(matc)o(h) e(the)h(\014rst)f(receiv)o(e)i(of)e(pro)q(cess)h(one,)g(ev)o(en)g(if)g(b)q (oth)75 2145 y(messages)e(are)h(sen)o(t)f(b)q(efore)h(pro)q(cess)g(one)g (executes)h(either)f(receiv)o(e.)22 b(This)17 b(precludes)g(an)f(implemen-)75 2202 y(tation)f(where)g(di\013eren)o(t)h(queues)g(are)f(used)g(to)g(p)q(ost)g (the)g(t)o(w)o(o)f(receiv)o(es.)166 2258 y(A)22 b(call)g(to)f Ft(MPI)p 458 2258 V 16 w(W)l(AIT)h Fy(that)f(completes)h(a)f(receiv)o(e)i (will)g(ev)o(en)o(tually)g(terminate)e(and)h(return)75 2315 y(if)f(a)f(matc)o(hing)g(send)h(has)f(b)q(een)h(started,)g(unless)g(the)g (send)g(is)f(satis\014ed)h(b)o(y)f(another)g(receiv)o(e.)36 b(In)75 2371 y(particular,)20 b(if)f(the)g(matc)o(hing)g(send)g(is)g(non)o (blo)q(c)o(king,)i(then)e(the)g(receiv)o(e)g(should)h(complete)f(ev)o(en)g (if)75 2428 y(no)c(call)i(is)e(executed)i(b)o(y)e(the)g(sender)h(to)f (complete)h(the)f(send.)20 b(Similarly)l(,)e(a)d(call)h(to)f Ft(MPI)p 1651 2428 V 15 w(W)l(AIT)h Fy(that)75 2484 y(completes)i(a)e(send)i (will)h(ev)o(en)o(tually)f(return)f(if)g(a)g(matc)o(hing)g(receiv)o(e)h(has)f (b)q(een)h(started,)f(unless)h(the)75 2540 y(receiv)o(e)e(is)g(satis\014ed)f (b)o(y)g(another)g(send,)g(and)g(ev)o(en)h(if)f(no)g(call)h(is)g(executed)g (to)e(complete)i(the)f(receiv)o(e.)166 2597 y(Example:)75 2704 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 23 23 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(23)75 49 y Fr(IF)24 b(\(RANK.EQ.0\))147 106 y(THEN)218 162 y(CALL)f(MPI_ISEND\(a,)g(1,)g(MPI_REAL,)g(1,)h(0,)f(comm,)g(h,)h(ierr\)) 218 219 y(CALL)f(MPI_RECV\(b,)g(1,)g(MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))218 275 y(CALL)g(MPI_WAIT\(h,)g(status,)g(ierr\))147 332 y(ELSE)94 b(!)24 b(rank.EQ.1)218 388 y(CALL)f(MPI_RECV\(a,)g(1,)g(MPI_REAL,)g(0,)h(0,)f (comm,)g(ierr\))218 444 y(CALL)g(MPI_SEND\(b,)g(1,)g(MPI_REAL,)g(0,)h(0,)f (comm,)g(ierr\))75 501 y(END)g(IF)166 633 y Fy(This)16 b(co)q(de)f(should)i (not)d(deadlo)q(c)o(k)j(in)f(a)f(correct)f(MPI)h(implemen)o(tation.)166 694 y(If)j Ft(MPI)p 299 694 14 2 v 16 w(TEST)g Fy(that)e(completes)j(a)e (receiv)o(e)i(is)f(rep)q(eatedly)h(called)g(with)f(the)g(same)f(argumen)o (ts,)75 751 y(and)d(a)g(matc)o(hing)g(send)g(has)g(b)q(een)h(started,)e(then) h(the)g(call)h(will)g(ev)o(en)o(tually)g(return)f Ft(\015ag=true)p Fy(,)h(unless)75 807 y(the)i(send)g(is)g(satis\014ed)h(b)o(y)f(another)f (receiv)o(e.)25 b(If)17 b Ft(MPI)p 1020 807 V 16 w(TEST)g Fy(that)f (completes)i(a)e(send)i(is)f(rep)q(eatedly)75 864 y(called)e(with)f(the)g (same)f(argumen)o(ts,)g(and)h(a)f(matc)o(hing)g(receiv)o(e)i(has)e(b)q(een)i (started,)e(then)h(the)g(call)g(will)75 920 y(ev)o(en)o(tually)i(return)f Ft(\015ag=true)p Fy(,)h(unless)g(the)g(receiv)o(e)g(is)f(satis\014ed)h(b)o(y) f(another)g(send.)166 982 y(Non)o(blo)q(c)o(king)g(comm)o(unications)g (cannot)f(b)q(e)g(replaced)i(b)o(y)e(blo)q(c)o(king)h(comm)o(unications,)g (ev)o(en)f(in)75 1038 y(the)h Ft(synchronous)i Fy(comm)o(unication)f(mo)q (de.)k(Consider)c(the)f(follo)o(wing)h(example:)75 1157 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1214 y(IF\(rank.EQ.0\))123 1270 y(THEN)170 1327 y(CALL)h(MPI_ISEND\(a\(1\),)e(count,)h(MPI_REAL,)f(1,)i (tag1,)f(comm,)g(request1,)g(ierr\))170 1383 y(CALL)h(MPI_ISEND\(a\(2\),)e (count,)h(MPI_REAL,)f(1,)i(tag2,)f(comm,)g(request2,)g(ierr\))170 1440 y(CALL)h(MPI_WAIT\(request1,)d(status,)i(ierr\))170 1496 y(CALL)h(MPI_WAIT\(request2,)d(status,)i(ierr\))123 1552 y(ELSE)170 1609 y(CALL)h(MPI_IRECV\(a\(2\),)e(count,)h(MPI_REAL,)f(0,)i(tag2,)f(comm,)g (request2,)g(ierr\))170 1665 y(CALL)h(MPI_IRECV\(a\(1\),)e(count,)h (MPI_REAL,)f(0,)i(tag1,)f(comm,)g(request1,)g(ierr\))170 1722 y(CALL)h(MPI_WAIT\(request2,)d(status,)i(ierr\))170 1778 y(CALL)h (MPI_WAIT\(request1,)d(status,)i(ierr\))75 1835 y(END)g(IF)166 1949 y Fy(The)16 b(co)q(de)g(is)g(guaran)o(teed)f(to)g(execute)h(correctly)l (,)g(ev)o(en)g(in)g(an)g(implemen)o(tation)h(that)d(do)q(es)i(not)75 2005 y(bu\013er)d(messages)f(\(the)h(only)h(resource)f(requiremen)o(t)g(is)h (the)f(abilit)o(y)h(to)e(ha)o(v)o(e)h(t)o(w)o(o)f(p)q(ending)i(comm)o(uni-)75 2062 y(cations\).)20 b(If)15 b(one)h(replaces)g(the)f(non)o(blo)q(c)o(king)i (sends)f(\(receiv)o(es\))f(b)o(y)g(blo)q(c)o(king)i(sends)f(\(receiv)o(es\),) f(and)75 2118 y(deletes)j(the)f(calls)h(to)e Ft(MPI)p 551 2118 V 15 w(W)l(AIT)p Fy(,)h(then)g(the)g(co)q(de)g(ma)o(y)f(deadlo)q(c)o(k.)26 b(The)17 b(co)q(de)g(ma)o(y)f(also)h(deadlo)q(c)o(k)75 2175 y(if)f(one)f(deletes)h(the)f(calls)h(that)f(initiate)h(comm)o(unication)g (and)f(replaces)h(eac)o(h)f(call)i(to)d Ft(MPI)p 1686 2175 V 16 w(W)l(AIT)h Fy(b)o(y)75 2231 y(the)g(appropriate)g(blo)q(c)o(king)i (send)f(or)f(receiv)o(e.)75 2382 y Fq(1.8.5)49 b(Multiple)17 b(Completions)75 2478 y Fy(It)23 b(is)h(con)o(v)o(enien)o(t)g(to)f(b)q(e)h (able)g(to)e(w)o(ait)h(for)g(the)g(completion)i(of)d(an)o(y)h(or)g(all)h(the) g(op)q(erations)f(in)75 2534 y(a)g(set,)h(rather)e(than)h(ha)o(ving)g(to)f(w) o(ait)h(for)f(a)g(sp)q(eci\014c)j(message.)43 b(A)23 b(call)h(to)e Ft(MPI)p 1597 2534 V 16 w(W)l(AIT)l(ANY)h Fy(or)75 2591 y Ft(MPI)p 160 2591 V 16 w(TEST)l(ANY)e Fy(can)f(b)q(e)g(used)g(to)f(w)o(ait)h(for)f (the)h(completion)g(of)g(one)g(out)f(of)g(sev)o(eral)h(op)q(erations;)75 2647 y(a)14 b(call)h(to)27 b Ft(MPI)p 347 2647 V 15 w(W)l(AIT)l(ALL)14 b Fy(or)f Ft(MPI)p 701 2647 V 16 w(TEST)l(ALL)h Fy(can)g(b)q(e)g(used)h(to)e (w)o(ait)g(for)h(all)g(p)q(ending)i(op)q(erations)e(in)75 2704 y(a)h(list.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 24 24 bop 75 -100 a Fy(24)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(W)l(AIT)l(ANY)g(\(count,)g(a)o(rra)o(y)p 640 49 V 14 w(of)p 691 49 V 16 w(requests,)h(index,)f(status\))117 126 y Fs(IN)155 b Ft(count)482 b Fs(list)14 b(length)f(\(in)o(teger\))117 198 y(INOUT)62 b Ft(a)o(rra)o(y)p 416 198 V 15 w(of)p 468 198 V 16 w(requests)272 b Fs(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f (handles\))117 270 y(OUT)108 b Ft(index)489 b Fs(index)12 b(of)f(handle)g (for)g(op)q(eration)h(that)f(completed)g(\(in)o(teger\))117 342 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 466 y Fr(int)23 b(MPI)p 245 466 15 2 v 17 w(Waitany\(int)g(count,)g(MPI)p 788 466 V 16 w(Comm)p 900 466 V 17 w(request)g(*array)p 1252 466 V 16 w(of)p 1316 466 V 17 w(requests,)g(int)g(*index,)393 523 y(MPI)p 468 523 V 17 w(Status)g(*status\))75 609 y(MPI)p 150 609 V 17 w(WAITANY\(COUNT,)f(ARRAY)p 645 609 V 16 w(OF)p 709 609 V 17 w(REQUESTS,)h(INDEX,)g(STATUS,)g(IERROR\))170 666 y(INTEGER)g(COUNT,)g(ARRAY)p 651 666 V 17 w(OF)p 716 666 V 17 w(REQUESTS\(*\),)f(INDEX,)h(STATUS\(MPI)p 1450 666 V 16 w(STATUS)p 1610 666 V 16 w(SIZE\),)170 722 y(IERROR)166 809 y Fy(Blo)q(c)o(ks)17 b(un)o(til)h(one)e(of)g(the)h(op)q(erations)f(asso)q (ciated)h(with)g(the)f(comm)o(unication)h(requests)g(in)g(the)75 865 y(arra)o(y)c(has)h(completed.)20 b(Returns)15 b(in)g Ft(index)g Fy(the)f(index)h(of)f(that)f(request)h(in)h(the)f(arra)o(y)f(and)h(returns)g (in)75 922 y Ft(status)j Fy(the)e(status)g(of)g(the)g(completing)h(comm)o (unication.)166 978 y(The)j(execution)h(of)e Ft(MPI)p 610 978 14 2 v 16 w(W)l(AIT)l(ANY\(count,)j(a)o(rra)o(y)p 1079 978 V 14 w(of)p 1130 978 V 16 w(requests,)h(index,)e(status\))g Fy(has)f(the)f(same)75 1035 y(e\013ect)10 b(as)g(the)g(execution)i(of)20 b Ft(MPI)p 655 1035 V 16 w(W)l(AIT\(a)o(rra)o(y)p 897 1035 V 14 w(of)p 948 1035 V 16 w(requests[i],)13 b(status\))p Fy(,)g(where)e Ft(i)f Fy(is)h(the)f(v)m(alue)i(returned)75 1091 y(b)o(y)h Ft(index)p Fy(.)21 b(In)14 b(particular,)g(the)g(asso)q(ciated)f(comm)o (unication)i(ob)s(ject)d(is)i(deallo)q(cated,)h(and)f(the)g(handle)75 1148 y(to)21 b(it)h(in)g Ft(a)o(rra)o(y)p 346 1148 V 15 w(of)p 398 1148 V 16 w(requests)i Fy(is)e(set)f(to)g(n)o(ull.)41 b(If)22 b(more)f(then)h(one)g(op)q(eration)g(is)g(enabled)h(and)f(can)75 1204 y(terminate,)d(one)g(is)g(arbitrarily)g(c)o(hosen.)30 b(There)19 b(is)g(no)g(requiremen)o(t)g(that)f(the)g(c)o(hoice)i(satisfy)e (an)o(y)75 1260 y(fairness)e(criterion.)166 1317 y Ft(MPI)p 251 1317 V 16 w(W)l(AIT)30 b Fy(is)16 b(equiv)m(alen)o(t)h(to)d Ft(MPI)p 809 1317 V 16 w(W)l(AIT)l(ANY)i Fy(with)g(an)f(arra)o(y)f(of)h (length)g(one.)-1624 b Fm(\))75 1421 y Ft(MPI)p 160 1421 V 16 w(TEST)l(ANY\(count,)17 b(a)o(rra)o(y)p 626 1421 V 14 w(of)p 677 1421 V 16 w(requests,)g(index,)f(\015ag,)f(status\))117 1498 y Fs(IN)155 b Ft(count)482 b Fs(list)14 b(length)f(\(in)o(teger\))117 1570 y(INOUT)62 b Ft(a)o(rra)o(y)p 416 1570 V 15 w(of)p 468 1570 V 16 w(requests)272 b Fs(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f (handles\))117 1641 y(OUT)108 b Ft(index)489 b Fs(index)9 b(of)g(op)q (eration)g(that)h(completed,)f(or)g Fo(MPI)p 1651 1641 13 2 v 14 w(UNDEFINED)905 1698 y Fs(if)k(none)h(completed)g(\(in)o(teger\))117 1770 y(OUT)108 b Ft(\015ag)518 b Fo(TRUE)13 b Fs(if)g(one)i(of)e(the)h(op)q (erations)h(is)e(complete)h(\(logical\))117 1842 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 1966 y Fr(int)23 b(MPI)p 245 1966 15 2 v 17 w(Testany\(int)g(count,)g(MPI)p 788 1966 V 16 w(Comm)p 900 1966 V 17 w(request)g(*array)p 1252 1966 V 16 w(of)p 1316 1966 V 17 w(requests,)g(int)g(*index,)393 2023 y(int)h(*flag,)f(MPI)p 731 2023 V 16 w(Status)g(*status\))75 2109 y(MPI)p 150 2109 V 17 w(TESTANY\(COUNT,)f(ARRAY)p 645 2109 V 16 w(OF)p 709 2109 V 17 w(REQUESTS,)h(INDEX,)g(FLAG,)g(STATUS,)g (IERROR\))170 2165 y(LOGICAL)g(FLAG)170 2222 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2222 V 17 w(OF)p 716 2222 V 17 w(REQUESTS\(*\),)f(INDEX,)h(STATUS\(MPI)p 1450 2222 V 16 w(STATUS)p 1610 2222 V 16 w(SIZE\),)170 2278 y(IERROR)166 2365 y Fy(Causes)11 b(either)h(one)f(or)g(none)h(of)f(the)g(op)q (erations)h(asso)q(ciated)f(with)h(the)f(comm)o(unication)h(requests)75 2421 y(to)g(return.)19 b(In)13 b(the)f(former)g(case,)h(it)f(returns)h(in)g Ft(index)g Fy(the)g(index)h(of)e(its)g(request)h(in)g(the)g(arra)o(y)l(,)e (returns)75 2478 y(in)j Ft(status)h Fy(the)e(status)f(of)g(that)g(op)q (eration,)i(deallo)q(cates)g(the)f(asso)q(ciated)g(comm)o(unication)g(ob)s (ject,)g(and)75 2534 y(sets)k(the)h(request)g(to)f(n)o(ull.)29 b(In)18 b(the)g(latter)f(case,)h(it)g(returns)g(a)f(v)m(alue)i(of)e Fo(MPI)p 1458 2534 13 2 v 15 w(UNDEFINED)f Fy(in)j Ft(index)75 2591 y Fy(and)c Ft(status)j Fy(is)d(unde\014ned.)166 2647 y(The)k(execution)g (of)f Ft(MPI)p 609 2647 14 2 v 16 w(TEST)l(ANY\(count,)j(a)o(rra)o(y)p 1079 2647 V 15 w(of)p 1131 2647 V 16 w(requests,)g(index,)f(status\))g Fy(has)f(the)f(same)75 2704 y(e\013ect)c(as)f(the)h(execution)h(of)e Ft(MPI)p 662 2704 V 16 w(TEST\(a)o(rra)o(y)p 905 2704 V 15 w(of)p 957 2704 V 16 w(requests[i],)j(\015ag,)d(status\))p Fy(,)j(for)d Ft(i=0,)g(1)h(,...,)e(count-1)p Fy(,)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 25 25 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(25)75 49 y(in)14 b(some)e(arbitrary)h(order,)g(un)o(til)h(one)f(call)h (returns)f Ft(\015ag=true)p Fy(,)h(or)e(all)i(fail.)20 b(In)14 b(the)f(former)f(case,)h Ft(index)75 106 y Fy(is)19 b(set)f(to)g(the)g(last)h (v)m(alue)g(of)f Ft(i)p Fy(;)i(in)f(the)f(later)h(case,)g(it)f(is)h(set)f(to) g Fo(MPI)p 1314 106 13 2 v 14 w(UNDEFINED)p Fy(.)36 b(In)19 b(particular,)69 b Fm(\()75 162 y Ft(MPI)p 160 162 14 2 v 16 w(TEST)15 b Fy(is)h(equiv)m(alen)o(t)h(to)d Ft(MPI)p 704 162 V 16 w(TEST)l(ANY)j Fy(with)e(an)g(arra)o(y)f(of)h(length)h(one.)166 295 y Fl(Discussion:)48 b Fo(MPI)p 525 295 13 2 v 15 w(TEST)p 644 295 V 15 w(ANY)11 b Fs(pro)o(vides)h(t)o(w)o(o)e(indicators)i(of)e (success/failure:)19 b(the)12 b(v)n(alue)e(of)h Fo(\015ag)g Fs(and)68 b Fk(\()75 344 y Fs(the)12 b(v)n(alue)g(of)f Fo(index)i Fs(\(i.e.,)e(whether)i Fo(index)g Fs(=)f Fo(MPI)p 870 344 V 14 w(UNDEFINED)p Fs(\).)f(One)h(could)g(do)f(with)h(one)g(only;)f(e.g.,)g (get)h(rid)75 394 y(of)i Fo(\015ag)p Fs(,)g(or)h(allo)o(w)e Fo(index)j Fs(to)e(b)q(e)i(unde\014ned)f(if)f(no)h(test)g(succeeded.)23 b(This)15 b(ma)o(y)e(buy)h(some)g(e\016ciency)h(and)g(ma)o(y)75 444 y(sa)o(v)o(e)c(a)f(parameter,)g(at)g(the)h(exp)q(ense)i(of)c(increasing)i (the)g(di\013erence)h(b)q(et)o(w)o(een)g Fo(MPI)p 1384 444 V 15 w(TEST)e Fs(and)h Fo(MPI)p 1666 444 V 14 w(TEST)m(ANY)p Fs(.)166 501 y(Opinions?)75 687 y Ft(MPI)p 160 687 14 2 v 16 w(W)l(AIT)l(ALL\()j(count,)i(a)o(rra)o(y)p 626 687 V 15 w(of)p 678 687 V 16 w(requests,)h(a)o(rra)o(y)p 971 687 V 15 w(of)p 1023 687 V 15 w(statuses\))117 764 y Fs(IN)155 b Ft(count)482 b Fs(lists)14 b(length)g(\(in)o(teger\))117 839 y(INOUT)62 b Ft(a)o(rra)o(y)p 416 839 V 15 w(of)p 468 839 V 16 w(requests)272 b Fs(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f(handles\))117 914 y(OUT)108 b Ft(a)o(rra)o(y)p 416 914 V 15 w(of)p 468 914 V 16 w(statuses)276 b Fs(arra)o(y)14 b(of)f(status)i(ob)r(jects)g(\(arra)o(y) f(of)f(Status\))75 1038 y Fr(int)23 b(MPI)p 245 1038 15 2 v 17 w(Waitall\(int)g(count,)g(MPI)p 788 1038 V 16 w(Comm)p 900 1038 V 17 w(request)g(*array)p 1252 1038 V 16 w(of)p 1316 1038 V 17 w(requests,)393 1094 y(MPI)p 468 1094 V 17 w(Status)g(*array)p 796 1094 V 16 w(of)p 860 1094 V 17 w(statuses\))75 1181 y(MPI)p 150 1181 V 17 w(WAITALL\(COUNT,)f(ARRAY)p 645 1181 V 16 w(OF)p 709 1181 V 17 w(REQUESTS,)h(ARRAY)p 1085 1181 V 16 w(OF)p 1149 1181 V 17 w(STATUSES,)g(IERROR\))170 1237 y(INTEGER)g(COUNT,)g(ARRAY)p 651 1237 V 17 w(OF)p 716 1237 V 17 w(REQUESTS\(*\),)170 1294 y(ARRAY)p 293 1294 V 17 w(OF)p 358 1294 V 17 w(STATUSES\(MPI)p 663 1294 V 15 w(STATUS)p 822 1294 V 17 w(SIZE,*\),)g(IERROR)166 1380 y Fy(Blo)q(c)o(ks)12 b(un)o(til)h(all)g(comm)o(unication)g(op)q (erations)f(asso)q(ciated)g(with)g(requests)f(in)i(the)f(list)h(complete,)75 1437 y(and)k(return)g(the)g(status)f(of)h(all)h(these)f(op)q(erations.)26 b(Both)16 b(arra)o(ys)g(ha)o(v)o(e)h(the)g(same)f(n)o(um)o(b)q(er)i(of)e(v)m (alid)75 1493 y(en)o(tries.)24 b(The)16 b Ft(i)p Fy(-th)h(en)o(try)f(in)h Ft(a)o(rra)o(y)p 694 1493 14 2 v 14 w(of)p 745 1493 V 16 w(statuses)i Fy(is)e(set)f(to)g(the)h(return)f(status)f(of)h(the)h Ft(i)p Fy(-th)f(op)q(eration.)75 1550 y(All)g(comm)o(unication)g(ob)s(jects)f(are)g (deallo)q(cated)h(and)g(all)g(requests)f(are)g(set)g(to)g(n)o(ull.)166 1606 y(The)f(execution)h(of)f Ft(MPI)p 596 1606 V 16 w(W)l(AIT)l(ALL\(count,) g(a)o(rra)o(y)p 1046 1606 V 15 w(of)p 1098 1606 V 16 w(requests,)i(a)o(rra)o (y)p 1390 1606 V 15 w(of)p 1442 1606 V 16 w(statuses)g Fy(has)e(the)g(same)75 1663 y(e\013ect)24 b(as)g(the)g(execution)h(of)f Ft(MPI)p 714 1663 V 16 w(W)l(AIT\(a)o(rra)o(y)p 956 1663 V 14 w(of)p 1007 1663 V 16 w(request[i],)k(a)o(rra)o(y)p 1331 1663 V 15 w(of)p 1383 1663 V 16 w(statuses[i]\))p Fy(,)g(for)c Ft(i=0)g(,...,)75 1719 y(count-1)p Fy(,)16 b(in)g(some)f(arbitrary)f(order.)75 1823 y Ft(MPI)p 160 1823 V 16 w(TEST)l(ALL\(count,)i(a)o(rra)o(y)p 613 1823 V 14 w(of)p 664 1823 V 16 w(requests,)h(\015ag,)e(a)o(rra)o(y)p 1054 1823 V 14 w(of)p 1105 1823 V 16 w(statuses\))117 1900 y Fs(IN)155 b Ft(count)482 b Fs(lists)14 b(length)g(\(in)o(teger\))117 1975 y(INOUT)62 b Ft(a)o(rra)o(y)p 416 1975 V 15 w(of)p 468 1975 V 16 w(requests)272 b Fs(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f (handle\))117 2049 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))117 2124 y(OUT)108 b Ft(a)o(rra)o(y)p 416 2124 V 15 w(of)p 468 2124 V 16 w(statuses)276 b Fs(arra)o(y)14 b(of)f(status)i(ob)r(jects)g (\(Status\))75 2248 y Fr(int)23 b(MPI)p 245 2248 15 2 v 17 w(Testall\(int)g(count,)g(MPI)p 788 2248 V 16 w(Comm)p 900 2248 V 17 w(request)g(*array)p 1252 2248 V 16 w(of)p 1316 2248 V 17 w(requests,)g(int)g(*flag,)393 2305 y(MPI)p 468 2305 V 17 w(Status)g(*array)p 796 2305 V 16 w(of)p 860 2305 V 17 w(statuses\))75 2391 y(MPI)p 150 2391 V 17 w(TESTALL\(COUNT,)f(ARRAY)p 645 2391 V 16 w(OF)p 709 2391 V 17 w(REQUESTS,)h(FLAG,)g(ARRAY)p 1228 2391 V 16 w(OF)p 1292 2391 V 17 w(STATUSES,)g(IERROR\))170 2448 y(LOGICAL)g(FLAG)170 2504 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2504 V 17 w(OF)p 716 2504 V 17 w(REQUESTS\(*\),)170 2561 y(ARRAY)p 293 2561 V 17 w(OF)p 358 2561 V 17 w(STATUSES\(MPI)p 663 2561 V 15 w(STATUS)p 822 2561 V 17 w(SIZE,*\),)g(IERROR)166 2647 y Fy(Returns)18 b Ft(\015ag)g(=)g(true)h Fy(if)f(all)h(comm)o (unications)g(asso)q(ciated)f(with)g(requests)g(in)h(the)f(arra)o(y)e(ha)o(v) o(e)75 2704 y(completed.)30 b(In)19 b(this)f(case,)h(eac)o(h)f(status)g(en)o (try)g(is)g(set)g(to)g(the)g(status)g(of)g(the)g(corresp)q(onding)h(com-)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 26 26 bop 75 -100 a Fy(26)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fy(m)o(unication.)32 b(All)21 b(comm)o(unication)e(ob)s(jects)g(are)g(deallo)q(cated,)i(and)e(all) h(requests)f(are)g(set)f(to)h(n)o(ull.)75 106 y(Otherwise,)k Ft(\015ag)e(=)g(false)h Fy(is)f(returned,)i(no)e(request)g(is)h(mo)q (di\014ed)g(and)f(the)h(v)m(alues)g(of)f(the)g(status)75 162 y(en)o(tries)16 b(are)e(unde\014ned.)22 b(This)16 b(is)g(a)f(non)o(blo)q(c)o (king)h(op)q(eration)g(that)e(returns)h(immediately)l(.)166 301 y Fl(Discussion:)51 b Fs(W)m(e)13 b(do)g(not)g(ha)o(v)o(e)g(a)g(function) g(of)g(the)h(form:)i(\\complete)c(as)h(man)o(y)f(comm)o(unicati)o(ons)f(as) -1904 b Fk(\))75 358 y Fs(p)q(ossible)14 b(and)g(return.")166 497 y Fy(Example:)75 645 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h (ierr\))75 702 y(IF\(rank)g(<)h(2\))123 758 y(THEN)214 b(!)24 b(rank=0)f(or)g(rank=1)170 815 y(CALL)h(MPI_ISEND\(a,)e(n,)i(MPI_REAL,)e(2,)i (tag,)f(comm,)g(request,)g(ierr\))170 871 y(****)h(do)f(some)h(computation)e (to)i(mask)f(latency)g(****)170 927 y(CALL)h(MPI_WAIT\(request,)d(status,)i (ierr\))123 984 y(ELSE)214 b(!)24 b(rank=2)170 1040 y(CALL)g(MPI_IRECV\(a,)e (n,)i(MPI_REAL,)e(0,)i(tag,)f(comm,)g(request_list\(0\),)f(ierr\))170 1097 y(CALL)i(MPI_IRECV\(b,)e(n,)i(MPI_REAL,)e(1,)i(tag,)f(comm,)g (request_list\(1\),)f(ierr\))170 1153 y(****)i(do)f(some)h(computation)e(to)i (mask)f(latency)g(****)170 1210 y(CALL)h(MPI_WAITANY\(2,)e(request_list,)g (index,)h(status,)g(ierr\))170 1266 y(IF\(index.EQ.0\))218 1323 y(THEN)266 1379 y(****)g(handle)g(message)g(from)g(process)g(0)h(****) 266 1436 y(CALL)f(MPI_WAIT\(request_list\(1\),)e(status,)i(ierr\))266 1492 y(****)g(handle)g(message)g(from)g(process)g(1)h(****)218 1548 y(ELSE)266 1605 y(****)f(handle)g(message)g(from)g(process)g(1)h(****) 266 1661 y(CALL)f(MPI_WAIT\(request_list\(0\),)e(status,)i(ierr\))266 1718 y(****)g(handle)g(message)g(from)g(process)g(0)h(****)170 1774 y(END)g(IF)75 1831 y(END)f(IF)166 1923 y Fy(The)13 b(calls)h(in)o(tro)q (duced)f(in)h(this)f(subsection)h(can)f(b)q(e)g(used)g(to)f(w)o(ait)g(or)h (test)f(for)g(the)h(completion)g(of)75 1979 y(an)h(arbitrary)f(mix)h(of)f (non)o(blo)q(c)o(king)i(comm)o(unication)f(calls,)h(including)h(a)e(mix)g(of) f(sends)h(and)g(receiv)o(es)75 2036 y(and)h(of)g(an)o(y)g(additional)i(non)o (blo)q(c)o(king)f(comm)o(unication)g(calls)g(pro)o(vided)g(b)o(y)f(MPI.)166 2092 y(The)44 b Ft(status)25 b Fy(argumen)o(t)c(in)i Ft(MPI)p 779 2092 14 2 v 16 w(W)l(AIT)l(ANY)g Fy(and)f Ft(MPI)p 1196 2092 V 16 w(W)l(AIT)l(ALL)g Fy(and)g(the)h(en)o(tries)f(of)g(the)-1903 b Fm(\))75 2148 y Ft(a)o(rra)o(y)p 173 2148 V 15 w(of)p 225 2148 V 16 w(statuses)18 b Fy(argumen)o(t)d(in)i Ft(MPI)p 747 2148 V 16 w(W)l(AIT)l(ALL)e Fy(and)h Ft(MPI)p 1138 2148 V 16 w(TEST)l(ALL)g Fy(ma)o(y)f(b)q(e)i(used)f(and)g(up)q(dated)75 2205 y(for)j(a)g(send)h(op)q(eration)g(as)f(w)o(ell,)i(when)f(the)g(op)q (eration)g(is)g(canceled.)34 b(Th)o(us,)21 b(one)e(alw)o(a)o(ys)g(need)i(to) 75 2261 y(pro)o(vide)16 b(a)f(v)m(alid)h Ft(status)i Fy(\(resp.)c Ft(a)o(rra)o(y)p 737 2261 V 15 w(of)p 789 2261 V 16 w(statuses)p Fy(\))j(argumen)o(t.)166 2318 y(An)k Ft(a)o(rra)o(y)p 344 2318 V 15 w(of)p 396 2318 V 16 w(requests)i Fy(argumen)o(t)e(that)f(is)i(passed)g (to)e(an)o(y)h(of)g(the)h(last)f(four)g(functions)h(ma)o(y)75 2374 y(con)o(tain)14 b(n)o(ull)i(en)o(tries.)k(These)14 b(en)o(tries)h(are)e (ignored)i(and)f(the)h(call)g(executes)g(as)e(if)i(it)f(w)o(as)g(supplied)i (an)75 2431 y(arra)o(y)g(of)h(requests)h(con)o(taining)g(only)g(the)g(non-n)o (ull)h(en)o(tries.)27 b(As)17 b(a)h(request)f(in)h Ft(a)o(rra)o(y)p 1608 2431 V 15 w(of)p 1660 2431 V 16 w(requests)h Fy(is)75 2487 y(set)13 b(to)g(n)o(ull)i(when)f(the)g(corresp)q(onding)g(op)q(eration)g (completes,)g(this)g(allo)o(ws)g(to)f(reuse)g(the)h(same)f(arra)o(y)75 2544 y(for)j(completing)h(in)g(successiv)o(e)g(calls)h(the)e(op)q(erations)g (asso)q(ciated)h(with)f(this)h(arra)o(y)l(.)22 b(Th)o(us,)16 b(the)g(last)75 2600 y(example)g(can)f(b)q(e)h(rewritten)f(as)g(follo)o(ws:) 1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 27 27 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(27)75 49 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 106 y(IF\(rank)g(<)h(2\))123 162 y(THEN)214 b(!)24 b(rank=0)f(or)g(rank=1)170 219 y(CALL)h(MPI_ISEND\(a,)e(n,)i(MPI_REAL,)e(2,)i(tag,)f(comm,)g(request,)g (ierr\))170 275 y(CALL)h(MPI_WAIT\(request,)d(status,)i(ierr\))123 332 y(ELSE)214 b(!)24 b(rank=2)170 388 y(CALL)g(MPI_IRECV\(a,)e(n,)i (MPI_REAL,)e(0,)i(tag,)f(comm,)g(request_list\(0\),)f(ierr\))170 444 y(CALL)i(MPI_IRECV\(b,)e(n,)i(MPI_REAL,)e(1,)i(tag,)f(comm,)g (request_list\(1\),)f(ierr\))170 501 y(CALL)i(MPI_WAITANY\(2,)e (request_list,)g(index,)h(status,)g(ierr\))266 557 y(****)g(handle)g(first)g (arriving)g(message)47 b(****)170 614 y(CALL)24 b(MPI_WAITANY\(2,)e (request_list,)g(index,)h(status,)g(ierr\))266 670 y(****)g(handle)g(second)g (arriving)g(message)47 b(****)166 797 y Fy(If)18 b Ft(MPI)p 299 797 14 2 v 16 w(W)l(AIT)l(ANY)i Fy(or)d Ft(MPI)p 676 797 V 16 w(TEST)l(ANY)j Fy(are)e(called)i(with)e(an)g(empt)o(y)g Ft(a)o(rra)o(y)p 1534 797 V 15 w(of)p 1586 797 V 16 w(requests)i Fy(argu-)75 853 y(men)o(t)d(\(i.e.,)g(an)g(arra)o(y)f(of)h(length)h(zero,)f (or)f(an)i(arra)o(y)d(with)j(all)g(en)o(tries)g(n)o(ull\))g(then)f(they)h (return)f(the)75 910 y(v)m(alue)k Fo(MPI)p 274 910 13 2 v 14 w(UNDEFINED)d Fy(in)i Ft(index)p Fy(;)i Ft(MPI)p 812 910 14 2 v 16 w(W)l(AIT)l(ALL)d Fy(with)h(an)f(empt)o(y)g Ft(a)o(rra)o(y)p 1449 910 V 14 w(of)p 1500 910 V 16 w(requests)j Fy(argumen)o(t)75 966 y(has)15 b(no)g(e\013ect;)g Ft(MPI)p 437 966 V 16 w(TEST)l(ALL)g Fy(with)g(an)g(empt)o(y)g Ft(a)o(rra)o(y)p 1058 966 V 15 w(of)p 1110 966 V 16 w(requests)i Fy(argumen)o(t)e(returns)g Ft(\015ag=true)p Fy(.)166 1027 y(A)h(p)q(ersisten)o(t)g(request)f(for)g(whic)o(h)i(there)f(is) g(no)f(p)q(ending)j(comm)o(unication)e(is)g(handled)i(as)d(a)g(n)o(ul)75 1083 y(handle)h(when)g(it)g(o)q(ccurs)f(in)h(an)f(arra)o(y)f(of)h(requests)g ({)g(see)h(Section)g(1.10,)d(b)q(elo)o(w.)75 1228 y Fq(1.8.6)49 b(Alternative)16 b(p)o(rop)q(osal)189 1322 y Fp(R)n(ationale.)457 1306 y Fj(1)498 1322 y Fy(Some)g(applications)i(ma)o(y)d(p)q(ost)g(a)h(large) g(n)o(um)o(b)q(er)g(of)g(receiv)o(es)g(\(10's)f({)h(100's\))67 b Fm(\()189 1379 y Fy(and)18 b(w)o(ait)f(for)g(the)h(completion)h(of)e(an)o (y)l(.)27 b(This,)19 b(in)g(order)e(to)g(tak)o(e)g(adv)m(an)o(tage)g(of)h (ready)f(send)189 1435 y(proto)q(col)k(and)g(amortize)g(for)f(load)h(im)o (balances.)39 b(The)21 b(use)g(of)g(a)f(sequen)o(tial)j(structure)d(\(an)189 1492 y(arra)o(y)12 b(of)h(requests\))g(is)h(ine\016cien)o(t)i(as)d(it)h(has)f (to)g(b)q(e)h(searc)o(hed)g(sequen)o(tially)h(for)e(eac)o(h)h(call.)20 b(This)189 1548 y(results)d(in)h(a)f(linear)h(o)o(v)o(erhead)f(p)q(er)g(in)o (v)o(o)q(cation)h(and)f(a)g(total)f(of)h Fi(O)q Fy(\()p Fi(n)1429 1532 y Fj(2)1447 1548 y Fy(\))g(time)h(for)e(handling)j Fi(n)189 1604 y Fy(comm)o(unications.)k(Instead,)17 b(if)f(the)h(list)g(of)f(requests) g(is)h(an)f(opaque)g(MPI)g(ob)s(ject,)g(then)g(more)189 1661 y(e\016cien)o(t)k(data)f(structures)g(can)h(b)q(e)g(used,)h(reducing)g(the)f (o)o(v)o(erhead)f(p)q(er)h(comm)o(unication)h(to)189 1717 y Fi(O)q Fy(\(log)8 b(log)g Fi(n)p Fy(\),)20 b(where)g Fi(n)h Fy(is)f(the)g(total)f(size)i(of)e(the)h(lists)h(\(constan)o(t)e(time)h(is)g (p)q(ossible,)j(if)d(the)189 1774 y(lists)14 b(are)g(disjoin)o(t\).)20 b(Also,)14 b(if)h(the)f(list)h(of)e(requests)h(is)h(an)f(opaque)g(ob)s(ject,) f(one)i(ma)o(y)e(reconsider)189 1830 y(the)20 b(decision)j(to)d(ha)o(v)o(e)g (functions)h(that)f(complete)h(comm)o(unications)h(n)o(ullify)g(nonp)q (ersisten)o(t)189 1887 y(requests:)d(this)d(b)q(eha)o(vior)g(is)g(essen)o (tial)g(only)f(for)g(arra)o(y)f(of)h(requests.)189 1970 y(An)d(in)o(terface)h (whic)o(h)g(allo)o(ws)g(incremen)o(tal)g(mo)q(di\014cations)h(to)e(these)g (lists)h(of)f(requests)h(is)g(prefer-)189 2027 y(able)j(to)e(rep)q(eated)i (uses)g(of)e(a)h(commit-lik)o(e)i(function)f(on)f(a)g(static)g(arra)o(y)f (since)j(these)e(lists)h(will)189 2083 y(c)o(hange)11 b(gradually)g(in)h(man) o(y)e(applications)j(\(e.g.)k(n-b)q(o)q(dy)12 b(tree)f(algorithm\))g(\()p Fp(End)g(of)i(r)n(ationale.)p Fy(\))166 2210 y(A)23 b(new)f(opaque)h(ob)s (ject,)h(a)e Fo(collection)i Fy(is)f(added)g(to)f(MPI.)g(The)h(functions)h Ft(MPI)p 1648 2210 V 15 w(W)l(AIT)l(ANY)p Fy(,)75 2266 y Ft(MPI)p 160 2266 V 16 w(W)l(AIT)l(ALL)p Fy(,)17 b Ft(MPI)p 477 2266 V 16 w(TEST)l(ANY)j Fy(and)e Ft(MPI)p 888 2266 V 16 w(TEST)l(ALL)g Fy(accept)h(a)f Ft(collection)g Fy(argumen)o(t,)g(whic)o(h)h(is)75 2323 y(a)i(handle)i(to)e(a)g(set)g(of)g(requests,)i(instead)f(of)f(the)g Ft(a)o(rra)o(y)p 1113 2323 V 15 w(of)p 1165 2323 V 16 w(requests)j Fy(argumen)o(t)c(they)i(ha)o(v)o(e)f(no)o(w.)75 2379 y(Whenev)o(er)13 b(a)f(nonp)q(ersisten)o(t)h(request)f(is)h(satis\014ed)f(b)o(y)h(a)e(call)j (to)e Ft(MPI)p 1282 2379 V 15 w(W)l(AIT)l(ANY)i Fy(or)d Ft(MPI)p 1646 2379 V 16 w(TEST)l(ANY)p Fy(,)75 2436 y(it)16 b(is)g(dropp)q(ed)h(from)e (the)g(collection.)23 b(The)16 b(user)g(sp)q(eci\014es)h(an)f(index)h(for)e (eac)o(h)h(item)g(in)g(a)g(collection.)75 2492 y(When)f(a)f(call)h(to)f Ft(MPI)p 466 2492 V 16 w(W)l(AIT)p 595 2492 V 16 w(ANY)h Fy(or)f(other)g (similar)i(function)f(satis\014es)f(a)g(request,)h(then)f(the)h(index)75 2548 y(asso)q(ciated)d(with)g(this)h(request)e(is)i(returned)f(b)o(y)g(the)g (call.)20 b(This)12 b(index)h(can)f(b)q(e)h(an)f(address)g(\(a)f(p)q(oin)o (ter\))75 2605 y(in)16 b(C,)f(and)g(can)g(b)q(e)h(used)g(directly)h(to)d (access)h(the)h(data)e(that)h(w)o(as)f(receiv)o(ed.)p 75 2661 720 2 v 127 2688 a Fw(1)144 2704 y Fv(Prop)q(osal)h(and)f(rationale)h(b)o(y)e (Salmon)h(and)g(Edelsohn)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 28 28 bop 75 -100 a Fy(28)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(COLLECTION)p 453 49 V 16 w(CREA)l(TE\(collection\))117 128 y Fs(OUT)108 b Ft(collection)409 b Fs(collection)14 b(\(handle\))75 254 y Fr(int)23 b(int)h(MPI)p 341 254 15 2 v 17 w(Collection)p 598 254 V 15 w(create\(MPI)p 853 254 V 16 w(Comm)p 965 254 V 17 w(collection)e(*collection\))75 342 y(MPI)p 150 342 V 17 w(COLLECTION)p 407 342 V 16 w(CREATE\(COLLECTION,)f(IERROR\))170 399 y(INTEGER)i(COLLECTION,)g(IERROR)166 487 y Fy(The)14 b(function)h Ft(MPI)p 520 487 14 2 v 16 w(COLLECTION)p 813 487 V 16 w(CREA)l(TE)g Fy(creates)f(a)f(collection)j(ob)s(ject)d(for)g(m)o(ultiple)j(com-)75 544 y(pletion)h(tests.)75 649 y Ft(MPI)p 160 649 V 16 w(ADD)p 272 649 V 16 w(TO)p 352 649 V 17 w(COLLECTION\(request,)f(collection,)g(k)o (ey\))117 728 y Fs(IN)155 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))117 806 y(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 885 y(IN)155 b Ft(k)o(ey)525 b Fs(user-sp)q(eci\014ed)17 b(k)o(ey)d(for)f(request)j(\(address\))75 1011 y Fr(int)23 b(int)h(MPI)p 341 1011 15 2 v 17 w(Add)p 430 1011 V 16 w(to)p 494 1011 V 17 w(collection\(MPI)p 847 1011 V 16 w(Comm)p 959 1011 V 16 w(request)f(request,)g(MPI)p 1453 1011 V 17 w(Comm)p 1566 1011 V 16 w(collection)393 1068 y(collection,)g(MPI)p 755 1068 V 16 w(Aint)h(key\))75 1156 y(MPI)p 150 1156 V 17 w(ADD)p 239 1156 V 17 w(TO)p 304 1156 V 16 w(COLLECTION\(REQUEST,)e (COLLECTION,)g(KEY,)h(IERROR\))170 1212 y(INTEGER)g(REQUEST,)g(COLLECTION,)f (KEY,)i(IERROR)166 1300 y Fy(The)15 b(function)g Ft(MPI)p 521 1300 14 2 v 16 w(ADD)p 633 1300 V 16 w(TO)p 713 1300 V 17 w(COLLECTION)f Fy(inserts)h Ft(request)h Fy(in)o(to)f(the)f(set)g(of)g(requests)h(asso-)75 1357 y(ciated)h(with)f Ft(collection)h Fy(and)f(asso)q(ciates)h(the)f (user-sp)q(eci\014ed)i Ft(k)o(ey)f Fy(with)f(it.)75 1462 y Ft(MPI)p 160 1462 V 16 w(DELETE)p 346 1462 V 16 w(FROM)p 490 1462 V 17 w(COLLECTION\(request,)h(collection\))117 1541 y Fs(IN)155 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\)) 117 1620 y(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))75 1746 y Fr(int)23 b(int)h(MPI)p 341 1746 15 2 v 17 w(Delete)p 502 1746 V 16 w(from)p 614 1746 V 17 w(collection\(MPI)p 967 1746 V 15 w(Comm)p 1078 1746 V 17 w(request)e(request,)393 1802 y(MPI)p 468 1802 V 17 w(Comm)p 581 1802 V 17 w(collection)g (collection\))75 1890 y(MPI)p 150 1890 V 17 w(DELETE)p 311 1890 V 16 w(FROM)p 423 1890 V 17 w(COLLECTION\(REQUEST,)f(COLLECTION,)h (IERROR\))170 1947 y(INTEGER)h(REQUEST,)g(COLLECTION,)f(IERROR)166 2035 y Fy(The)d(function)g Ft(MPI)p 529 2035 14 2 v 16 w(DELETE)p 715 2035 V 16 w(FROM)p 859 2035 V 17 w(COLLECTION)f Fy(remo)o(v)o(es)g Ft(request)i Fy(from)e(the)h(set)f(of)g(re-)75 2091 y(quests)d(asso)q(ciated) h(with)f Ft(collection)p Fy(.)75 2197 y Ft(MPI)p 160 2197 V 16 w(W)l(AIT)l(ANY\(collection,)h(value,)f(status\))117 2276 y Fs(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 2354 y(OUT)108 b Ft(k)o(ey)525 b Fs(user-sp)q(eci\014ed)17 b(k)o(ey)d(\(address\))117 2433 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 2559 y Fr(int)23 b(int)h(MPI)p 341 2559 15 2 v 17 w(Waitany\(MPI)p 622 2559 V 15 w(Comm)p 733 2559 V 17 w(collection)e(collection,)h(MPI)p 1371 2559 V 17 w(Aint)g(*key,)393 2615 y(MPI)p 468 2615 V 17 w(Status)g(*status\))75 2704 y(MPI)p 150 2704 V 17 w(WAITANY\(COLLECTION,)e(KEY,)i(STATUS,)g (IERROR\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 29 29 bop 75 -100 a Fu(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b Fy(29)170 49 y Fr(INTEGER)23 b(COLLECTION,)g(KEY,)g(STATUS\(MPI)p 1010 49 15 2 v 16 w(STATUS)p 1170 49 V 16 w(SIZE\),)g(IERROR)166 157 y Fy(The)11 b(function)g Ft(MPI)p 513 157 14 2 v 16 w(W)l(AIT)l(ANY)h Fy(w)o(aits)e(for)g(the)h(completion)h(of)e(one)h(request)g(from)f(the)h (collection.)75 213 y(If)k(this)g(request)g(is)h(nonp)q(ersisten)o(t,)f(then) g(its)g(is)g(dropp)q(ed)h(from)e(the)h(collection.)22 b(The)15 b(k)o(ey)g(asso)q(ciated)75 270 y(with)h(the)f(request)g(is)h(returned)f(in)h Ft(k)o(ey)p Fy(.)75 395 y Ft(MPI)p 160 395 V 16 w(TEST)l(ANY\(collection,)g (\015ag,)f(k)o(ey)l(,)g(status\))117 493 y Fs(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 611 y(OUT)108 b Ft(\015ag)518 b Fo(true)11 b Fs(if)d(one)i(of)f(the)h(op)q(erations)g(is)f(complete)g Fo(false)p Fs(,)h(otherwise)905 667 y(\(logical\))117 785 y(OUT)108 b Ft(k)o(ey)525 b Fs(user-sp)q(eci\014ed)17 b(k)o(ey)d(\(address\))117 902 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 1048 y Fr(int)23 b(int)h(MPI)p 341 1048 15 2 v 17 w(Testany\(MPI)p 622 1048 V 15 w(Comm)p 733 1048 V 17 w(collection)e(collection,)h(int)g (*flag,)g(MPI)p 1633 1048 V 17 w(Aint)393 1104 y(*key,)g(MPI)p 611 1104 V 17 w(Status)g(*status\))75 1212 y(MPI)p 150 1212 V 17 w(TESTANY\(COLLECTION,)e(FLAG,)i(KEY,)h(STATUS,)e(IERROR\))170 1268 y(INTEGER)h(COLLECTION,)g(FLAG,)g(KEY,)g(STATUS\(MPI)p 1153 1268 V 16 w(STATUS)p 1313 1268 V 16 w(SIZE\),)g(IERROR)166 1376 y Ft(MPI)p 251 1376 14 2 v 16 w(TEST)l(ANY)c Fy(returns)f Ft(\015ag)g(=)g(true)g Fy(if)h(an)o(y)e(of)h(the)g(op)q(erations)g(asso)q (ciated)g(with)g Ft(collection)75 1433 y Fy(has)f(completed.)28 b(In)19 b(this)f(case,)f(it)h(returns)g(in)g Ft(k)o(ey)g Fy(the)f(user-sp)q (eci\014ed)j(v)m(alue)f(asso)q(ciated)f(with)g(the)75 1489 y(request)d(handle)g(whic)o(h)h(completed,)f(in)h Ft(status)g Fy(the)f(status)f(of)g(that)g(op)q(eration,)g(and)h(deallo)q(cates)h(the)75 1545 y(asso)q(ciated)f(comm)o(unications)h(ob)s(ject)f(if)h(ephemeral.)21 b(It)15 b(returns)g Ft(\015ag)g(=)g(false)p Fy(,)g(otherwise.)75 1670 y Ft(MPI)p 160 1670 V 16 w(W)l(AIT)l(ALL\(collection,)g(a)o(rra)o(y)p 684 1670 V 14 w(of)p 735 1670 V 16 w(k)o(eys,)g(a)o(rra)o(y)p 953 1670 V 15 w(of)p 1005 1670 V 16 w(statuses\))117 1769 y Fs(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 1886 y(OUT)108 b Ft(a)o(rra)o(y)p 416 1886 V 15 w(of)p 468 1886 V 16 w(k)o(eys)345 b Fs(arra)o(y)14 b(of)f(user-sp)q(eci\014ed)k(k)o (eys)d(\(arra)o(y)g(of)f(addresses\))117 2004 y(OUT)108 b Ft(a)o(rra)o(y)p 416 2004 V 15 w(of)p 468 2004 V 16 w(statuses)276 b Fs(arra)o(y)14 b(of)f(status)i(ob)r(jects)g(\(arra)o(y)f(of)f(Status\))75 2150 y Fr(int)23 b(int)h(MPI)p 341 2150 15 2 v 17 w(Waitall\(MPI)p 622 2150 V 15 w(Comm)p 733 2150 V 17 w(collection)e(collection,)h(MPI)p 1371 2150 V 17 w(Aint)g(*array)p 1651 2150 V 16 w(of)p 1715 2150 V 17 w(keys,)393 2206 y(MPI)p 468 2206 V 41 w(Status)g(*array)p 820 2206 V 16 w(of)p 884 2206 V 17 w(statuses\))75 2314 y(MPI)p 150 2314 V 17 w(WAITALL\(COLLECTION,)e(ARRAY)p 764 2314 V 17 w(OF)p 829 2314 V 16 w(KEYS,)j(ARRAY)p 1109 2314 V 16 w(OF)p 1173 2314 V 17 w(STATUSES,)f(IERROR\))170 2370 y(INTEGER)g(COLLECTION,)g (ARRAY)p 771 2370 V 16 w(OF)p 835 2370 V 17 w(KEYS,)g(ARRAY)p 1115 2370 V 17 w(OF)p 1180 2370 V 17 w(STATUSES\(MPI)p 1485 2370 V 15 w(STATUS)p 1644 2370 V 17 w(SIZE,*\),)170 2427 y(IERROR)166 2534 y Fy(The)14 b(function)g Ft(MPI)p 519 2534 14 2 v 16 w(W)l(AIT)l(ALL)f Fy(blo)q(c)o(ks)h(un)o(til)h(all)f(comm)o(unication)g(op)q(erations)g(asso)q (ciated)g(with)75 2591 y(the)g(requests)g(in)h Ft(collection)g Fy(complete.)20 b Ft(MPI)p 866 2591 V 16 w(W)l(AIT)l(ALL)13 b Fy(returns)h(the)g(user-sp)q(eci\014ed)j(k)o(ey)d(asso)q(ciated)75 2647 y(with)21 b(eac)o(h)g(request)g(handle)h(in)f Ft(a)o(rra)o(y)p 764 2647 V 15 w(of)p 816 2647 V 16 w(k)o(eys)g Fy(and)g(the)g(resp)q(ectiv)o (e)g(status)f(of)g(eac)o(h)h(op)q(eration)g(in)75 2704 y Ft(a)o(rra)o(y)p 173 2704 V 15 w(of)p 225 2704 V 16 w(statuses)p Fy(.)h(All)16 b(ephemeral)h(comm)o(unication)f(ob)s(jects)e(are)h(deallo)q(cated.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 30 30 bop 75 -100 a Fy(30)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(TEST)l(ALL\(collection,)f(\015ag,)g(a)o(rra)o(y)p 782 49 V 14 w(of)p 833 49 V 16 w(k)o(eys,)g(a)o(rra)o(y)p 1051 49 V 15 w(of)p 1103 49 V 16 w(statuses\))117 128 y Fs(INOUT)62 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 206 y(OUT)108 b Ft(\015ag)518 b Fo(true)15 b Fs(if)e(all)g(of)g(the)i(op)q (erations)f(are)g(complete)f(\(logical\))117 285 y(OUT)108 b Ft(a)o(rra)o(y)p 416 285 V 15 w(of)p 468 285 V 16 w(k)o(eys)345 b Fs(arra)o(y)14 b(of)f(user-sp)q(eci\014ed)k(k)o(eys)d(\(arra)o(y)g(of)f (addresses\))117 363 y(OUT)108 b Ft(a)o(rra)o(y)p 416 363 V 15 w(of)p 468 363 V 16 w(statuses)276 b Fs(arra)o(y)14 b(of)f(status)i(ob)r (jects)g(\(arra)o(y)f(of)f(Status\))75 489 y Fr(int)23 b(int)h(MPI)p 341 489 15 2 v 17 w(Testall\(MPI)p 622 489 V 15 w(Comm)p 733 489 V 17 w(collection)e(collection,)h(int)g(*flag,)g(MPI)p 1633 489 V 17 w(Aint)393 546 y(*array)p 540 546 V 17 w(of)p 605 546 V 16 w(keys,)h(MPI)p 837 546 V 16 w(Status)f(*array)p 1164 546 V 17 w(of)p 1229 546 V 17 w(statuses\))75 634 y(MPI)p 150 634 V 17 w(TESTALL\(COLLECTION,)e(FLAG,)i(ARRAY)p 907 634 V 17 w(OF)p 972 634 V 17 w(KEYS,)g(ARRAY)p 1252 634 V 16 w(OF)p 1316 634 V 17 w(STATUSES,)g(IERROR\))170 690 y(INTEGER)g(COLLECTION,)g(FLAG,) g(ARRAY)p 914 690 V 16 w(OF)p 978 690 V 17 w(KEYS,)170 747 y(ARRAY)p 293 747 V 17 w(OF)p 358 747 V 17 w(STATUSES\(MPI)p 663 747 V 15 w(STATUS)p 822 747 V 17 w(SIZE,*\),)g(IERROR)166 835 y Ft(MPI)p 251 835 14 2 v 16 w(TEST)l(ALL)12 b Fy(returns)g Ft(\015ag)h(=)g(true)g Fy(if)f(all)i(of)e(the)g(op)q(erations)h(asso)q (ciated)g(with)f Ft(collection)h Fy(ha)o(v)o(e)75 891 y(completed.)31 b(In)19 b(this)g(case,)g(it)f(returns)h(in)g(the)g(user-sp)q(eci\014ed)i(k)o (ey)d(asso)q(ciated)h(with)f(eac)o(h)h(request)75 947 y(handle)f(in)f Ft(a)o(rra)o(y)p 375 947 V 14 w(of)p 426 947 V 16 w(k)o(eys)g Fy(and)f(the)h(resp)q(ectiv)o(e)g(status)f(of)f(eac)o(h)i(op)q(eration)f(in)h Ft(a)o(rra)o(y)p 1564 947 V 15 w(of)p 1616 947 V 16 w(statuses)p Fy(.)26 b(All)75 1004 y(ephemeral)17 b(comm)o(unication)g(ob)s(jects)f(are)g (deallo)q(cated.)25 b(It)16 b(returns)g Ft(\015ag)g(=)h(false)p Fy(,)f(otherwise.)23 b(This)75 1060 y(is)16 b(a)f(non)o(blo)q(c)o(king)h(op)q (erations)f(that)g(returns)g(immediately)l(.)75 1166 y Ft(MPI)p 160 1166 V 16 w(COLLECTION)p 453 1166 V 16 w(SIZE\(collection,)g(count\))117 1244 y Fs(IN)155 b Ft(collection)409 b Fs(collection)14 b(\(handle\))117 1323 y(OUT)108 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(requests)i(in)d (collection)g(\(in)o(teger\))75 1449 y Fr(int)23 b(int)h(MPI)p 341 1449 15 2 v 17 w(Collection)p 598 1449 V 15 w(size\(MPI)p 805 1449 V 17 w(Comm)p 918 1449 V 16 w(collection)f(collection,int)f (*count\))75 1537 y(MPI)p 150 1537 V 17 w(COLLECTION)p 407 1537 V 16 w(SIZE\(COLLECTION,)f(COUNT,)i(IERROR\))p Ft(I)p Fr(NTEGER)f(COLLECTION,)g(COUNT,)393 1593 y(IERROR)166 1681 y Ft(MPI)p 251 1681 14 2 v 16 w(COLLECTION)p 544 1681 V 16 w(SIZE)f Fy(returns)h(the)f(n)o(um)o(b)q(er)h(of)f(requests)h(in)g(the)g (collection)h(for)e(whic)o(h)75 1738 y(there)16 b(is)h(an)f(incomplete)i (comm)o(unication.)24 b(The)16 b(coun)o(t)g(ma)o(y)g(b)q(e)g(used)h(as)f(the) g(size)i(of)d(the)i(arra)o(y)e(to)75 1794 y(pass)g(to)g Ft(MPI)p 315 1794 V 15 w(TEST)l(ALL)h Fy(and)f Ft(MPI)p 706 1794 V 16 w(W)l(AIT)l(ALL)p Fy(.)75 1900 y Ft(MPI)p 160 1900 V 16 w(COLLECTION)p 453 1900 V 16 w(FREE\(collection\))117 1978 y Fs(IN)155 b Ft(collection)409 b Fs(collection)14 b(\(handle\))75 2104 y Fr(int)23 b(int)h(MPI)p 341 2104 15 2 v 17 w(Collection)p 598 2104 V 15 w(free\(MPI)p 805 2104 V 17 w(Comm)p 918 2104 V 16 w(collection)f(collection\))75 2193 y(MPI)p 150 2193 V 17 w(COLLECTION)p 407 2193 V 16 w(FREE\(COLLECTION,)e (IERROR\))170 2249 y(INTEGER)i(COLLECTION,)g(IERROR)166 2337 y Fy(The)12 b(function)g Ft(MPI)p 515 2337 14 2 v 16 w(COLLECTION)p 808 2337 V 16 w(FREE)h Fy(deallo)q(cates)f(the)g(collection)h(ob)s(ject.)18 b(It)12 b(is)g(erroneous)75 2393 y(to)j(initiate)h(a)f(m)o(ultiple)i (completion)f(test)f(with)h(a)e(collection)j(that)e(has)g(b)q(een)h(freed.) 166 2534 y Fl(Discussion:)34 b Fs(Some)13 b(\(less)h(p)q(o)o(w)o(erful\))g (alternativ)o(es:)75 2591 y(Drop)f(the)g(k)o(ey:)18 b(the)13 b(user)h(will)e(need)h(to)g(use)h(message)e(tags,)h(in)f(order)i(to)f(iden)o (tify)f(requests)j(\(to)e(the)g(least,)g(this)75 2647 y(requires)i Fo(MPI)p 308 2647 13 2 v 14 w(GET)p 403 2647 V 15 w(T)m(A)o(G)e Fs(to)h(w)o(ork)f(after)h(a)g(send,)g(to)q(o\).)75 2704 y(Drop)g(the)p 255 2704 V 29 w(ADD)f(and)p 459 2704 V 29 w(DELETE.)h(Basically)m(,)e(this)i (amoun)o(ts)e(to)i(ha)o(ving)f(a)g(\\commit)p 1482 2704 V 12 w(arra)o(y")h(function.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 31 31 bop 75 -100 a Fu(1.9.)34 b(PR)o(OBE)16 b(AND)f(CANCEL)1161 b Fy(31)75 49 y Fz(1.9)59 b(Prob)r(e)20 b(and)f(Cancel)75 151 y Fy(The)d Ft(MPI)p 254 151 14 2 v 16 w(PROBE)g Fy(and)g Ft(MPI)p 605 151 V 16 w(IPROBE)g Fy(op)q(erations)g(allo)o(w)g(incoming)h(messages)e (to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)75 207 y(without)g(actually)h (receiving)h(them.)k(The)17 b(user)f(can)g(then)h(decide)g(ho)o(w)f(to)g (receiv)o(e)h(them,)f(based)g(on)75 264 y(the)i(information)f(returned)h(b)o (y)f(the)h(prob)q(e)g(\(basically)l(,)h(the)e(information)h(returned)g(b)o(y) f Ft(status)p Fy(\).)28 b(In)75 320 y(particular,)16 b(the)g(user)g(ma)o(y)f (allo)q(cate)h(memory)f(for)g(the)h(receiv)o(e)h(bu\013er,)e(according)h(to)f (the)h(length)g(of)75 377 y(the)f(prob)q(ed)h(message.)166 433 y(The)i Ft(MPI)p 347 433 V 16 w(CANCEL)g Fy(op)q(eration)g(allo)o(ws)g(p) q(ending)i(comm)o(unications)e(to)g(b)q(e)g(canceled.)30 b(This)18 b(is)75 489 y(required)i(for)f(clean)o(up.)33 b(P)o(osting)19 b(a)g(send)h(or)f(a)g(receiv)o(e)h(ties)f(up)h(user)f(resources)h(\(send)f (or)g(receiv)o(e)75 546 y(bu\013ers\),)14 b(and)i(a)f(cancel)h(ma)o(y)f(b)q (e)g(needed)i(to)d(free)i(these)f(resources)g(gracefully)l(.)75 650 y Ft(MPI)p 160 650 V 16 w(IPROBE\(source,)h(tag,)f(comm)m(,)c(\015ag,)k (status\))117 727 y Fs(IN)155 b Ft(source)468 b Fs(source)15 b(rank,)f(or)27 b Fo(MPI)p 1281 727 13 2 v 15 w(ANY)p 1381 727 V 14 w(SOURCE)13 b Fs(\(in)o(teger\))117 801 y(IN)155 b Ft(tag)526 b Fs(tag)14 b(v)n(alue)f(or)28 b Fo(MPI)p 1226 801 V 14 w(ANY)p 1325 801 V 15 w(T)m(A)o(G)12 b Fs(\(in)o(teger\))117 876 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 951 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))117 1025 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 1150 y Fr(int)23 b(MPI)p 245 1150 15 2 v 17 w(Iprobe\(int)g(source,)g(int)g (tag,)g(MPI)p 1002 1150 V 17 w(Comm)g(comm,)h(int)f(*flag,)393 1206 y(MPI)p 468 1206 V 17 w(Status)g(*status\))75 1292 y(MPI)p 150 1292 V 17 w(IPROBE\(SOURCE,)f(TAG,)h(COMM,)g(FLAG,)g(STATUS,)g(IERROR\)) 170 1349 y(LOGICAL)g(FLAG)170 1405 y(INTEGER)g(SOURCE,)g(TAG,)h(COMM,)f (STATUS\(MPI)p 1058 1405 V 16 w(STATUS)p 1218 1405 V 16 w(SIZE\),)g(IERROR) 166 1492 y Ft(MPI)p 251 1492 14 2 v 16 w(IPROBE)c Fy(returns)f Ft(\015ag)g(=)h(true)f Fy(if)h(there)g(is)f(a)g(message)g(that)g(can)g(b)q(e) h(receiv)o(ed)g(and)g(that)75 1548 y(matc)o(hes)i(the)g(pattern)f(sp)q (eci\014ed)j(b)o(y)e(the)h(argumen)o(ts)e Ft(source)p Fy(,)i Ft(tag)p Fy(,)g(and)g Ft(comm)m Fy(.)34 b(It)21 b(returns)g Ft(\015ag)75 1605 y(=)i(false)p Fy(,)h(otherwise.)41 b(If)23 b Ft(MPI)p 633 1605 V 16 w(IPROBE)g Fy(returns)f Ft(\015ag)g(=)h(true)p Fy(,)h(then)f(the)f(status)g(ob)s(ject)g(can)g(b)q(e)75 1661 y(subsequen)o(tly)c(queried)g(using)g(the)f(functions)h(describ)q(ed)g(in)g (section)g(1.3.1)d(to)h(\014nd)i(the)f(source,)g(tag)75 1718 y(and)f(length)h(of)f(the)g(message.)23 b(The)16 b(v)m(alues)i(returned)e(b)o (y)h(these)f(queries)h(are)f(the)g(same)g(v)m(alues)i(that)75 1774 y(w)o(ould)12 b(ha)o(v)o(e)e(b)q(een)j(returned)e(b)o(y)h(a)f(call)h(to) e Ft(MPI)p 897 1774 V 16 w(RECV)i Fy(executed)g(at)f(the)g(same)g(p)q(oin)o (t)h(in)g(the)f(program.)75 1831 y(The)k(return)h(status)e(ob)s(ject)h(is)g (unde\014ned)i(if)f Ft(\015ag)f(=)h(false)p Fy(.)166 1887 y(A)d(subsequen)o (t)i(receiv)o(e)f(executed)g(with)g(the)g(same)f(con)o(text,)g(and)g(the)h (source)f(and)h(tag)f(returned)75 1943 y(b)o(y)g(querying)h(the)f(status)f (returned)i(b)o(y)f Ft(MPI)p 858 1943 V 16 w(IPROBE)h Fy(will)h(receiv)o(e)f (the)f(message)f(that)h(w)o(as)f(matc)o(hed)75 2000 y(b)o(y)21 b(the)g(prob)q(e,)h(if)g(no)f(other)f(in)o(terv)o(ening)i(receiv)o(e)g(o)q (ccurred)f(after)g(the)g(prob)q(e.)37 b(If)21 b(the)g(receiving)75 2056 y(pro)q(cess)e(is)h(m)o(ulti-threaded,)h(it)f(is)f(the)h(user's)f(resp)q (onsibilit)o(y)i(to)e(ensure)h(that)e(the)i(last)f(condition)75 2113 y(holds.)166 2169 y(The)d Ft(source)h Fy(argumen)o(t)e(of)h Ft(MPI)p 738 2169 V 16 w(PROBE)h Fy(can)f(b)q(e)h Fo(MPI)p 1140 2169 13 2 v 14 w(ANY)p 1239 2169 V 15 w(SOURCE)p Fy(,)d(and)i(the)h Ft(tag)f Fy(argumen)o(t)75 2226 y(can)g(b)q(e)g Fo(MPI)p 298 2226 V 15 w(ANY)p 398 2226 V 14 w(T)m(A)o(G)p Fy(,)f(so)g(that)g(one)h(can)g (prob)q(e)g(for)f(messages)h(from)f(an)h(arbitrary)f(source)h(and/or)75 2282 y(with)g(an)g(arbitrary)f(tag.)20 b(Ho)o(w)o(ev)o(er,)15 b(a)g(sp)q(eci\014c)j(comm)o(unication)e(con)o(text)f(m)o(ust)g(pro)o(vided)i (with)f(the)75 2339 y Ft(comm)9 b Fy(argumen)o(t.)166 2395 y(A)32 b(call)17 b(to)e Ft(MPI)p 457 2395 14 2 v 16 w(PROBE)i Fy(will)g(not)f(matc)o(h)f(a)g(message)h(that)f(has)h(already)g(b)q(een)h (matc)o(hed)e(b)o(y)h(a)69 b Fm(\()75 2452 y Fy(previous)17 b(receiv)o(e)g(op)q(eration.)23 b(It)16 b(ma)o(y)f(matc)o(h)h(a)f(message)h (that)f(has)h(b)q(een)h(matc)o(hed)f(b)o(y)g(a)g(previous)75 2508 y(prob)q(e)f(op)q(eration)g(or)f(b)o(y)h(a)f(concurren)o(t)h(receiv)o(e) g(\(e.g.,)e(a)h(receiv)o(e)i(executed)g(b)o(y)e(a)g(concurren)o(t)h(thread)75 2564 y(in)h(a)f(m)o(ultithreaded)h(en)o(vironmen)o(t.)166 2704 y Fl(Implemen)o(tati)o(on)g(note:)47 b Fs(MPI)17 b(guaran)o(tees)h(that)f (successiv)o(e)i(messages)d(sen)o(t)i(from)d(a)i(source)h(to)e(a)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 32 32 bop 75 -100 a Fy(32)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fs(destination)10 b(within)f(the)i (same)f(con)o(text)h(are)f(receiv)o(ed)i(in)d(the)i(order)g(they)g(are)g(sen) o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)g(supp)q(ort,)75 106 y(either)k(explicitly)e(or)h(implicitly)l(,)d(a)j(FIF)o(O)g(structure)i(to)e (manage)f(messages)h(b)q(et)o(w)o(een)h(eac)o(h)g(pair)e(of)h(pro)q(cesses,) 75 162 y(for)k(eac)o(h)g(con)o(text.)56 b Fo(MPI)p 516 162 13 2 v 14 w(PROBE)16 b Fs(returns)i(information)c(on)j(the)h(\014rst)g(matc)o (hing)d(message)i(in)f(this)h(FIF)o(O;)75 219 y(this)d(will)e(also)i(b)q(e)g (the)h(message)e(receiv)o(ed)i(b)o(y)f(the)g(\014rst)h(subsequen)o(t)h (receiv)o(e)f(with)e(the)i(same)e(source,)i(tag)e(and)75 275 y(con)o(text)i(as)f(the)g(message)g(matc)o(hed)f(b)o(y)g Fo(MPI)p 806 275 V 15 w(PROBE)p Fs(.)75 463 y Ft(MPI)p 160 463 14 2 v 16 w(PROBE\(source,)j(tag,)f(comm)m(,)d(status\))117 543 y Fs(IN)155 b Ft(source)468 b Fs(source)15 b(rank,)f(or)27 b Fo(MPI)p 1281 543 13 2 v 15 w(ANY)p 1381 543 V 14 w(SOURCE)13 b Fs(\(in)o(teger\))117 622 y(IN)155 b Ft(tag)526 b Fs(tag)14 b(v)n(alue,)e(or)28 b Fo(MPI)p 1237 622 V 14 w(ANY)p 1336 622 V 15 w(T)m(A)o(G)13 b Fs(\(in)o(teger\))117 701 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 780 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 906 y Fr(int)23 b(MPI)p 245 906 15 2 v 17 w(Probe\(int)g(source,)g(int)g (tag,)h(MPI)p 979 906 V 16 w(Comm)g(comm,)f(MPI)p 1330 906 V 17 w(Status)g(*status\))75 994 y(MPI)p 150 994 V 17 w(PROBE\(SOURCE,)f (TAG,)h(COMM,)g(STATUS,)g(IERROR\))170 1051 y(INTEGER)g(SOURCE,)g(TAG,)h (COMM,)f(STATUS\(MPI)p 1058 1051 V 16 w(STATUS)p 1218 1051 V 16 w(SIZE\),)g(IERROR)166 1139 y Ft(MPI)p 251 1139 14 2 v 16 w(PROBE)17 b Fy(b)q(eha)o(v)o(es)g(lik)o(e)g Ft(MPI)p 769 1139 V 16 w(IPROBE)g Fy(except)g(that)f(it)g(is)h(a)f(blo)q(c)o(king)i(call)f (whic)o(h)g(returns)75 1196 y(only)f(after)e(a)h(matc)o(hing)g(message)g(has) g(b)q(een)h(found.)166 1254 y(An)24 b(MPI)f(implemen)o(tation)h(of)f Ft(MPI)p 838 1254 V 16 w(PROBE)i Fy(needs)f(to)e(guaran)o(tee)h(progress:)35 b(if)24 b(a)f(call)i(to)75 1310 y Ft(MPI)p 160 1310 V 16 w(PROBE)20 b Fy(has)f(b)q(een)h(issued)h(b)o(y)e(a)f(pro)q(cess,)i(and)g(a)e(send)i (that)e(matc)o(hes)h(the)g(prob)q(e)h(has)f(b)q(een)75 1367 y(initiated)g(b)o(y)e(some)g(pro)q(cess,)h(then)g(the)f(call)i(to)e Ft(MPI)p 1029 1367 V 15 w(PROBE)i Fy(will)g(return,)f(unless)g(the)g(message) e(is)75 1423 y(receiv)o(ed)21 b(b)o(y)e(another)g(concurren)o(t)h(receiv)o(e) h(op)q(eration)e(\(that)g(is)h(executed)g(b)o(y)g(another)f(thread)h(at)75 1480 y(the)c(probing)g(pro)q(cess\).)21 b(Similarly)l(,)d(if)e(a)g(pro)q (cess)g(busy)g(w)o(aits)f(with)h Ft(MPI)p 1374 1480 V 16 w(IPROBE)h Fy(and)f(a)f(matc)o(hing)75 1536 y(message)j(has)g(b)q(een)h(issued,)h(then)e (the)g(call)i(to)d Ft(MPI)p 1015 1536 V 16 w(IPROBE)i Fy(will)h(ev)o(en)o (tually)f(return)f Ft(\015ag)g(=)g(true)75 1593 y Fy(unless)e(the)g(message)e (is)i(receiv)o(ed)g(b)o(y)g(another)e(concurren)o(t)i(receiv)o(e)g(op)q (eration.)75 1698 y Ft(MPI)p 160 1698 V 16 w(CANCEL\(request\))117 1777 y Fs(IN)155 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 1904 y Fr(int)23 b(MPI)p 245 1904 15 2 v 17 w(Cancel\(MPI)p 502 1904 V 16 w(Comm)p 614 1904 V 17 w(request)g(*request\))75 1992 y(MPI)p 150 1992 V 17 w(CANCEL\(REQUEST,)f (IERROR\))170 2049 y(INTEGER)h(REQUEST,)g(IERROR)166 2137 y Fy(A)17 b(call)h(to)e Ft(MPI)p 444 2137 14 2 v 15 w(CANCEL)i Fy(marks)e(for)g(cancellation)i(a)f(p)q(ending)h(non)o(blo)q(c)o(king)g(comm) o(unication)75 2194 y(op)q(eration)f(\(send)g(or)g(receiv)o(e\).)25 b(The)18 b(cancel)g(call)g(is)f(non)o(blo)q(c)o(king,)h(and)f(lo)q(cal.)27 b(It)17 b(returns)f(immedi-)75 2250 y(ately)l(,)f(p)q(ossibly)i(b)q(efore)e (the)h(comm)o(unication)g(is)f(actually)h(canceled.)166 2308 y(It)c(is)g(still)h(necessary)f(to)g(complete)g(a)g(comm)o(unication)g(that)f (has)h(b)q(een)h(mark)o(ed)e(for)h(cancellation,)75 2365 y(using)i(a)g(call)h (to)e Ft(MPI)p 450 2365 V 15 w(W)l(AIT)h Fy(or)f Ft(MPI)p 728 2365 V 16 w(TEST)h Fy(\(or)f(an)o(y)g(of)g(the)h(deriv)o(ed)h(op)q (erations\).)k(If)14 b(the)g(op)q(eration)75 2421 y(has)e(b)q(een)h (canceled,)h(then)f(information)f(to)g(that)f(e\013ect)h(will)i(b)q(e)f (returned)f(in)h(the)g(status)e(argumen)o(t)g(of)75 2478 y(the)j(op)q (eration)g(that)f(completes)h(the)g(comm)o(unication.)20 b(If)14 b(a)g(comm)o(unication)g(is)g(mark)o(ed)g(for)f(cancel-)75 2534 y(lation,)j(then)h(a)e Ft(MPI)p 445 2534 V 16 w(W)l(AIT)h Fy(call)h(for)f(that)f(comm)o(unication)h(is)h(guaran)o(teed)e(to)h(return,)f (irresp)q(ectiv)o(e)75 2591 y(of)i(the)g(activities)i(of)e(other)g(pro)q (cesses)h(\(i.e.,)f Ft(MPI)p 970 2591 V 16 w(W)l(AIT)g Fy(b)q(eha)o(v)o(es)h (as)f(a)g(lo)q(cal)i(function\);)f(similarly)75 2647 y(if)g Ft(MPI)p 204 2647 V 16 w(TEST)f Fy(is)h(rep)q(eatedly)g(called)h(in)f(a)f (busy)h(w)o(ait)e(lo)q(op)i(for)f(a)g(canceled)i(comm)o(unication,)f(then)75 2704 y Ft(MPI)p 160 2704 V 16 w(TEST)d Fy(will)i(ev)o(en)o(tually)f(b)q(e)g (successful.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 33 33 bop 75 -100 a Fu(1.10.)34 b(PERSISTENT)16 b(COMMUNICA)l(TION)g(REQUESTS)638 b Fy(33)166 49 y(Either)19 b(the)f(cancellation)i(succeeds,)f(or)f(the)g (comm)o(unication)h(succeeds,)h(but)e(not)g(b)q(oth.)29 b(If)18 b(a)75 106 y(send)h(is)h(mark)o(ed)e(for)g(cancellation,)j(then)e(it)g(m)o (ust)f(b)q(e)i(the)f(case)g(that)f(either)h(the)g(send)g(completes)75 162 y(normally)l(,)f(in)h(whic)o(h)f(case)f(the)h(message)f(sen)o(t)g(w)o(as) g(receiv)o(ed)h(at)f(the)h(destination)g(pro)q(cess,)g(or)f(that)75 219 y(the)f(send)g(is)g(successfully)i(canceled,)f(in)g(whic)o(h)f(case)g(no) g(part)f(of)g(the)h(message)f(w)o(as)g(receiv)o(ed)i(at)e(the)75 275 y(destination.)21 b(Then,)15 b(an)o(y)f(matc)o(hing)h(receiv)o(e)h(has)e (to)h(b)q(e)g(satis\014ed)h(b)o(y)e(another)h(send.)20 b(If)15 b(a)g(receiv)o(e)g(is)75 332 y(mark)o(ed)d(for)f(cancellation,)k(then)d(it)h (m)o(ust)e(b)q(e)i(the)g(case)f(that)f(either)i(the)g(receiv)o(e)g(completes) g(normally)l(,)75 388 y(or)18 b(that)g(the)g(receiv)o(e)i(is)f(successfully)h (canceled,)h(in)e(whic)o(h)h(case)e(no)h(part)f(of)g(the)g(receiv)o(e)i (bu\013er)e(is)75 444 y(altered.)i(Then,)c(an)o(y)e(matc)o(hing)i(send)g(has) f(to)f(b)q(e)i(satis\014ed)g(b)o(y)f(another)g(receiv)o(e.)75 548 y Ft(MPI)p 160 548 14 2 v 16 w(TEST)p 290 548 V 16 w(CANCELLED\(status,)i (\015ag\))117 625 y Fs(IN)155 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))117 699 y(OUT)108 b Ft(\015ag)518 b Fs(\(logical\))75 824 y Fr(int)23 b(MPI)p 245 824 15 2 v 17 w(Test)p 358 824 V 17 w(cancelled\(MPI)p 687 824 V 15 w(Status)g(status,)g (int)h(*flag\))75 910 y(MPI)p 150 910 V 17 w(TEST)p 263 910 V 16 w(CANCELLED\(STATUS,)e(FLAG,)h(IERROR\))170 966 y(LOGICAL)g(FLAG)170 1023 y(INTEGER)g(STATUS\(MPI)p 604 1023 V 16 w(STATUS)p 764 1023 V 17 w(SIZE\),)g(IERROR)166 1109 y Fy(Returns)12 b Ft(\015ag)f(=)h(true) g Fy(if)f(the)h(comm)o(unication)g(asso)q(ciated)f(with)h(the)g(status)e(ob)s (ject)h(w)o(as)f(canceled)75 1166 y(successfully)l(.)27 b(In)18 b(suc)o(h)f(a)g(case,)g(all)g(other)g(\014elds)h(of)e Ft(status)j Fy(\(suc)o(h)e(as)g Ft(count)h Fy(or)e Ft(tag)p Fy(\))h(are)f(unde\014ned.)75 1222 y(Returns)i Ft(\015ag)f(=)h(false)p Fy(,)f(otherwise.)27 b(If)18 b(a)f(receiv)o(e)h(op)q(eration)g(migh)o(t)f(b)q(e)h(canceled)h(then) e(one)h(should)75 1279 y(call)41 b Ft(MPI)p 268 1279 14 2 v 16 w(TEST)p 398 1279 V 17 w(CANCELLED)20 b Fy(\014rst,)h(to)e(c)o(hec)o(k)i (whether)f(the)g(op)q(eration)h(w)o(as)e(canceled,)k(b)q(efore)75 1335 y(c)o(hec)o(king)16 b(on)f(the)g(other)g(\014elds)i(of)d(the)i(return)f (status.)166 1474 y Fl(Implemen)o(tati)o(on)h(note:)51 b Fs(If)18 b(a)f(send)h(op)q(eration)g(uses)h(an)e(\\eager")h(proto)q(col)f(\(data)h(is) f(transferred)75 1531 y(to)e(the)h(receiv)o(er)h(b)q(efore)f(a)g(matc)o(hing) d(receiv)o(e)k(is)e(p)q(osted\),)i(then)f(the)g(cancelation)f(of)g(this)g (send)i(ma)o(y)c(require)75 1587 y(comm)o(unication)j(with)j(the)h(in)o (tended)g(receiv)o(er,)h(in)e(order)h(to)f(free)h(allo)q(cated)f(bu\013ers.) 36 b(On)19 b(some)g(systems)75 1644 y(this)14 b(ma)o(y)e(require)j(an)f(in)o (terrupt)h(to)e(the)i(in)o(tended)g(receiv)o(er.)20 b(Therefore,)15 b(it)e(is)h(exp)q(ected)i(that)e Fo(MPI)p 1705 1644 13 2 v 15 w(CANCEL)75 1700 y Fs(will)h(b)q(e)i(an)f(exp)q(ensiv)o(e)i(op)q(eration)e (on)g(man)o(y)f(systems,)i(to)f(b)q(e)h(used)g(only)f(exceptionally)m(.)25 b(Note)17 b(that,)f(while)75 1757 y(comm)o(unication)f(ma)o(y)h(b)q(e)i (needed)i(to)e(implemen)o(t)d Fo(MPI)p 1006 1757 V 14 w(CANCEL)p Fs(,)i(this)h(is)g(still)f(a)h(lo)q(cal)f(op)q(eration,)i(in)e(the)75 1813 y(sense)h(that)e(its)g(completion)e(do)q(es)j(not)f(dep)q(end)h(on)f (the)g(co)q(de)h(executed)h(b)o(y)e(other)g(pro)q(cesses.)27 b(If)16 b(pro)q(cessing)75 1869 y(is)d(required)i(on)e(another)h(pro)q(cess,) h(this)e(should)g(b)q(e)i(transparen)o(t)f(to)f(the)i(application)d(\(hence)j (the)f(need)g(for)f(an)75 1926 y(in)o(terrupt)i(and)e(an)h(in)o(terrupt)h (handler\).)75 2150 y Fz(1.10)60 b(P)n(ersistent)21 b(comm)o(unication)c (requests)75 2252 y Fy(Often)22 b(a)g(comm)o(unication)h(with)f(the)g(same)g (argumen)o(t)f(list)i(is)g(rep)q(eatedly)g(executed)g(within)g(the)75 2308 y(inner)c(lo)q(op)g(of)e(a)h(parallel)i(computation.)28 b(In)18 b(suc)o(h)h(a)f(situation,)g(it)h(ma)o(y)e(b)q(e)i(p)q(ossible)g(to)f (optimize)75 2365 y(the)h(comm)o(unication)g(b)o(y)g(binding)h(the)f(list)g (of)g(comm)o(unication)g(argumen)o(ts)f(to)g(a)g(comm)o(unication)75 2421 y(request)11 b(once)g(and,)g(then,)h(rep)q(eatedly)g(using)g(the)f(comm) o(unication)g(request)g(to)f(initiate)i(and)f(complete)75 2478 y(messages.)19 b(The)d(comm)o(unication)f(request)h(th)o(us)f(created)g(can)g (b)q(e)h(though)o(t)e(of)h(as)f(a)h(comm)o(unication)75 2534 y(p)q(ort)c(or)f(a)h(\\half-c)o(hannel")h(.)18 b(It)12 b(do)q(es)f(not)f(pro) o(vide)i(the)f(full)h(functionalit)o(y)h(of)d(a)h(con)o(v)o(en)o(tional)g(c)o (hannel,)75 2591 y(since)i(there)g(is)g(no)f(binding)i(of)e(the)g(send)h(p)q (ort)f(to)f(the)i(receiv)o(e)g(p)q(ort:)18 b(this)13 b(construct)f(allo)o(ws) g(reduction)75 2647 y(of)18 b(the)g(o)o(v)o(erhead)f(for)h(comm)o(unication)g (b)q(et)o(w)o(een)h(pro)q(cessor)e(and)h(comm)o(unication)h(con)o(troller,)g (but)75 2704 y(not)13 b(of)g(the)g(o)o(v)o(erhead)g(for)g(comm)o(unication)h (b)q(et)o(w)o(een)g(one)f(comm)o(unication)h(con)o(troller)g(and)g(another.) -32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 34 34 bop 75 -100 a Fy(34)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fy(A)h(comm)o(unication)g(request)g (is)g(created)g(using)h(one)f(of)f(the)h(four)f(follo)o(wing)i(calls.)26 b(These)17 b(calls)75 106 y(in)o(v)o(olv)o(e)f(no)f(comm)o(unication.)166 167 y(The)10 b(function)i Ft(MPI)p 513 167 14 2 v 15 w(CREA)l(TE)p 697 167 V 18 w(SEND)f Fy(creates)f(a)g(comm)o(unication)h(request)f(for)g(a)g (standard)g(mo)q(de)75 224 y(send)16 b(op)q(eration,)f(and)g(binds)i(to)d(it) i(all)g(the)f(argumen)o(ts)f(of)h(a)g(send)h(op)q(eration.)75 332 y Ft(MPI)p 160 332 V 16 w(CREA)l(TE)p 345 332 V 17 w(SEND\(buf,)g(count,) g(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(,)d(request\))117 415 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 500 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117 585 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h (elemen)o(t)g(\(handle\))117 670 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 755 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 840 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 925 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 1055 y Fr(int)23 b(MPI)p 245 1055 15 2 v 17 w(Create)p 406 1055 V 17 w(send\(void*)f(buf,)h (int)h(count,)f(MPI)p 1139 1055 V 17 w(Datatype)f(datatype,)h(int)h(dest,)393 1111 y(int)g(tag,)f(MPI)p 683 1111 V 17 w(Comm)g(comm,)g(MPI)p 1034 1111 V 17 w(Comm)p 1147 1111 V 17 w(request)g(*request\))75 1203 y(MPI)p 150 1203 V 17 w(CREATE)p 311 1203 V 16 w(SEND\(BUF,)g(COUNT,)g (DATATYPE,)f(DEST,)i(TAG,)f(COMM,)g(REQUEST,)g(IERROR\))170 1259 y()g(BUF\(*\))170 1316 y(INTEGER)g(REQUEST,)g(COUNT,)g(DATATYPE,)g (DEST,)g(TAG,)g(COMM,)g(REQUEST,)g(IERROR)166 1407 y Fy(The)16 b(function)h Ft(MPI)p 524 1407 14 2 v 16 w(CREA)l(TE)p 709 1407 V 17 w(RSEND)g Fy(creates)e(a)h(comm)o(unication)h(ob)s(ject)e(for)g(a)h (ready)g(mo)q(de)75 1464 y(send)g(op)q(eration.)75 1572 y Ft(MPI)p 160 1572 V 16 w(CREA)l(TE)p 345 1572 V 17 w(RSEND\(buf,)g(count,)g(datat)o (yp)q(e,)h(dest,)f(tag,)f(comm)m(,)d(request\))117 1655 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 1740 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117 1825 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g (\(handle\))117 1910 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h (\(in)o(teger\))117 1995 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 2080 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator) 11 b(\(handle\))117 2165 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 2295 y Fr(int)23 b(MPI)p 245 2295 15 2 v 17 w(Create)p 406 2295 V 17 w(rsend\(void*)f(buf,)h(int)h(count,)f (MPI)p 1163 2295 V 17 w(Datatype)f(datatype,)h(int)g(dest,)393 2351 y(int)h(tag,)f(MPI)p 683 2351 V 17 w(Comm)g(comm,)g(MPI)p 1034 2351 V 17 w(Comm)p 1147 2351 V 17 w(request)g(*request\))75 2443 y(MPI)p 150 2443 V 17 w(CREATE)p 311 2443 V 16 w(RSEND\(BUF,)g(COUNT,)g (DATATYPE,)f(DEST,)h(TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 2499 y()g(BUF\(*\))170 2556 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 2647 y Fy(The)15 b(function)i Ft(MPI)p 523 2647 14 2 v 15 w(CREA)l(TE)p 707 2647 V 18 w(SSEND)f Fy(creates)f(a)g(comm)o(unication)h(ob)s(ject)e(for)h(a)g(sync)o(hronous)75 2704 y(mo)q(de)g(send)h(op)q(eration.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 35 35 bop 75 -100 a Fu(1.10.)34 b(PERSISTENT)16 b(COMMUNICA)l(TION)g(REQUESTS)638 b Fy(35)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(CREA)l(TE)p 345 49 V 17 w(SSEND\(buf,)16 b(count,)g(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m (,)d(request\))117 126 y Fs(IN)155 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 202 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o (teger\))117 277 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 352 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 427 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(\(in)o(teger\))117 502 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 577 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))75 702 y Fr(int)23 b(MPI)p 245 702 15 2 v 17 w(Create)p 406 702 V 17 w(ssend\(void*)f(buf,)h(int)h(count,)f(MPI)p 1163 702 V 17 w(Datatype)f(datatype,)h(int)g(dest,)393 758 y(int)h(tag,)f(MPI)p 683 758 V 17 w(Comm)g(comm,)g(MPI)p 1034 758 V 17 w(Comm)p 1147 758 V 17 w(request)g(*request\))75 845 y(MPI)p 150 845 V 17 w(CREATE)p 311 845 V 16 w(SSEND\(BUF,)g(COUNT,)g(DATATYPE,)f(DEST,)h (TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 901 y()g(BUF\(*\))170 958 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f (IERROR)166 1044 y Fy(The)15 b(function)h Ft(MPI)p 522 1044 14 2 v 15 w(CREA)l(TE)p 706 1044 V 18 w(RECV)g Fy(creates)e(a)h(comm)o (unication)g(ob)s(ject)f(for)h(a)f(receiv)o(e)i(op)q(era-)75 1100 y(tion.)75 1204 y Ft(MPI)p 160 1204 V 16 w(CREA)l(TE)p 345 1204 V 17 w(RECV\(buf,)g(count,)h(datat)o(yp)q(e,)f(source,)f(tag,)g (comm)m(,)d(request\))117 1281 y Fs(OUT)108 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 1356 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (receiv)o(ed)j(\(in)o(teger\))117 1432 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 1507 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(source)j(or)d(MPI)p 1309 1507 13 2 v 16 w(ANY)p 1418 1507 V 15 w(SOUR)o(CE)h(\(in)o(teger\))117 1582 y(IN)155 b Ft(tag)526 b Fs(message)14 b(tag)f(or)h(MPI)p 1270 1582 V 15 w(ANY)p 1378 1582 V 16 w(T)m(A)o(G)f(\(in)o(teger\))117 1657 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 1732 y(OUT)108 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k (\(handle\))75 1857 y Fr(int)23 b(MPI)p 245 1857 15 2 v 17 w(Create)p 406 1857 V 17 w(recv\(void*)f(buf,)h(int)h(count,)f(MPI)p 1139 1857 V 17 w(Datatype)f(datatype,)h(int)h(source,)393 1913 y(int)g(tag,)f(MPI)p 683 1913 V 17 w(Comm)g(comm,)g(MPI)p 1034 1913 V 17 w(Comm)p 1147 1913 V 17 w(request)g(*request\))75 1999 y(MPI)p 150 1999 V 17 w(CREATE)p 311 1999 V 16 w(RECV\(BUF,)g(COUNT,)g (DATATYPE,)f(SOURCE,)h(TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 2056 y()g(BUF\(*\))170 2112 y(INTEGER)g(COUNT,)g(DATATYPE,)g(SOURCE,)g (TAG,)g(COMM,)g(REQUEST,)g(IERROR)166 2199 y Fy(A)18 b(comm)o(unication)g (\(send)g(or)g(receiv)o(e\))g(that)f(uses)h(a)g(prede\014ned)h(request)f(is)h (initiated)g(b)o(y)f(the)75 2255 y(function)e Ft(MPI)p 338 2255 14 2 v 16 w(ST)l(ART)p Fy(.)75 2359 y Ft(MPI)p 160 2359 V 16 w(ST)l(ART\(request\))117 2436 y Fs(INOUT)62 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 2561 y Fr(int)23 b(MPI)p 245 2561 15 2 v 17 w(Start\(MPI)p 478 2561 V 16 w(Comm)p 590 2561 V 17 w(request)g(*request\))75 2647 y(MPI)p 150 2647 V 17 w(START\(REQUEST,)f(IERROR\))170 2704 y(INTEGER)h(REQUEST,)g(IERROR)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 36 36 bop 75 -100 a Fy(36)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fy(The)11 b(comm)o(unication)h(ob)s (ject)f(asso)q(ciated)g(with)h Ft(request)h Fy(should)f(b)q(e)g(an)f(ob)s (ject)g(that)g(w)o(as)f(created)75 106 y(b)o(y)k(one)g(of)g(the)g(previous)h (four)e(functions,)i(so)f(that)f(all)i(the)f(comm)o(unication)h(argumen)o(ts) e(are)h(already)75 162 y(de\014ned.)23 b(A)16 b(send)g(can)g(b)q(e)h(started) e(pro)o(vided)h(that)f(the)h(previous)h(send)f(using)h(the)e(same)h(ob)s (ject)f(has)75 219 y(completed,)d(or)f(as)f(so)q(on)h(as)f(the)h(ob)s(ject)f (is)i(created,)f(if)g(it)g(has)g(not)g(y)o(et)f(b)q(een)i(used)f(in)h(a)f (comm)o(unication.)75 275 y(In)k(addition,)h(if)f(the)f(comm)o(unication)i (mo)q(de)e(is)h Ft(ready)g Fy(then)g(a)f(matc)o(hing)h(receiv)o(e)g(should)h (b)q(e)f(p)q(osted.)75 332 y(The)j(send)h(bu\013er)f(should)g(not)g(b)q(e)h (accessed)f(after)f(the)h(send)h(is)f(started,)g(and)g(un)o(til)h(the)f(op)q (eration)75 388 y(completes.)166 446 y(A)c(receiv)o(e)g(can)g(b)q(e)g (started)f(pro)o(vided)i(that)e(the)h(preceding)h(receiv)o(e)f(using)h(the)f (same)f(ob)s(ject)g(has)75 502 y(completed,)f(or)f(as)f(so)q(on)h(as)f(the)h (ob)s(ject)f(is)i(created,)f(if)g(it)g(has)g(not)g(y)o(et)f(b)q(een)i(used)f (in)h(a)f(comm)o(unication.)75 559 y(The)18 b(receiv)o(e)h(bu\013er)f(should) h(not)e(b)q(e)i(accessed)g(after)e(the)h(receiv)o(e)h(is)f(started,)g(un)o (til)h(the)f(op)q(eration)75 615 y(completes.)166 673 y(The)23 b(call)g(is)g(non)o(blo)q(c)o(king,)i(with)e(similar)h(seman)o(tics)e(to)g (the)g(non)o(blo)q(c)o(king)i(comm)o(unication)75 729 y(op)q(erations)e (describ)q(ed)i(in)e(section)g(1.8.)39 b(I.e.,)23 b(a)e(call)i(to)e Ft(MPI)p 1218 729 14 2 v 16 w(ST)l(ART)h Fy(starts)f(a)g(comm)o(unication)75 786 y(in)g(the)e(same)h(manner)g(as)f(a)g(call)i(to)e Ft(MPI)p 838 786 V 16 w(ISEND)p Fy(,)h Ft(MPI)p 1099 786 V 15 w(IRSEND)p Fy(,)g Ft(MPI)p 1388 786 V 16 w(ISSEND)g Fy(or)f Ft(MPI)p 1721 786 V 16 w(IRECV)p Fy(,)75 842 y(resp)q(ectiv)o(ely)l(.)166 983 y Fl(Discussion:)34 b Fs(It)14 b(has)g(b)q(een)h(suggested)g(that)f Fo(MPI)p 1005 983 13 2 v 14 w(IST)m(ART)f Fs(w)o(ould)g(mak)o(e)g(this)h(p)q (oin)o(t)f(b)q(etter)75 1170 y Ft(MPI)p 160 1170 14 2 v 16 w(ST)l(ART)l(ALL\(count,)j(a)o(rra)o(y)p 641 1170 V 15 w(of)p 693 1170 V 16 w(requests\))117 1249 y Fs(IN)155 b Ft(count)482 b Fs(list)14 b(length)f(\(in)o(teger\))117 1327 y(INOUT)62 b Ft(a)o(rra)o(y)p 416 1327 V 15 w(of)p 468 1327 V 16 w(requests)272 b Fs(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f(handle\))75 1453 y Fr(int)23 b(MPI)p 245 1453 15 2 v 17 w(Startall\(int)f(count,)h(MPI)p 811 1453 V 17 w(Comm)p 924 1453 V 17 w(request)g(*array)p 1276 1453 V 16 w(of)p 1340 1453 V 17 w(requests\))75 1540 y(MPI)p 150 1540 V 17 w(STARTALL\(COUNT,)f(ARRAY)p 669 1540 V 16 w(OF)p 733 1540 V 17 w(REQUESTS,)h(IERROR\))170 1597 y(INTEGER)g(COUNT,)g(ARRAY)p 651 1597 V 17 w(OF)p 716 1597 V 17 w(REQUESTS\(*\),)f(IERROR)166 1685 y Fy(Start)i(all)j(comm)o(unications)f(asso)q(ciated)g(with)f(requests)h (in)g Ft(a)o(rra)o(y)p 1399 1685 14 2 v 15 w(of)p 1451 1685 V 16 w(requests)p Fy(.)52 b(A)26 b(call)h(to)75 1741 y Ft(MPI)p 160 1741 V 16 w(ST)l(ART)l(ALL\(count,)76 b(a)o(rra)o(y)p 701 1741 V 14 w(of)p 752 1741 V 16 w(requests\))65 b Fy(has)d(the)h(same)f (e\013ect)h(as)f(calls)i(to)75 1798 y Ft(MPI)p 160 1798 V 16 w(ST)l(ART\(a)o(rra)o(y)p 431 1798 V 15 w(of)p 483 1798 V 16 w(requests[i]\))p Fy(,)17 b(executed)f(for)e Ft(i=0)i(,...,)d(count-1)p Fy(,)j(in)g(some)f(arbitrary)f(order.)166 1855 y(A)j(comm)o(unication)g (started)f(with)h(a)g(call)h(to)e Ft(MPI)p 1063 1855 V 16 w(ST)l(ART)h Fy(or)f Ft(MPI)p 1377 1855 V 16 w(ST)l(ART)l(ALL)i Fy(is)f(completed)75 1912 y(b)o(y)11 b(a)h(call)g(to)f Ft(MPI)p 385 1912 V 16 w(W)l(AIT)p Fy(,)g Ft(MPI)p 620 1912 V 16 w(TEST)p Fy(,)h(or)f(one)g(of)g(the)h(deriv)o (ed)h(functions)f(describ)q(ed)h(in)g(section)f(1.8.5.)75 1968 y(These)k(comm)o(unication)h(completion)g(functions)g(do)f(not)f(deallo)q (cate)j(the)e(request,)g(and)g(this)g(can)g(b)q(e)75 2025 y(reused)g(anew)g (b)o(y)f(a)h Ft(MPI)p 521 2025 V 15 w(ST)l(ART)h Fy(call.)22 b(The)16 b(ob)s(ject)f(needs)h(to)f(b)q(e)i(explicitly)h(deallo)q(cated)f(b)o (y)f(a)f(call)75 2081 y(to)j(the)g(function)i Ft(MPI)p 482 2081 V 16 w(REQUEST)p 701 2081 V 17 w(FREE)p Fy(,)f(b)q(elo)o(w.)30 b(W)l(e)19 b(th)o(us)f(ha)o(v)o(e)g(t)o(w)o(o)g(t)o(yp)q(es)g(of)g(comm)o (unication)75 2138 y(requests:)34 b Fx(p)q(ersisten)o(t)23 b Fy(requests,)h(whic)o(h)f(are)f(allo)q(cated)i(b)o(y)e(a)g(call)i(to)e Ft(MPI)p 1503 2138 V 15 w(CREA)l(TE)p 1687 2138 V 18 w(xxx)p Fy(,)i(and)75 2194 y(are)18 b(explicitly)j(deallo)q(cated)e(b)o(y)f Ft(MPI)p 746 2194 V 16 w(REQUEST)p 965 2194 V 18 w(FREE)p Fy(,)g(and)g Fx(ephemeral)g Fy(requests)g(that)f(p)q(ersist)75 2251 y(for)f(one)i(comm)o (unication)g(only:)24 b(they)17 b(are)g(created)g(b)o(y)g(a)g(non)o(blo)q(c)o (king)h(comm)o(unication)g(initiation)75 2307 y(function,)e(and)f(are)g (freed)g(b)o(y)h(the)f(comm)o(unication)h(completion)g(call.)166 2365 y(If)40 b(a)20 b(call)h(to)e Ft(MPI)p 512 2365 V 16 w(W)l(AIT)p Fy(,)g Ft(MPI)p 755 2365 V 16 w(TEST)p Fy(,)h(or)f(an)o(y)h(of)f(the)h(deriv) o(ed)h(functions,)h(is)e(executed)h(on)-1903 b Fm(\))75 2421 y Fy(a)22 b(p)q(ersisten)o(t)h(comm)o(unication)g(request)f(for)g(whic)o(h)h (there)g(is)g(no)f(p)q(ending)i(comm)o(unication)f(\(e.g.)75 2478 y(immediately)13 b(after)e(it)h(w)o(as)f(created)h(or)f(immediately)i (after)e(a)h(call)g(to)f Ft(MPI)p 1378 2478 V 16 w(W)l(AIT)p Fy(\))h(then)g(the)f(outcome)75 2534 y(is)16 b(as)f(if)h(a)f(n)o(ull)i (request)e(w)o(as)g(passed)g(as)g(argumen)o(t:)20 b Ft(MPI)p 1100 2534 V 16 w(W)l(AIT)15 b Fy(returns)g(immediately)i(and)f(has)f(no)75 2591 y(e\013ect,)d Ft(MPI)p 290 2591 V 16 w(TEST)g Fy(returns)g(with)h Fo(\015ag)d(=)i(false)p Fy(,)g(etc.)19 b(In)12 b(particular,)h(one)g(can)f (pass)g(to)f Ft(MPI)p 1660 2591 V 16 w(W)l(AIT)l(ANY)75 2647 y Fy(and)20 b(the)h(other)e(functions)i(in)g(section)g(1.8.5)d(an)i Ft(a)o(rra)o(y)p 1070 2647 V 15 w(of)p 1122 2647 V 16 w(request)i Fy(argumen)o(t)d(where)h(some)g(of)g(the)75 2704 y(requests)10 b(are)h(p)q(ersisten)o(t)g(ones,)g(and)f(ha)o(v)o(e)g(the)h(desired)h (outcome:)17 b(the)10 b(call)i(will)g(ignore)f(those)f(requests)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 37 37 bop 75 -100 a Fu(1.11.)34 b(SEND-RECEIVE)1277 b Fy(37)75 49 y(in)13 b(the)g(arra)o(y)e(for)h(whic)o(h)i(there)e(are)g(no)h(p)q(ending)h (comm)o(unications)f(\(in)g(the)g(same)f(manner)h(that)f(these)75 106 y(functions)18 b(ignore)g(n)o(ull)h(request)f(en)o(tries\).)27 b(Th)o(us,)18 b(the)f(last)h(example)g(on)g(page)f(27)g(w)o(orks)g(correctly) 75 162 y(with)f(p)q(ersisten)o(t)f(requests,)g(to)q(o.)75 282 y Ft(MPI)p 160 282 14 2 v 16 w(REQUEST)p 379 282 V 18 w(FREE\(request\))117 375 y Fs(INOUT)62 b Ft(request)452 b Fs(comm)o(unication)11 b(request)k(\(handle\))75 515 y Fr(int)23 b(MPI)p 245 515 15 2 v 17 w(Request)p 430 515 V 16 w(free\(MPI)p 638 515 V 17 w(Comm)p 751 515 V 16 w(request)g(*request\))75 617 y(MPI)p 150 617 V 17 w(REQUEST)p 335 617 V 16 w(FREE\(REQUEST,)f(IERROR\))170 674 y(INTEGER)h(REQUEST,)g(IERROR)166 776 y Fy(Mark)c Ft(request)i Fy(for)e(deallo)q(cation.)35 b(The)20 b(request)f(ob)s(ject)h(is)g(deallo)q (cated)h(when)f(there)g(are)f(no)75 832 y(p)q(ending)e(comm)o(unications)f (in)o(v)o(olving)g(this)g(ob)s(ject,)e(at)h(whic)o(h)h(p)q(oin)o(t)f(the)h (request)f(b)q(ecomes)h(n)o(ull.)166 904 y(The)j(call)g(is)g(non)o(blo)q(c)o (king.)31 b(It)18 b(is)h(p)q(ermissible)i(to)d(call)h Ft(MPI)p 1249 904 14 2 v 16 w(REQUEST)p 1468 904 V 18 w(FREE\(request\))h Fy(after)75 961 y(a)f(comm)o(unication)g(that)f(uses)h Ft(request)i Fy(has)e(b)q(een)h(initiated,)h(but)e(b)q(efore)g(it)g(has)g(completed.)32 b(The)75 1017 y(ob)s(ject)15 b(will)j(b)q(e)f(deallo)q(cated)g(after)e(the)h (comm)o(unication)h(completes.)22 b(It)16 b(is)h(erroneous)f(to)f(initiate)i (a)75 1074 y(comm)o(unication)f(with)f(a)g(request)h(that)e(has)h(b)q(een)h (freed.)166 1146 y(A)f(correct)f(in)o(v)o(o)q(cation)i(of)e(the)h(functions)h (describ)q(ed)g(in)g(this)f(section)h(will)g(o)q(ccur)g(in)f(a)g(sequence)75 1202 y(of)g(the)g(form)166 1275 y Fx(Create)i Fy(\()p Fx(Start)h(Complete)p Fy(\))725 1258 y Fh(\003)761 1275 y Fx(F)l(ree)f Fi(;)166 1347 y Fy(where)f Fn(\003)g Fy(indicates)i(zero)e(or)f(more)h(rep)q(etitions.)24 b(If)16 b(the)g(same)g(comm)o(unication)h(ob)s(ject)f(is)g(used)75 1403 y(in)k(sev)o(eral)f(concurren)o(t)g(threads,)g(it)g(is)h(the)f(user)g (resp)q(onsibilit)o(y)i(to)e(co)q(ordinate)g(calls)h(so)f(that)f(the)75 1460 y(correct)d(sequence)h(is)g(ob)q(ey)o(ed.)166 1532 y(A)c(send)g(op)q (eration)f(initiated)j(with)d Ft(MPI)p 874 1532 V 16 w(ST)l(ART)i Fy(can)e(b)q(e)i(matc)o(hed)e(with)h(an)o(y)f(receiv)o(e)i(op)q(eration)75 1588 y(and,)g(lik)o(ewise,)h(a)e(receiv)o(e)i(op)q(eration)e(initiated)i (with)f Ft(MPI)p 1098 1588 V 16 w(ST)l(ART)g Fy(can)g(receiv)o(e)g(messages)f (generated)75 1645 y(b)o(y)j(an)o(y)g(send)h(op)q(eration.)75 1878 y Fz(1.11)60 b(Send-receive)75 2010 y Fy(The)18 b Fx(send-receiv)o(e)g Fy(op)q(erations)g(com)o(bine)h(in)g(one)f(call)h(the)f(sending)i(of)d(a)h (message)g(to)f(one)h(desti-)75 2067 y(nation)e(and)g(the)g(receiving)i(of)d (another)h(message,)f(from)g(another)h(destination,)g(p)q(ossibly)i(the)e (same.)75 2123 y(A)k(send-receiv)o(e)j(op)q(eration)d(is)h(v)o(ery)f(useful)i (for)d(executing)j(a)e(shift)h(op)q(eration)f(across)g(a)g(c)o(hain)h(of)75 2180 y(pro)q(cesses.)37 b(If)20 b(blo)q(c)o(king)i(sends)g(and)e(receiv)o(es) i(are)e(used)h(for)f(suc)o(h)h(shift,)h(then)f(one)g(needs)g(to)f(or-)75 2236 y(der)c(the)h(sends)f(and)g(receiv)o(es)h(correctly)g(\(e.g.,)d(ev)o(en) j(pro)q(cesses)f(send,)h(next)f(receiv)o(e,)h(o)q(dd)g(pro)q(cesses)75 2293 y(receiv)o(e)j(\014rst,)g(next)f(send\))g(so)g(as)g(to)g(prev)o(en)o(t)g (cyclic)i(dep)q(endencies)h(that)c(ma)o(y)h(lead)h(to)e(deadlo)q(c)o(k.)75 2349 y(When)g(a)g(send-receiv)o(e)h(op)q(eration)f(is)g(used,)h(the)f(comm)o (unication)g(subsystem)g(tak)o(es)f(care)g(of)h(these)75 2406 y(issues.)j(The)14 b(send-receiv)o(e)j(op)q(eration)e(can)f(b)q(e)i(used)f (in)g(conjunction)h(with)f(the)g(functions)g(describ)q(ed)75 2462 y(in)h(Chapter)f Fx(??)k Fy(in)d(order)f(to)f(p)q(erform)h(shifts)g(on)g (v)m(arious)h(logical)g(top)q(ologies.)k(Also,)c(a)e(send-receiv)o(e)75 2518 y(op)q(eration)h(is)h(useful)g(for)f(implemen)o(ting)i(remote)e(pro)q (cedure)h(calls.)166 2591 y(A)29 b(message)14 b(sen)o(t)h(b)o(y)f(a)h (send-receiv)o(e)h(op)q(eration)f(can)f(b)q(e)i(receiv)o(ed)g(b)o(y)e(a)g (regular)h(receiv)o(e)h(op)q(er-)69 b Fm(\()75 2647 y Fy(ation)16 b(or)f(prob)q(ed)i(b)o(y)f(a)g(prob)q(e)g(op)q(eration;)g(a)g(send-receiv)o (e)i(op)q(eration)e(can)g(receiv)o(e)h(a)f(message)f(sen)o(t)75 2704 y(b)o(y)g(a)g(regular)g(send)h(op)q(eration.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 38 38 bop 75 -100 a Fy(38)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(SENDRECV\(sendbuf,)24 b(sendcount,)f(sendt)o(yp)q(e,)g(dest,)f (sendtag,)g(recvbuf,)g(recvcount,)f(recvt)o(yp)q(e,)75 106 y(source,)15 b(recvtag,)g(comm)m(,)d(status\))117 183 y Fs(IN)155 b Ft(sendbuf)444 b Fs(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 254 y(IN)155 b Ft(sendcount)399 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o(teger\))117 325 y(IN)155 b Ft(sendt)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h (in)f(send)i(bu\013er)g(\(handle\))117 397 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 468 y(IN)155 b Ft(sendtag)443 b Fs(send)15 b(tag)e(\(in)o(teger\))117 539 y(OUT)108 b Ft(recvbuf)450 b Fs(initial)12 b(address)j(of)f(receiv)o(e)h (bu\013er)g(\(c)o(hoice\))117 610 y(IN)155 b Ft(recvcount)405 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g (\(in)o(teger\))117 682 y(IN)155 b Ft(recvt)o(yp)q(e)430 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(receiv)o(e)i(bu\013er)g (\(handle\))117 753 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j (\(in)o(teger\))117 824 y(IN)155 b Ft(recvtag)449 b Fs(receiv)o(e)15 b(tag)f(\(in)o(teger\))117 895 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 966 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 1091 y Fr(int)23 b(MPI)p 245 1091 15 2 v 17 w(Sendrecv\(void)f(*sendbuf,)h(int)g(sendcount,)g(MPI)p 1265 1091 V 17 w(Datatype)f(sendtype,)393 1147 y(int)i(dest,)f(int)g (sendtag,)g(void)g(*recvbuf,)g(int)g(recvcount,)393 1204 y(MPI)p 468 1204 V 17 w(Datatype)g(recvtype,)f(int)i(source,)f(MPI)p 1297 1204 V 16 w(Datatype)g(recvtag,)393 1260 y(MPI)p 468 1260 V 17 w(Comm)g(comm,)g(MPI)p 819 1260 V 17 w(Status)g(*status\))75 1347 y(MPI)p 150 1347 V 17 w(SENDRECV\(SENDBUF,)e(SENDCOUNT,)i(SENDTYPE,)f (DEST,)i(SENDTAG,)e(RECVBUF,)393 1403 y(RECVCOUNT,)h(RECVTYPE,)f(SOURCE,)h (RECVTAG,)g(COMM,)g(STATUS,)g(IERROR\))170 1460 y()g(SENDBUF\(*\),)g (RECVBUF\(*\))170 1516 y(INTEGER)g(SENDCOUNT,)g(SENDTYPE,)f(DEST,)i(SENDTAG,) e(RECVCOUNT,)h(RECVTYPE,)170 1573 y(SOURCE,)g(RECV)h(TAG,)f(COMM,)g (STATUS\(MPI)p 986 1573 V 16 w(STATUS)p 1146 1573 V 16 w(SIZE\),)g(IERROR)166 1659 y Fy(Execute)c(a)e(blo)q(c)o(king)j(send)f(and)f(receiv)o(e)h(op)q (eration.)29 b(Both)18 b(send)h(and)f(receiv)o(e)h(use)g(the)f(same)75 1715 y(comm)o(unicator,)c(but)g(p)q(ossibly)i(di\013eren)o(t)f(tags.)j(The)d (send)g(bu\013er)f(and)h(receiv)o(e)g(bu\013ers)g(are)f(disjoin)o(t,)75 1772 y(and)h(ma)o(y)g(ha)o(v)o(e)g(di\013eren)o(t)g(lengths)h(and)f(datat)o (yp)q(es.)75 1876 y Ft(MPI)p 160 1876 14 2 v 16 w(SENDRECV)p 408 1876 V 17 w(REPLA)o(CE\(buf,)k(count,)f(datat)o(yp)q(e,)h(dest,)f (sendtag,)h(source,)f(recvtag,)f(comm)m(,)d(sta-)75 1932 y(tus\))117 2009 y Fs(INOUT)62 b Ft(buf)527 b Fs(initial)12 b(address)j(of)f(send)g(and)g (receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 2081 y(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)9 b(of)g(elemen)o(ts)h(in)f(send)i(and)e (receiv)o(e)i(bu\013er)g(\(in)o(teger\))117 2152 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(send)i (and)f(receiv)o(e)h(bu\013er)g(\(handle\))117 2223 y(IN)155 b Ft(dest)511 b Fs(rank)14 b(of)f(destination)h(\(in)o(teger\))117 2294 y(IN)155 b Ft(sendtag)443 b Fs(send)15 b(message)f(tag)f(\(in)o(teger\)) 117 2365 y(IN)155 b Ft(source)468 b Fs(rank)14 b(of)f(source)j(\(in)o (teger\))117 2437 y(IN)155 b Ft(recvtag)449 b Fs(receiv)o(e)15 b(message)f(tag)f(\(in)o(teger\))117 2508 y(IN)155 b Ft(comm)466 b Fs(comm)o(unicator)11 b(\(handle\))117 2579 y(OUT)108 b Ft(status)476 b Fs(status)15 b(ob)r(ject)g(\(Status\))75 2704 y Fr(int)23 b(MPI)p 245 2704 15 2 v 17 w(Sendrecv)p 454 2704 V 16 w(replace\(void*)f (buf,)i(int)f(count,)g(MPI)p 1258 2704 V 17 w(Datatype)g(datatype,)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 39 39 bop 75 -100 a Fu(1.12.)34 b(NULL)16 b(PR)o(OCESSES)1218 b Fy(39)393 49 y Fr(int)24 b(dest,)f(int)g(sendtag,)g(int)g(source,)g(int)h(recvtag,)f (MPI)p 1614 49 15 2 v 16 w(Comm)h(comm,)393 106 y(MPI)p 468 106 V 17 w(Status)f(*status\))75 192 y(MPI)p 150 192 V 17 w(SENDRECV)p 359 192 V 16 w(REPLACE\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(SENDTAG,)g (SOURCE,)g(RECVTAG,)393 249 y(COMM,)g(STATUS,)g(IERROR\))170 305 y()g(BUF\(*\))170 362 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (SENDTAG,)g(SOURCE,)g(RECVTAG,)g(COMM,)170 418 y(STATUS\(MPI)p 413 418 V 16 w(STATUS)p 573 418 V 17 w(SIZE\),)g(IERROR)166 504 y Fy(Execute)15 b(a)f(blo)q(c)o(king)i(send)f(and)g(receiv)o(e;)g(the)g (same)f(bu\013er)g(is)h(used)h(b)q(oth)e(for)g(the)h(send)g(and)g(for)75 561 y(the)g(receiv)o(e,)h(so)f(that)f(the)i(message)e(sen)o(t)h(is)h (replaced)g(b)o(y)g(the)f(message)g(receiv)o(ed.)166 617 y(The)f(seman)o (tics)g(of)f(a)h(send-receiv)o(e)h(op)q(eration)f(is)g(what)g(w)o(ould)g (obtain)g(if)g(the)g(caller)h(fork)o(ed)e(t)o(w)o(o)75 674 y(concurren)o(t)19 b(threads,)f(one)h(to)f(execute)h(the)g(send,)g(and)g(one) g(to)e(execute)j(the)e(receiv)o(e,)i(follo)o(w)o(ed)f(b)o(y)75 730 y(a)e(join)g(of)g(these)g(t)o(w)o(o)f(threads.)25 b(Additional)19 b(in)o(termediate)f(bu\013ering)g(is)f(needed)i(for)d(the)i(\\replace")75 787 y(v)m(arian)o(t.)166 843 y(A)11 b(send-receiv)o(e)i(cannot)f(b)q(e)g (implemen)o(ted)h(b)o(y)e(a)g(blo)q(c)o(king)i(send)f(follo)o(w)o(ed)g(b)o(y) f(a)g(blo)q(c)o(king)i(receiv)o(e)75 900 y(or)i(a)g(blo)q(c)o(king)i(receiv)o (e,)f(follo)o(w)o(ed)g(b)o(y)g(a)f(blo)q(c)o(king)i(send.)22 b(Consider)16 b(the)f(follo)o(wing)i(co)q(de,)e(where)h(t)o(w)o(o)75 956 y(pro)q(cesses)g(exc)o(hange)f(messages:)75 1050 y Fr(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1107 y(IF)h(\(rank.EQ.0\))123 1163 y(THEN)170 1220 y(CALL)g(MPI_SENDRECV\(send_buff,)c(send_count,)j (MPI_REAL,)f(1,)i(send_tag,)242 1276 y(recv_buff,)f(recv_count,)f(MPI_REAL,)h (1,)g(recv_tag,)g(comm,)g(status,)g(ierr\))123 1333 y(ELSE)170 1389 y(CALL)h(MPI_SENDRECV\(send_buff,)c(send_count,)j(MPI_REAL,)f(0,)i (send_tag,)242 1446 y(recv_buff,)f(recv_count,)f(MPI_REAL,)h(0,)g(tag,)g (comm,)h(status,)f(ierr\))75 1502 y(END)g(IF)166 1596 y Fy(If)16 b(the)f(send)h(receiv)o(es)h(are)e(replaced)i(either)f(b)o(y)f(blo)q(c)o (king)i(send,)f(follo)o(w)o(ed)g(b)o(y)g(blo)q(c)o(king)g(receiv)o(e,)75 1653 y(or)g(blo)q(c)o(king)h(receiv)o(e,)g(follo)o(w)o(ed)g(b)o(y)f(blo)q(c)o (king)i(send,)f(then)f(the)h(co)q(de)g(ma)o(y)e(deadlo)q(c)o(k.)24 b(On)17 b(the)f(other)75 1709 y(hand,)f(send-receiv)o(e)i(can)f(b)q(e)f (implemen)o(ted)i(using)f(non)o(blo)q(c)o(king)h(sends)e(and)h(receiv)o(es.) 75 1851 y Fz(1.12)60 b(Null)19 b(p)n(ro)r(cesses)75 1952 y Fy(In)c(man)o(y)e(instances,)i(it)f(is)h(con)o(v)o(enien)o(t)g(to)e(sp)q (ecify)j(a)d(\\dumm)o(y")h(source)g(or)g(destination)h(for)e(comm)o(u-)75 2009 y(nication.)29 b(This)18 b(simpli\014es)i(the)e(co)q(de)g(that)f(is)h (needed)i(for)d(dealing)i(with)f(b)q(oundaries,)h(e.g.,)e(in)i(the)75 2065 y(case)c(of)g(a)g(non-circular)i(shift)e(done)h(with)f(calls)i(to)d (send-receiv)o(e.)166 2122 y(The)f(sp)q(ecial)j(v)m(alue)26 b Fo(MPI)p 608 2122 13 2 v 15 w(PROCNULL)12 b Fy(can)i(b)q(e)g(used)g (instead)g(of)f(a)g(rank)g(wherev)o(er)g(a)g(source)g(or)g(a)75 2178 y(destination)j(argumen)o(t)f(is)g(required)i(in)f(a)f(call.)21 b(A)15 b(comm)o(unication)h(with)g(pro)q(cess)29 b Fo(MPI)p 1649 2178 V 15 w(PROCNULL)75 2235 y Fy(has)13 b(no)h(e\013ect:)k(a)13 b(send)h(to)25 b Fo(MPI)p 633 2235 V 14 w(PROCNULL)13 b Fy(succeeds)i(and)e (returns)g(as)g(so)q(on)h(as)f(p)q(ossible.)21 b(A)13 b(receiv)o(e)75 2291 y(from)26 b Fo(MPI)p 271 2291 V 14 w(PROCNULL)13 b Fy(succeeds)i(and)f (returns)g(as)f(so)q(on)h(as)f(p)q(ossible)j(with)e(no)g(mo)q(di\014cations)h (to)e(the)75 2348 y(receiv)o(e)18 b(bu\013er.)25 b(When)17 b(a)g(receiv)o(e)h(with)f Ft(source)h Fy(=)32 b Fo(MPI)p 1090 2348 V 15 w(PROCNULL)16 b Fy(is)i(executed)f(then)h(the)f(status)75 2404 y(ob)s(ject)e(returns)30 b Ft(source)16 b Fy(=)29 b Fo(MPI)p 661 2404 V 14 w(PROCNULL)p Fy(,)14 b Ft(tag)h Fy(=)30 b Fo(MPI)p 1132 2404 V 14 w(ANY)p 1231 2404 V 15 w(T)m(A)o(G)14 b Fy(and)h Ft(count=0)p Fy(.)75 2546 y Fz(1.13)60 b(Derived)19 b(datat)n(yp)r(es)75 2647 y Fy(Up)c(to)f(no)o(w,)g(all)i(p)q(oin)o(t)f(to)f(p)q(oin)o(t)h(comm)o (unication)h(in)o(v)o(olv)o(ed)f(only)g(con)o(tiguous)g(bu\013ers)g(con)o (taining)g(a)75 2704 y(sequence)g(of)e(elemen)o(ts)h(of)f(the)g(same)g(t)o (yp)q(e.)19 b(This)14 b(is)g(to)q(o)f(constraining)h(on)f(t)o(w)o(o)f(accoun) o(ts:)19 b(One)14 b(often)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 40 40 bop 75 -100 a Fy(40)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fy(w)o(an)o(ts)d(to)h(pass)g (messages)g(that)f(con)o(tain)i(v)m(alues)g(with)g(di\013eren)o(t)f(datat)o (yp)q(es)g(\(e.g.,)f(an)h(in)o(teger)h(coun)o(t,)75 106 y(follo)o(w)o(ed)i(b) o(y)f(a)g(sequence)i(of)e(real)h(n)o(um)o(b)q(ers\);)f(and)h(one)g(often)f(w) o(an)o(ts)f(to)h(send)h(noncon)o(tiguous)g(data)75 162 y(\(e.g.)h(a)12 b(sub-blo)q(c)o(k)i(of)e(a)g(matrix\).)18 b(One)13 b(solution)g(is)g(to)f (pro)o(vide)h(functions)g(that)e(pac)o(k)i(noncon)o(tiguous)75 219 y(data)f(in)o(to)h(a)g(con)o(tiguous)g(bu\013er)g(at)f(the)h(sender)h (site)f(and)g(unpac)o(k)h(it)f(bac)o(k)g(at)f(the)h(receiv)o(er)h(site.)19 b(This)75 275 y(has)f(the)g(disadv)m(an)o(tage)g(of)g(requiring)h(additional) h(memory-to-memory)c(cop)o(y)i(op)q(erations)g(at)g(b)q(oth)75 332 y(sites,)12 b(ev)o(en)f(when)g(the)g(comm)o(unication)g(subsystem)g(has)f (scatter-gather)g(capabilities.)20 b(Instead,)12 b(MPI)75 388 y(pro)o(vides)22 b(mec)o(hanisms)f(to)g(sp)q(ecify)h(more)f(general,)i(mixed) f(and)f(noncon)o(tiguous)g(comm)o(unication)75 444 y(bu\013ers.)29 b(It)19 b(is)g(up)g(to)f(the)g(implemen)o(tation)i(to)e(decide)i(whether)e (data)g(should)h(b)q(e)h(\014rst)e(pac)o(k)o(ed)g(in)75 501 y(a)e(con)o(tiguous)h(bu\013er)g(b)q(efore)g(b)q(eing)h(transmitted,)f(or)f (whether)h(it)g(can)g(b)q(e)g(collected)i(directly)f(from)75 557 y(where)d(it)h(resides.)166 614 y(The)e(general)h(mec)o(hanisms)g(pro)o (vided)g(here)f(allo)o(w)h(one)f(to)g(transfer)f(directly)l(,)j(without)e (cop)o(ying,)75 670 y(ob)s(jects)j(of)g(v)m(arious)h(shap)q(e)h(and)f(size.) 28 b(It)17 b(is)i(not)e(assumed)h(that)f(the)g(MPI)h(library)g(is)h(cognizan) o(t)e(of)75 727 y(the)f(ob)s(jects)g(declared)h(in)g(the)f(host)g(language;)g (th)o(us,)g(if)g(one)g(w)o(an)o(ts)f(to)h(transfer)f(a)h(structure,)f(or)h (an)75 783 y(arra)o(y)d(section,)i(it)f(will)i(b)q(e)f(necessary)f(to)g(pro)o (vide)h(in)g(MPI)f(a)g(de\014nition)i(of)e(a)g(comm)o(unication)h(bu\013er)75 840 y(that)e(mimics)j(the)e(de\014nition)i(of)e(the)g(structure)g(or)f(arra)o (y)g(section)i(in)g(question.)20 b(These)15 b(facilities)h(can)75 896 y(b)q(e)j(used)g(b)o(y)f(library)h(designers)h(to)d(de\014ne)j(comm)o (unication)f(functions)g(that)f(can)g(transfer)g(ob)s(jects)75 953 y(de\014ned)g(in)f(the)f(host)g(language)g({)g(b)o(y)h(deco)q(ding)g (their)g(de\014nitions)h(as)e(a)o(v)m(ailable)i(in)f(a)f(sym)o(b)q(ol)h (table)75 1009 y(or)e(a)g(dop)q(e)g(v)o(ector.)20 b(Suc)o(h)15 b(higher-lev)o(el)j(comm)o(unication)e(functions)g(are)f(not)g(part)f(of)h (MPI.)166 1065 y(More)d(general)g(comm)o(unication)i(bu\013ers)e(are)g(sp)q (eci\014ed)i(b)o(y)f(replacing)g(the)g(basic)g(datat)o(yp)q(es)f(that)75 1122 y(ha)o(v)o(e)e(b)q(een)i(used)f(so)f(far)g(with)h(deriv)o(ed)g(datat)o (yp)q(es)f(that)g(are)g(constructed)g(from)g(basic)h(datat)o(yp)q(es)f(using) 75 1178 y(the)j(constructors)e(describ)q(ed)j(in)g(this)f(section.)19 b(These)13 b(metho)q(ds)f(of)g(constructing)h(deriv)o(ed)h(datat)o(yp)q(es)75 1235 y(can)h(b)q(e)h(applied)h(recursiv)o(ely)l(.)166 1291 y(A)e Fx(general)j(datat)o(yp)q(e)e Fy(is)g(an)f(opaque)h(ob)s(ject)e(that)h (sp)q(eci\014es)i(t)o(w)o(o)d(things:)143 1384 y Fn(\017)23 b Fy(A)15 b(sequence)h(of)f(basic)h(datat)o(yp)q(es)143 1478 y Fn(\017)23 b Fy(A)15 b(sequence)h(of)f(in)o(teger)g(\(b)o(yte\))g (displacemen)o(ts)166 1571 y(The)22 b(displacemen)o(ts)g(are)g(not)f (required)h(to)f(b)q(e)h(p)q(ositiv)o(e,)i(distinct,)g(or)d(in)h(increasing)h (order;)75 1628 y(therefore)17 b(the)g(order)g(of)f(items)i(need)g(not)f (coincide)i(with)e(their)h(order)f(in)g(store,)g(and)g(an)g(item)h(ma)o(y)75 1684 y(app)q(ear)c(more)g(than)g(once.)20 b(W)l(e)14 b(call)h(suc)o(h)g(pair) f(of)g(sequences)h(\(or)e(sequence)j(of)d(pairs\))h(a)g Fx(t)o(yp)q(e)i(map)p Fy(.)75 1740 y(The)22 b(sequence)g(of)f(basic)h(datat)o(yp)q(es)f (\(displacemen)o(ts)h(ignored\))g(is)f(the)h Fx(t)o(yp)q(e)i(signature)e Fy(of)f(the)75 1797 y(datat)o(yp)q(e.)166 1853 y(Let)476 1910 y Fi(T)6 b(y)r(pemap)12 b Fy(=)h Fn(f)p Fy(\()p Fi(ty)r(pe)849 1917 y Fj(0)867 1910 y Fi(;)8 b(disp)972 1917 y Fj(0)990 1910 y Fy(\))p Fi(;)g(:::;)f Fy(\()p Fi(ty)r(pe)1190 1917 y Fg(n)p Fh(\000)p Fj(1)1253 1910 y Fi(;)h(disp)1358 1917 y Fg(n)p Fh(\000)p Fj(1)1421 1910 y Fy(\))p Fn(g)p Fi(;)75 1993 y Fy(b)q(e)16 b(suc)o(h)f(a)g(t)o(yp)q(e)h(map,)e(where)i Fi(ty)r(pe)709 2000 y Fg(i)738 1993 y Fy(are)f(basic)h(t)o(yp)q(es,)f(and)g Fi(disp)1233 2000 y Fg(i)1261 1993 y Fy(are)g(displacemen)o(ts.)22 b(Let)676 2094 y Fi(T)6 b(y)r(pesig)14 b Fy(=)f Fn(f)p Fi(ty)r(pe)1005 2101 y Fj(0)1023 2094 y Fi(;)8 b(:::;)g(ty)r(pe)1188 2101 y Fg(n)p Fh(\000)p Fj(1)1251 2094 y Fn(g)75 2195 y Fy(b)q(e)14 b(the)f(asso)q(ciated)h(t)o(yp)q(e)f(signature.)19 b(This)14 b(t)o(yp)q(e)f(map,)g(together)g(with)g(a)g(base)g(address)h Fp(buf)p Fy(,)f(sp)q(eci\014es)75 2252 y(a)19 b(comm)o(unication)i(bu\013er:) 28 b(the)20 b(comm)o(unication)h(bu\013er)e(that)g(consists)h(of)f Fi(n)h Fy(en)o(tries,)h(where)f(the)75 2308 y Fi(i)p Fy(-th)e(en)o(try)g(is)h (at)e(address)h Fi(buf)f Fy(+)c Fi(disp)778 2315 y Fg(i)809 2308 y Fy(and)19 b(has)f(t)o(yp)q(e)g Fi(ty)r(pe)1174 2315 y Fg(i)1188 2308 y Fy(.)29 b(A)18 b(message)g(assem)o(bled)h(from)f(suc)o(h) 75 2365 y(comm)o(unication)e(bu\013er)f(will)i(consist)e(of)g Fi(n)h Fy(v)m(alues,)g(of)e(the)i(t)o(yp)q(es)f(de\014ned)h(b)o(y)g Fi(T)6 b(y)r(pesig)r Fy(.)166 2421 y(W)l(e)11 b(can)f(use)h(a)f(handle)i(to)e (a)g(general)h(datat)o(yp)q(e)f(as)h(an)f(argumen)o(t)g(in)h(a)f(send)i(or)e (receiv)o(e)h(op)q(eration,)75 2478 y(in)j(replacemen)o(t)f(of)f(a)h(basic)g (datat)o(yp)q(e)g(argumen)o(t.)18 b(The)13 b(op)q(eration)g Ft(MPI)p 1359 2478 14 2 v 16 w(SEND\(buf,)h(1,)e(datat)o(yp)q(e,...\))75 2534 y Fy(will)j(use)e(the)h(send)g(bu\013er)f(de\014ned)h(b)o(y)g(the)f (base)g(address)h Ft(buf)g Fy(and)f(the)g(general)h(datat)o(yp)q(e)f(asso)q (ciated)75 2591 y(with)f Ft(datat)o(yp)q(e)p Fy(;)i(it)e(will)i(generate)d(a) h(message)f(with)h(the)g(t)o(yp)q(e)g(signature)g(determined)h(b)o(y)e(the)h Ft(datat)o(yp)q(e)75 2647 y Fy(argumen)o(t.)28 b Ft(MPI)p 390 2647 V 16 w(RECV\(buf,)20 b(1,)e(datat)o(yp)q(e,...\))29 b Fy(will)20 b(use)e(the)h(receiv)o(e)g(bu\013er)f(de\014ned)h(b)o(y)g(the)f (base)75 2704 y(address)d Ft(buf)h Fy(and)g(the)f(general)h(datat)o(yp)q(e)e (asso)q(ciated)i(with)f Ft(datat)o(yp)q(e)p Fy(.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 41 41 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(41)166 49 y(General)16 b(datat)o(yp)q(es)e(can)i(b)q(e)g(used)f(in)h (all)h(send)e(and)h(receiv)o(e)g(op)q(erations.)k(W)l(e)c(address)f(later)g (in)75 106 y(section)h(1.13.5)d(the)i(case)h(where)f(the)g(second)h(argumen)o (t)e Ft(count)j Fy(has)e(v)m(alue)h Fi(>)d Fy(1.)166 162 y(The)e(basic)g (datat)o(yp)q(es)f(presen)o(ted)i(in)f(section)g(1.2.1)f(are)g(particular)h (cases)g(of)f(a)h(general)g(datat)o(yp)q(e,)75 219 y(and)19 b(are)f(prede\014ned.)32 b(Th)o(us,)19 b Ft(MPI)p 711 219 14 2 v 16 w(INT)f Fy(is)h(a)g(prede\014ned)h(handle)g(to)e(a)g(datat)o(yp)q(e)g (with)h(t)o(yp)q(e)g(map)75 275 y Fn(f)p Fy(\()p Ft(int)p Fi(;)8 b Fy(0\))p Fn(g)p Fy(,)j(with)i(one)g(en)o(try)f(of)g(t)o(yp)q(e)h Ft(int)g Fy(and)f(displacemen)o(t)i(zero.)19 b(And)13 b(similarly)i(for)d (all)h(other)f(basic)75 332 y(datat)o(yp)q(es.)166 388 y(The)j Fx(exten)o(t)f Fy(of)g(a)g(datat)o(yp)q(e)g(is)h(de\014ned)g(to)f(b)q(e)h (the)g(span)f(from)g(the)g(\014rst)g(b)o(yte)h(to)e(the)i(last)f(b)o(yte)75 444 y(o)q(ccupied)19 b(b)o(y)d(en)o(tries)h(in)h(this)f(datat)o(yp)q(e,)f (rounded)h(up)g(to)f(satisfy)h(alignmen)o(t)g(requiremen)o(ts.)25 b(That)75 501 y(is,)15 b(if)476 557 y Fi(T)6 b(y)r(pemap)12 b Fy(=)h Fn(f)p Fy(\()p Fi(ty)r(pe)849 564 y Fj(0)867 557 y Fi(;)8 b(disp)972 564 y Fj(0)990 557 y Fy(\))p Fi(;)g(:::;)f Fy(\()p Fi(ty)r(pe)1190 564 y Fg(n)p Fh(\000)p Fj(1)1253 557 y Fi(;)h(disp)1358 564 y Fg(n)p Fh(\000)p Fj(1)1421 557 y Fy(\))p Fn(g)p Fi(;)75 635 y Fy(then)501 726 y Fi(l)q(b)p Fy(\()p Fi(T)e(y)r(pemap)p Fy(\))39 b(=)j(min)906 752 y Fg(j)959 726 y Fi(disp)1043 733 y Fg(j)1060 726 y Fi(;)489 812 y(ub)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))40 b(=)i(max)911 839 y Fg(j)961 812 y Fy(\()p Fi(disp)1063 819 y Fg(j)1089 812 y Fy(+)11 b Fi(siz)r(eof)5 b Fy(\()p Fi(ty)r(pe)1367 819 y Fg(j)1385 812 y Fy(\)\))p Fi(;)22 b Fy(and)406 899 y Fi(extent)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))42 b(=)g Fi(ub)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))j Fn(\000)h Fi(l)q(b)p Fy(\()p Fi(T)c(y)r(pemap)p Fy(\))i(+)j Fi(\017:)238 b Fy(\(1.1\))75 990 y(If)17 b Fi(ty)r(pe)206 997 y Fg(i)237 990 y Fy(requires)g(alignmen)o(t) g(to)f(a)h(b)o(yte)f(address)h(that)f(is)h(is)g(a)f(m)o(ultiple)j(of)d Fi(k)1474 997 y Fg(i)1487 990 y Fy(,)h(then)g Fi(\017)g Fy(is)g(the)g(least) 75 1046 y(nonnegativ)o(e)f(incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))16 b(to)e(the)h(next)h(m)o (ultiple)h(of)d(max)1794 1053 y Fg(i)1815 1046 y Fi(k)1839 1053 y Fg(i)1853 1046 y Fy(.)166 1103 y(Example:)23 b(Assume)17 b(that)f Fi(T)6 b(y)r(pe)14 b Fy(=)h Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)15 b Fy(\(a)h Ft(double)h Fy(at)f(displacemen)o(t)i(zero,)75 1159 y(follo)o(w)o(ed)g(b)o(y)g(a)f Ft(cha)o(r)g Fy(at)g(displacemen)o(t)j (eigh)o(t\).)27 b(Assume,)18 b(furthermore,)f(that)g(doubles)i(ha)o(v)o(e)e (to)g(b)q(e)75 1216 y(strictly)e(aligned)g(at)f(addresses)h(that)e(are)h(m)o (ultiple)i(of)e(eigh)o(t.)20 b(Then,)15 b(the)f(exten)o(t)g(of)g(this)h (datat)o(yp)q(e)e(is)75 1272 y(16)d(\(9)g(rounded)h(to)f(the)h(next)g(m)o (ultiple)h(of)f(8\).)17 b(A)11 b(datat)o(yp)q(e)f(that)g(consists)h(of)f(a)g (c)o(haracter)g(immediately)75 1329 y(follo)o(w)o(ed)16 b(b)o(y)f(a)g(double) h(will)h(also)e(ha)o(v)o(e)g(an)g(exten)o(t)g(of)g(16.)189 1421 y Fp(R)n(ationale.)39 b Fy(The)15 b(de\014nition)h(of)e(exten)o(t)g(is)h (motiv)m(ated)f(b)o(y)h(the)f(assumption)h(that)e(the)i(amoun)o(t)189 1478 y(of)i(padding)i(added)g(at)e(the)h(end)g(of)g(eac)o(h)g(structure)f(in) i(an)f(arra)o(y)e(of)i(structures)f(is)i(the)f(least)189 1534 y(needed)d(to)e(ful\014ll)j(alignmen)o(t)e(constrain)o(ts.)19 b(More)14 b(explicit)i(con)o(trol)d(of)g(the)h(exten)o(t)g(is)g(pro)o(vided) 189 1591 y(in)i(section)g(1.13.3.)k(Suc)o(h)c(explicit)i(con)o(trol)e(is)g (needed)h(in)f(cases)g(where)g(the)g(assumption)g(do)q(es)189 1647 y(not)e(hold,)i(e.g.)j(where)d(union)g(t)o(yp)q(es)f(are)g(used.)21 b(\()p Fp(End)15 b(of)i(r)n(ationale.)p Fy(\))75 1767 y Fq(1.13.1)49 b(Datat)o(yp)q(e)17 b(constructo)o(rs)75 1853 y Ft(Contiguous)47 b Fy(The)16 b(simplest)i(datat)o(yp)q(e)e(constructor)f(is)i Ft(MPI)p 1171 1853 V 16 w(TYPE)p 1304 1853 V 17 w(CONTIGUOUS)h Fy(whic)o(h)f(allo)o(ws)75 1909 y(replication)g(of)e(a)g(datat)o(yp)q(e)f(in) o(to)h(con)o(tiguous)h(lo)q(cations.)75 2013 y Ft(MPI)p 160 2013 V 16 w(TYPE)p 293 2013 V 17 w(CONTIGUOUS\(count,)h(oldt)o(yp)q(e,)f (newt)o(yp)q(e\))117 2090 y Fs(IN)155 b Ft(count)482 b Fs(replication)14 b(coun)o(t)g(\(nonnegativ)o(e)g(in)o(teger\))117 2164 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2237 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2361 y Fr(int)23 b(MPI)p 245 2361 15 2 v 17 w(Type)p 358 2361 V 17 w(contiguous\(int)f(count,)h(MPI)p 972 2361 V 17 w(Datatype)f(oldtype,)393 2418 y(MPI)p 468 2418 V 17 w(Datatype)h(*newtype\))75 2504 y(MPI)p 150 2504 V 17 w(TYPE)p 263 2504 V 16 w(CONTIGUOUS\(COUNT,)f(OLDTYPE,)h(NEWTYPE,)f(IERROR\))170 2561 y(INTEGER)h(COUNT,)g(OLDTYPE,)g(NEWTYPE,)g(IERROR)166 2647 y Ft(newt)o(yp)q(e)c Fy(is)f(the)f(datat)o(yp)q(e)g(obtained)h(b)o(y)f (concatenating)h Ft(count)h Fy(copies)f(of)f Ft(oldt)o(yp)q(e)p Fy(.)27 b(Concate-)75 2704 y(nation)15 b(is)h(de\014ned)h(using)f Fp(extent)e Fy(as)h(the)h(size)g(of)e(the)i(concatenated)f(copies.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 42 42 bop 75 -100 a Fy(42)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fl(Discussion:)55 b Fs(Here,)15 b(and)f(in)g(the)h(other)f(datat)o(yp)q(e)h(constructors,)h (the)f(input)f(datat)o(yp)q(e)g(argumen)o(t)g(of)-1905 b Fk(\))75 106 y Fs(the)16 b(C)g(function)f(is)h(passed)h(b)o(y)e(v)n(alue,)g(and)h(the) g(output)g(datat)o(yp)q(e)g(argumen)o(t)f(is)h(passed)g(b)o(y)g(reference.)26 b(I.e.,)75 162 y(\(...,oldt)o(yp)q(e,)12 b(*newt)o(yp)q(e\).)18 b(It)c(has)g(b)q(een)h(argued)g(that)f(it)f(is)h(more)f(con)o(v)o(enien)o(t)h (to)g(pass)g(b)q(oth)g(via)f(a)h(p)q(oin)o(ter.)166 301 y Fy(Example:)31 b(let)21 b Ft(oldt)o(yp)q(e)h Fy(ha)o(v)o(e)e(t)o(yp)q(e)h(map)f Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)p Fi(;)18 b Fy(with)j(exten)o(t)g(16,)g (and)f(let)75 358 y Ft(count)14 b Fy(=)f(3.)19 b(The)d(t)o(yp)q(e)f(map)g(of) g(the)g(datat)o(yp)q(e)g(returned)g(b)o(y)h Ft(newt)o(yp)q(e)h Fy(is)308 458 y Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fi(;)g Fy(\()p Ft(double)n Fi(;)g Fy(16\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(24\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fy(32\))p Fi(;)g Fy(\()p Ft(cha)o(r)m Fi(;)g Fy(40\))p Fn(g)p Fy(;)75 558 y(i.e.,)15 b(alternating)g Ft(double)i Fy(and)e Ft(cha)o(r)g Fy(elemen)o(ts,)h(with)f (displacemen)o(ts)i(0)p Fi(;)8 b Fy(8)p Fi(;)g Fy(16)p Fi(;)f Fy(24)p Fi(;)g Fy(32)o Fi(;)h Fy(4)o(0.)166 615 y(In)16 b(general,)f(assume)g (that)g(the)g(t)o(yp)q(e)g(map)g(of)g Ft(oldt)o(yp)q(e)h Fy(is)600 715 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 722 y Fj(0)743 715 y Fi(;)8 b(disp)848 722 y Fj(0)865 715 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 722 y Fg(n)p Fh(\000)p Fj(1)1128 715 y Fi(;)g(disp)1233 722 y Fg(n)p Fh(\000)p Fj(1)1297 715 y Fy(\))p Fn(g)p Fi(;)75 815 y Fy(with)16 b(exten)o(t)e Fi(extent)p Fy(.)22 b(Then)15 b Ft(newt)o(yp)q(e)i Fy(has)e(a)g(t)o(yp)q(e)h(map)f(with)g Ft(count)c Fn(\001)f Ft(n)16 b Fy(en)o(tries)f(de\014ned)i(b)o(y:)75 915 y Fn(f)p Fy(\()p Fi(ty)r(pe)200 922 y Fj(0)218 915 y Fi(;)8 b(disp)323 922 y Fj(0)341 915 y Fy(\))p Fi(;)g(:::;)g Fy(\()o Fi(ty)r(pe)541 922 y Fg(n)p Fh(\000)p Fj(1)604 915 y Fi(;)g(disp)709 922 y Fg(n)p Fh(\000)p Fj(1)773 915 y Fy(\))p Fi(;)g Fy(\()p Fi(ty)r(pe)914 922 y Fj(0)931 915 y Fi(;)g(disp)1036 922 y Fj(0)1061 915 y Fy(+)h Fi(extent)p Fy(\))p Fi(;)f(:::;)g Fy(\()p Fi(ty)r(pe)1433 922 y Fg(n)p Fh(\000)p Fj(1)1496 915 y Fi(;)g(disp)1601 922 y Fg(n)p Fh(\000)p Fj(1)1673 915 y Fy(+)g Fi(extent)p Fy(\))p Fi(;)145 1015 y(:::;)g Fy(\()p Fi(ty)r(pe)307 1022 y Fj(0)324 1015 y Fi(;)g(disp)429 1022 y Fj(0)456 1015 y Fy(+)j Fi(extent)g Fn(\001)f Fy(\()p Ft(count)h Fn(\000)g Fy(1\)\))p Fi(;)d(:::;)f Fy(\()p Fi(ty)r(pe)1084 1022 y Fg(n)p Fh(\000)p Fj(1)1146 1015 y Fi(;)h(disp)1251 1022 y Fg(n)p Fh(\000)p Fj(1)1325 1015 y Fy(+)j Fi(extent)g Fn(\001)f Fy(\()p Ft(count)h Fn(\000)f Fy(1\)\))p Fn(g)p Fi(:)75 1135 y Ft(V)o(ecto)o(r)45 b Fy(The)16 b(function)g Ft(MPI)p 600 1135 14 2 v 16 w(TYPE)p 733 1135 V 17 w(VECTOR)g Fy(is)g(a)f(more)g(general)h(constructor)e(that)h(allo)o(ws)g(repli-)75 1192 y(cation)20 b(of)g(a)g(datat)o(yp)q(e)f(in)o(to)h(lo)q(cations)h(that)e (consist)i(of)e(equally)j(spaced)f(blo)q(c)o(ks.)35 b(Eac)o(h)20 b(blo)q(c)o(k)g(is)75 1248 y(obtained)g(b)o(y)f(concatenating)h(the)f(same)g (n)o(um)o(b)q(er)h(of)f(copies)h(of)f(the)g(old)h(datat)o(yp)q(e.)32 b(The)19 b(spacing)75 1304 y(b)q(et)o(w)o(een)d(blo)q(c)o(ks)f(is)h(a)f(m)o (ultiple)i(of)e(the)g(exten)o(t)g(of)g(the)g(old)h(datat)o(yp)q(e.)75 1408 y Ft(MPI)p 160 1408 V 16 w(TYPE)p 293 1408 V 17 w(VECTOR\()g(count,)g (blo)q(cklength,)g(stride,)g(oldt)o(yp)q(e,)g(newt)o(yp)q(e\))117 1485 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g (\(nonnegativ)o(e)f(in)o(teger\))117 1560 y(IN)155 b Ft(blo)q(cklength)371 b Fs(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o(h)i(blo)q(c)o(k)e (\(nonnegativ)o(e)h(in)o(te-)905 1617 y(ger\))117 1691 y(IN)155 b Ft(stride)484 b Fs(n)o(um)o(b)q(er)13 b(of)f(elemen)o(ts)h(b)q(et)o(w)o (een)h(start)g(of)f(eac)o(h)g(blo)q(c)o(k)g(\(in)o(te-)905 1748 y(ger\))117 1823 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 1898 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))75 2022 y Fr(int)23 b(MPI)p 245 2022 15 2 v 17 w(Type)p 358 2022 V 17 w(vector\(int)f(count,)h (int)h(blocklength,)e(int)i(stride,)393 2078 y(MPI)p 468 2078 V 17 w(Datatype)f(oldtype,)f(MPI)p 986 2078 V 17 w(Datatype)h(*newtype\))75 2165 y(MPI)p 150 2165 V 17 w(TYPE)p 263 2165 V 16 w(VECTOR\(COUNT,)f (BLOCKLENGTH,)h(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h(IERROR\))170 2221 y(INTEGER)g(COUNT,)g(BLOCKLENGTH,)g(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h (IERROR)166 2308 y Fy(Example:)c(Assume,)12 b(again,)g(that)f Ft(oldt)o(yp)q(e)i Fy(has)f(t)o(yp)q(e)g(map)g Fn(f)p Fy(\()p Ft(double)p Fi(;)c Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)p Fi(;)i Fy(with)i(exten)o(t)75 2364 y(16.)20 b(A)c(call)h(to)d Ft(MPI)p 428 2364 14 2 v 16 w(TYPE)p 561 2364 V 17 w(VECTOR\()i(2,)f(3,)g(4,)g(oldt)o(yp)q(e,)i(newt)o(yp)q(e\))g Fy(will)g(create)e(the)h(datat)o(yp)q(e)f(with)75 2421 y(t)o(yp)q(e)g(map)320 2521 y Fn(f)p Fy(\()p Ft(double)p Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fy(16\))p Fi(;)g Fy(\()p Ft(cha)o(r)m Fi(;)g Fy(24\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fy(32\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(40\))p Fi(;)279 2621 y Fy(\()p Ft(double)q Fi(;)g Fy(64\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(72\))p Fi(;)g Fy(\()p Ft(double)n Fi(;)g Fy(80\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(88\))p Fi(;)g Fy(\()p Ft(double)n Fi(;)g Fy(96\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(104\))p Fn(g)i Fy(:)75 2704 y(t)o(w)o(o)k(blo)q(c)o(ks)i(with)f(three)h(copies)g(eac)o(h)f (of)g(the)g(old)h(t)o(yp)q(e,)f(starting)f(4*16)g(apart.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 43 43 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(43)166 49 y(A)15 b(call)h(to)f Ft(MPI)p 439 49 14 2 v 16 w(TYPE)p 572 49 V 17 w(VECTOR\(3,)g(1,)g(-2,)f(oldt)o(yp)q(e,)i(newt)o(yp) q(e\))h Fy(will)g(create)e(the)g(datat)o(yp)q(e)237 154 y Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fn(\000)p Fy(32\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fn(\000)p Fy(24\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fn(\000)p Fy(64\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fn(\000)p Fy(56\))p Fn(g)p Fi(:)166 260 y Fy(In)16 b(general,)f(assume)g(that)g Ft(oldt)o(yp)q(e)h Fy(has)f(t)o(yp)q(e)g(map)600 365 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 372 y Fj(0)743 365 y Fi(;)8 b(disp)848 372 y Fj(0)865 365 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 372 y Fg(n)p Fh(\000)p Fj(1)1128 365 y Fi(;)g(disp)1233 372 y Fg(n)p Fh(\000)p Fj(1)1297 365 y Fy(\))p Fn(g)p Fi(;)75 470 y Fy(with)15 b(exten)o(t)e Fi(extent)p Fy(.)21 b(The)15 b(newly)g(created)f(datat)o(yp)q(e)f(has)i(a)e (t)o(yp)q(e)i(map)f(with)g Ft(count)9 b Fn(\001)f Ft(blo)q(cklength)i Fn(\001)e Fi(n)75 526 y Fy(en)o(tries:)611 586 y Fn(f)p Fy(\()p Fi(ty)r(pe)736 593 y Fj(0)754 586 y Fi(;)g(disp)859 593 y Fj(0)877 586 y Fy(\))p Fi(;)g(:::;)g Fy(\()o Fi(ty)r(pe)1077 593 y Fg(n)p Fh(\000)p Fj(1)1140 586 y Fi(;)g(disp)1245 593 y Fg(n)p Fh(\000)p Fj(1)1309 586 y Fy(\))p Fi(;)409 675 y Fy(\()p Fi(ty)r(pe)511 682 y Fj(0)530 675 y Fi(;)g(disp)635 682 y Fj(0)662 675 y Fy(+)j Fi(extent)p Fy(\))p Fi(;)d(:::;)g Fy(\()p Fi(ty)r(pe)1036 682 y Fg(n)p Fh(\000)p Fj(1)1099 675 y Fi(;)g(disp)1204 682 y Fg(n)p Fh(\000)p Fj(1)1278 675 y Fy(+)j Fi(extent)p Fy(\))p Fi(;)d(:::;)75 765 y Fy(\()p Fi(ty)r(pe)177 772 y Fj(0)196 765 y Fi(;)g(disp)301 772 y Fj(0)328 765 y Fy(+)i(\()p Ft(blo)q(cklength)i Fn(\000)e Fy(1\))f Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1065 772 y Fg(n)p Fh(\000)p Fj(1)1128 765 y Fi(;)g(disp)1233 772 y Fg(n)p Fh(\000)p Fj(1)1307 765 y Fy(+)j(\()p Ft(blo)q(cklength)g Fn(\000)f Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)273 854 y Fy(\()p Fi(ty)r(pe)375 861 y Fj(0)393 854 y Fi(;)f(disp)498 861 y Fj(0)526 854 y Fy(+)i Ft(stride)h Fn(\001)f Fi(extent)p Fy(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1036 861 y Fg(n)p Fh(\000)p Fj(1)1099 854 y Fi(;)g(disp)1204 861 y Fg(n)p Fh(\000)p Fj(1)1278 854 y Fy(+)j Ft(stride)g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)e(:::;)455 944 y Fy(\()p Fi(ty)r(pe)557 951 y Fj(0)576 944 y Fi(;)g(disp)681 951 y Fj(0)708 944 y Fy(+)j(\()p Ft(stride)g Fy(+)f Ft(blo)q(cklength)i Fn(\000)f Fy(1\))e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)403 1033 y Fy(\()p Fi(ty)r(pe)505 1040 y Fg(n)p Fh(\000)p Fj(1)570 1033 y Fi(;)g(disp)675 1040 y Fg(n)p Fh(\000)p Fj(1)749 1033 y Fy(+)i(\()p Ft(stride)h Fy(+)f Ft(blo)q(cklength)i Fn(\000)f Fy(1\))e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(::::;)523 1123 y Fy(\()p Fi(ty)r(pe)625 1130 y Fj(0)643 1123 y Fi(;)g(disp)748 1130 y Fj(0)776 1123 y Fy(+)i Ft(stride)h Fn(\001)f Fy(\()p Ft(count)h Fn(\000)f Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fi(;)e(:::;)476 1213 y Fy(\()p Fi(ty)r(pe)578 1220 y Fg(n)p Fh(\000)p Fj(1)643 1213 y Fi(;)g(disp)748 1220 y Fg(n)p Fh(\000)p Fj(1)822 1213 y Fy(+)j Ft(stride)g Fn(\001)e Fy(\()p Ft(count)i Fn(\000)g Fy(1\))e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)329 1302 y Fy(\()p Fi(ty)r(pe)431 1309 y Fj(0)450 1302 y Fi(;)g(disp)555 1309 y Fj(0)582 1302 y Fy(+)j(\()p Ft(stride)f Fn(\001)g Fy(\()p Ft(count)h Fn(\000)g Fy(1\))e(+)h Ft(blo)q(cklength)j Fn(\000)d Fy(1\))f Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)307 1392 y Fy(\()p Fi(ty)r(pe)409 1399 y Fg(n)p Fh(\000)p Fj(1)474 1392 y Fi(;)g(disp)579 1399 y Fg(n)p Fh(\000)p Fj(1)653 1392 y Fy(+)i(\()p Ft(stride)h Fn(\001)e Fy(\()p Ft(count)j Fn(\000)e Fy(1\))f(+)i Ft(blo)q(cklength)h Fn(\000)e Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fn(g)166 1479 y Fy(A)16 b(call)i(to)e Ft(MPI)p 443 1479 V 16 w(TYPE)p 576 1479 V 16 w(CONTIGUOUS\(count,)k(oldt)o(yp)q(e,)d (newt)o(yp)q(e\))h Fy(is)f(equiv)m(alen)o(t)h(to)e(a)g(call)i(to)75 1536 y Ft(MPI)p 160 1536 V 16 w(TYPE)p 293 1536 V 17 w(VECTOR\(count,)g(1,)f (1,)f(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))p Fy(,)h(or)e(to)g(a)h(call)g(to)f Ft(MPI)p 1495 1536 V 16 w(TYPE)p 1628 1536 V 17 w(VECTOR\(1,)75 1592 y(count,)g(n,)f(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))p Fy(,)h Ft(n)e Fy(arbitrary)l(.)75 1718 y Ft(Hvecto)o(r)45 b Fy(The)19 b(function)h Ft(MPI)p 631 1718 V 16 w(TYPE)p 764 1718 V 17 w(HVECTOR)g Fy(is)f(iden)o(tical)i(to)d Ft(MPI)p 1389 1718 V 16 w(TYPE)p 1522 1718 V 17 w(VECTOR)p Fy(,)h(except)75 1774 y(that)h Ft(stride)h Fy(is)g(giv)o(en)g(in)g(b)o(ytes,)g(rather)f(then)g(in)i (elemen)o(ts.)36 b(The)20 b(use)h(for)f(b)q(oth)g(t)o(yp)q(es)h(of)f(v)o (ector)75 1831 y(constructors)14 b(is)i(illustrated)h(in)f(section)f(1.13.7.) j(\()p Ft(H)d Fy(is)h(used)g(for)e(\\heterogeneous"\).)75 1935 y Ft(MPI)p 160 1935 V 16 w(TYPE)p 293 1935 V 17 w(HVECTOR\()i(count,)g(blo)q (cklength,)h(stride,)e(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))117 2013 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g (\(nonnegativ)o(e)f(in)o(teger\))117 2090 y(IN)155 b Ft(blo)q(cklength)371 b Fs(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o(h)i(blo)q(c)o(k)e (\(nonnegativ)o(e)h(in)o(te-)905 2147 y(ger\))117 2224 y(IN)155 b Ft(stride)484 b Fs(n)o(um)o(b)q(er)13 b(of)g(b)o(ytes)h(b)q(et)o(w)o(een)h (start)f(of)f(eac)o(h)h(blo)q(c)o(k)f(\(in)o(teger\))117 2301 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2378 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2503 y Fr(int)23 b(MPI)p 245 2503 15 2 v 17 w(Type)p 358 2503 V 17 w(hvector\(int)f(count,)h(int)h(blocklength,)e(MPI)p 1306 2503 V 17 w(Aint)h(stride,)393 2560 y(MPI)p 468 2560 V 17 w(Datatype)g(oldtype,)f(MPI)p 986 2560 V 17 w(Datatype)h(*newtype\))75 2647 y(MPI)p 150 2647 V 17 w(TYPE)p 263 2647 V 16 w(HVECTOR\(COUNT,)f (BLOCKLENGTH,)h(STRIDE,)f(OLDTYPE,)h(NEWTYPE,)g(IERROR\))170 2704 y(INTEGER)g(COUNT,)g(BLOCKLENGTH,)g(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h (IERROR)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 44 44 bop 75 -100 a Fy(44)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fy(Assume)f(that)g Ft(oldt)o(yp)q(e)h Fy(has)f(t)o(yp)q(e)g(map)600 155 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 162 y Fj(0)743 155 y Fi(;)8 b(disp)848 162 y Fj(0)865 155 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 162 y Fg(n)p Fh(\000)p Fj(1)1128 155 y Fi(;)g(disp)1233 162 y Fg(n)p Fh(\000)p Fj(1)1297 155 y Fy(\))p Fn(g)p Fi(;)75 260 y Fy(with)15 b(exten)o(t)e Fi(extent)p Fy(.)21 b(The)15 b(newly)g(created)f (datat)o(yp)q(e)f(has)i(a)e(t)o(yp)q(e)i(map)f(with)g Ft(count)9 b Fn(\001)f Ft(blo)q(cklength)i Fn(\001)e Fi(n)75 317 y Fy(en)o(tries:)611 377 y Fn(f)p Fy(\()p Fi(ty)r(pe)736 384 y Fj(0)754 377 y Fi(;)g(disp)859 384 y Fj(0)877 377 y Fy(\))p Fi(;)g(:::;)g Fy(\()o Fi(ty)r(pe)1077 384 y Fg(n)p Fh(\000)p Fj(1)1140 377 y Fi(;)g(disp)1245 384 y Fg(n)p Fh(\000)p Fj(1)1309 377 y Fy(\))p Fi(;)409 467 y Fy(\()p Fi(ty)r(pe)511 474 y Fj(0)530 467 y Fi(;)g(disp)635 474 y Fj(0)662 467 y Fy(+)j Fi(extent)p Fy(\))p Fi(;)d(:::;)g Fy(\()p Fi(ty)r(pe)1036 474 y Fg(n)p Fh(\000)p Fj(1)1099 467 y Fi(;)g(disp)1204 474 y Fg(n)p Fh(\000)p Fj(1)1278 467 y Fy(+)j Fi(extent)p Fy(\))p Fi(;)d(:::;)75 558 y Fy(\()p Fi(ty)r(pe)177 565 y Fj(0)196 558 y Fi(;)g(disp)301 565 y Fj(0)328 558 y Fy(+)i(\()p Ft(blo)q(cklength)i Fn(\000)e Fy(1\))f Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1065 565 y Fg(n)p Fh(\000)p Fj(1)1128 558 y Fi(;)g(disp)1233 565 y Fg(n)p Fh(\000)p Fj(1)1307 558 y Fy(+)j(\()p Ft(blo)q(cklength)g Fn(\000)f Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)434 649 y Fy(\()p Fi(ty)r(pe)536 656 y Fj(0)555 649 y Fi(;)f(disp)660 656 y Fj(0)687 649 y Fy(+)i Ft(stride)q Fy(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1037 656 y Fg(n)p Fh(\000)p Fj(1)1099 649 y Fi(;)g(disp)1204 656 y Fg(n)p Fh(\000)p Fj(1)1278 649 y Fy(+)j Ft(stride)q Fy(\))p Fi(;)d(:::;)455 740 y Fy(\()p Fi(ty)r(pe)557 747 y Fj(0)576 740 y Fi(;)g(disp)681 747 y Fj(0)708 740 y Fy(+)j Ft(stride)g Fy(+)f(\()p Ft(blo)q(cklength)i Fn(\000)f Fy(1\))e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)403 831 y Fy(\()p Fi(ty)r(pe)505 838 y Fg(n)p Fh(\000)p Fj(1)570 831 y Fi(;)g(disp)675 838 y Fg(n)p Fh(\000)p Fj(1)749 831 y Fy(+)i Ft(stride)h Fy(+)g(\()p Ft(blo)q(cklength)g Fn(\000)g Fy(1\))e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(::::;)603 921 y Fy(\()p Fi(ty)r(pe)705 928 y Fj(0)724 921 y Fi(;)g(disp)829 928 y Fj(0)856 921 y Fy(+)j Ft(stride)g Fn(\001)e Fy(\()p Ft(count)j Fn(\000)e Fy(1\)\))p Fi(;)e(:::;)557 1012 y Fy(\()p Fi(ty)r(pe)659 1019 y Fg(n)p Fh(\000)p Fj(1)724 1012 y Fi(;)g(disp)829 1019 y Fg(n)p Fh(\000)p Fj(1)903 1012 y Fy(+)i Ft(stride)h Fn(\001)f Fy(\()p Ft(count)h Fn(\000)f Fy(1\)\))p Fi(;)e(:::;)329 1103 y Fy(\()p Fi(ty)r(pe)431 1110 y Fj(0)450 1103 y Fi(;)g(disp)555 1110 y Fj(0)582 1103 y Fy(+)j Ft(stride)g Fn(\001)e Fy(\()p Ft(count)i Fn(\000)g Fy(1\))e(+)i(\()p Ft(blo)q(cklength)h Fn(\000)e Fy(1\))f Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)307 1194 y Fy(\()p Fi(ty)r(pe)409 1201 y Fg(n)p Fh(\000)p Fj(1)474 1194 y Fi(;)g(disp)579 1201 y Fg(n)p Fh(\000)p Fj(1)653 1194 y Fy(+)i Ft(stride)h Fn(\001)f Fy(\()p Ft(count)h Fn(\000)f Fy(1\))g(+)g(\()p Ft(blo)q(cklength)i Fn(\000)e Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fn(g)75 1320 y Ft(Indexed)47 b Fy(The)19 b(function)h Ft(MPI)p 629 1320 14 2 v 16 w(TYPE)p 762 1320 V 17 w(INDEXED)f Fy(allo)o(ws)h(replication)h(of)d(an)i(old)f(datat)o(yp)q(e) g(in)o(to)g(a)75 1377 y(sequence)c(of)e(con)o(tiguous)h(blo)q(c)o(ks,)g (where)g(eac)o(h)g(blo)q(c)o(k)h(can)f(con)o(tain)g(di\013eren)o(t)g(n)o(um)o (b)q(er)g(of)f(copies)i(and)75 1433 y(ha)o(v)o(e)f(a)g(di\013eren)o(t)g (displacemen)o(t.)21 b(All)16 b(blo)q(c)o(k)f(displacemen)o(ts)g(are)f(m)o (ultiples)i(of)e(the)h(old)f(t)o(yp)q(e)h(exten)o(t.)75 1538 y Ft(MPI)p 160 1538 V 16 w(TYPE)p 293 1538 V 17 w(INDEXED\()d(count,)i(a)o (rra)o(y)p 762 1538 V 14 w(of)p 813 1538 V 16 w(blo)q(cklengths,)h(a)o(rra)o (y)p 1185 1538 V 15 w(of)p 1237 1538 V 16 w(displacements,)e(oldt)o(yp)q(e,)h (newt)o(yp)q(e\))117 1673 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)34 b(of)f(blo)q(c)o(ks)h({)g(also)f(n)o(um)o(b)q(er)h(of)f(en)o(tries)i(in)905 1729 y Fo(a)o(rra)o(y)p 992 1729 13 2 v 15 w(of)p 1041 1729 V 15 w(displacements)17 b Fs(and)h Fo(a)o(rra)o(y)p 1483 1729 V 15 w(of)p 1532 1729 V 15 w(blo)q(cklengths)i Fs(\(non-)905 1786 y(negativ)o(e)14 b(in)o(teger\))117 1863 y(IN)155 b Ft(a)o(rra)o(y)p 416 1863 14 2 v 15 w(of)p 468 1863 V 16 w(blo)q(cklengths)191 b Fs(n)o(um)o(b)q(er)18 b(of)f(elemen)o(ts)h(p)q(er)i(blo)q(c)o(k)d(\(arra)o (y)i(of)e(nonnegativ)o(e)905 1919 y(in)o(tegers\))117 1997 y(IN)155 b Ft(a)o(rra)o(y)p 416 1997 V 15 w(of)p 468 1997 V 16 w(displacements)162 b Fs(displacemen)o(t)19 b(for)h(eac)o(h)g(blo)q(c)o (k,)g(in)g(m)o(ultiples)e(of)h Fo(oldt)o(yp)q(e)905 2053 y Fs(exten)o(t)c(\(arra)o(y)f(of)f(in)o(teger\))117 2131 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2208 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2334 y Fr(int)23 b(MPI)p 245 2334 15 2 v 17 w(Type)p 358 2334 V 17 w(indexed\(int)f(count,)h(int)h(*array)p 1068 2334 V 16 w(of)p 1132 2334 V 17 w(blocklengths,)393 2390 y(int)g(*array)p 636 2390 V 16 w(of)p 700 2390 V 17 w(displacements,)e(MPI)p 1147 2390 V 17 w(Datatype)g(oldtype,)393 2447 y(MPI)p 468 2447 V 17 w(Datatype)h(*newtype\))75 2534 y(MPI)p 150 2534 V 17 w(TYPE)p 263 2534 V 16 w(INDEXED\(COUNT,)f(ARRAY)p 757 2534 V 17 w(OF)p 822 2534 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1293 2534 V 16 w(OF)p 1357 2534 V 17 w(DISPLACEMENTS,)393 2591 y(OLDTYPE,)h(NEWTYPE,)g (IERROR\))170 2647 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2647 V 17 w(OF)p 716 2647 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 2647 V 17 w(OF)p 1323 2647 V 17 w(DISPLACEMENTS\(*\),)170 2704 y(OLDTYPE,)i(NEWTYPE,)g(IERROR)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 45 45 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(45)166 49 y(Example:)25 b(let)19 b Ft(oldt)o(yp)q(e)f Fy(ha)o(v)o(e)g(t)o(yp)q(e)g(map)f Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)p Fi(;)15 b Fy(with)j(exten)o(t)g(16.)27 b(Let)18 b Ft(B)g(=)75 106 y(\(3,)13 b(1\))h Fy(and)g(let)h Ft(D)f(=)g(\(4,)f(0\))p Fy(.)19 b(A)14 b(call)i(to)d Ft(MPI)p 855 106 14 2 v 16 w(TYPE)p 988 106 V 17 w(INDEXED\(2,)g(B,)h(D,)f(oldt)o(yp)q(e,)j(newt)o(yp)q(e\))f Fy(returns)75 162 y(a)g(datat)o(yp)q(e)g(with)g(t)o(yp)q(e)g(map)285 251 y Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(64\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(72\))p Fi(;)g Fy(\()p Ft(double)n Fi(;)g Fy(80\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(88\))p Fi(;)g Fy(\()p Ft(double)n Fi(;)g Fy(96\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(104\))p Fi(;)760 339 y Fy(\()p Ft(double)q Fi(;)g Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)j Fy(:)75 416 y(three)k(copies)h(of)e(the)i(old)f(t)o(yp)q(e)g(starting) f(at)h(displacemen)o(t)h(64,)e(and)h(one)h(cop)o(y)e(starting)h(at)f (displace-)75 473 y(men)o(t)h(0.)166 529 y(In)h(general,)f(assume)g(that)g Ft(oldt)o(yp)q(e)h Fy(has)f(t)o(yp)q(e)g(map)600 618 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 625 y Fj(0)743 618 y Fi(;)8 b(disp)848 625 y Fj(0)865 618 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 625 y Fg(n)p Fh(\000)p Fj(1)1128 618 y Fi(;)g(disp)1233 625 y Fg(n)p Fh(\000)p Fj(1)1297 618 y Fy(\))p Fn(g)p Fi(;)75 706 y Fy(with)35 b(exten)o(t)g Fp(extent)p Fy(.)79 b(Let)36 b Ft(B)f Fy(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1013 706 V 15 w(of)p 1065 706 V 16 w(blo)q(cklength)i Fy(argumen)o(t)d(and)h Ft(D)g Fy(b)q(e)h(the)75 763 y Ft(a)o(rra)o(y)p 173 763 V 15 w(of)p 225 763 V 16 w(displacements)15 b Fy(argumen)o(t.)k(The)c(newly)h(created)f(datat)o(yp)q(e)g(has)g Fi(n)c Fn(\001)1450 731 y Ff(P)1494 741 y Fe(count)p Fh(\000)p Fj(1)1494 774 y Fg(i)p Fj(=0)1618 763 y Ft(B)p Fy([)p Ft(i)p Fy(])k(en)o(tries:)284 851 y Fn(f)p Fy(\()p Fi(ty)r(pe)409 858 y Fj(0)428 851 y Fi(;)8 b(disp)533 858 y Fj(0)560 851 y Fy(+)j Ft(D)p Fy([)p Ft(0)p Fy(])e Fn(\001)g Fi(extent)p Fy(\))p Fi(;)f(:::;)g Fy(\()p Fi(ty)r(pe)1047 858 y Fg(n)p Fh(\000)p Fj(1)1111 851 y Fi(;)g(disp)1216 858 y Fg(n)p Fh(\000)p Fj(1)1290 851 y Fy(+)i Ft(D)p Fy([)p Ft(0)p Fy(])f Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)75 940 y Fy(\()p Fi(ty)r(pe)177 947 y Fj(0)196 940 y Fi(;)g(disp)301 947 y Fj(0)325 940 y Fy(+)f(\()p Ft(D)p Fy([)p Ft(0)p Fy(])e(+)i Ft(B)p Fy([)p Ft(0)p Fy(])f Fn(\000)h Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fi(;)h(:::;)g Fy(\()p Fi(ty)r(pe)1037 947 y Fg(n)p Fh(\000)p Fj(1)1099 940 y Fi(;)g(disp)1204 947 y Fg(n)p Fh(\000)p Fj(1)1275 940 y Fy(+)f(\()p Ft(D)p Fy([)p Ft(0)p Fy(])e(+)i Ft(B)p Fy([)p Ft(0)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fi(;)h(:::;)134 1017 y Fy(\()p Fi(ty)r(pe)236 1024 y Fj(0)255 1017 y Fi(;)g(disp)360 1024 y Fj(0)387 1017 y Fy(+)i Ft(D)p Fy([)p Ft(count)h Fn(\000)g Ft(1)p Fy(])e Fn(\001)h Fi(extent)p Fy(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1036 1024 y Fg(n)p Fh(\000)p Fj(1)1099 1017 y Fi(;)g(disp)1204 1024 y Fg(n)p Fh(\000)p Fj(1)1278 1017 y Fy(+)j Ft(D)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)f(:::;)375 1093 y Fy(\()p Fi(ty)r(pe)477 1100 y Fj(0)496 1093 y Fi(;)g(disp)601 1100 y Fj(0)628 1093 y Fy(+)i(\()p Ft(D)p Fy([)p Ft(count)h Fn(\000)f Ft(1)p Fy(])g(+)g Ft(B)p Fy([)p Ft(count)i Fn(\000)e Ft(1)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)f(:::;)346 1170 y Fy(\()p Fi(ty)r(pe)448 1177 y Fg(n)p Fh(\000)p Fj(1)513 1170 y Fi(;)g(disp)618 1177 y Fg(n)p Fh(\000)p Fj(1)692 1170 y Fy(+)j(\()p Ft(D)p Fy([)p Ft(count)f Fn(\000)h Ft(1)p Fy(])e(+)h Ft(B)p Fy([)p Ft(count)i Fn(\000)e Ft(1)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fn(g)p Fi(:)166 1247 y Fy(A)k(call)g(to)f Ft(MPI)p 431 1247 V 16 w(TYPE)p 564 1247 V 17 w(VECTOR\(count,)j(blo)q(cklength,)g (stride,)e(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))g Fy(is)f(equiv)m(alen)o(t)75 1303 y(to)i(a)f(call)j(to)d Ft(MPI)p 392 1303 V 16 w(TYPE)p 525 1303 V 17 w(INDEX\(count,)i(B,)f(D,)f(oldt)o(yp)q(e,)i(newt)o(yp)q(e\))h Fy(where)623 1392 y Ft(D)p Fy([)p Ft(j)p Fy(])12 b(=)h Fi(j)f Fn(\001)e Ft(stride)16 b Fi(;)8 b(j)14 b Fy(=)f(0)p Fi(;)8 b(:::;)g Ft(count)g Fn(\000)j Fy(1)p Fi(;)75 1481 y Fy(and)594 1537 y Ft(B)p Fy([)p Ft(j)p Fy(])h(=)h Ft(blo)q(cklength)18 b Fi(;)8 b(j)14 b Fy(=)f(0)p Fi(;)8 b(:::;)g Ft(count)g Fn(\000)j Fy(1)p Fi(:)75 1655 y Ft(Hindexed)47 b Fy(The)11 b(function)h Ft(MPI)p 643 1655 V 15 w(TYPE)p 775 1655 V 17 w(HINDEXED)f Fy(is)g(iden)o(tical)i(to)d Ft(MPI)p 1383 1655 V 15 w(TYPE)p 1515 1655 V 17 w(INDEXED)p Fy(,)g(except)75 1711 y(that)19 b(blo)q(c)o(k)i(displacemen)o(ts)h(in)41 b Ft(a)o(rra)o(y)p 774 1711 V 14 w(of)p 825 1711 V 16 w(displacements)20 b Fy(are)g(sp)q (eci\014ed)i(in)f(b)o(ytes,)g(rather)e(than)h(in)75 1768 y(m)o(ultiples)d(of) e(the)g Ft(oldt)o(yp)q(e)h Fy(exten)o(t.)75 1872 y Ft(MPI)p 160 1872 V 16 w(TYPE)p 293 1872 V 17 w(HINDEXED\()21 b(count,)k(a)o(rra)o(y)p 814 1872 V 14 w(of)p 865 1872 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1248 1872 V 15 w(of)p 1300 1872 V 16 w(displacements,)d(oldt)o(yp)q(e,)h (new-)75 1928 y(t)o(yp)q(e\))117 2005 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er)34 b(of)f(blo)q(c)o(ks)h({)g(also)f(n)o(um)o(b)q(er)h (of)f(en)o(tries)i(in)905 2062 y Fo(a)o(rra)o(y)p 992 2062 13 2 v 15 w(of)p 1041 2062 V 15 w(displacements)17 b Fs(and)h Fo(a)o(rra)o(y)p 1483 2062 V 15 w(of)p 1532 2062 V 14 w(blo)q(cklengths)i Fs(\(in)o(te-)905 2118 y(ger\))117 2191 y(IN)155 b Ft(a)o(rra)o(y)p 416 2191 14 2 v 15 w(of)p 468 2191 V 16 w(blo)q(cklengths)191 b Fs(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)f(of)g(nonnega-)905 2248 y(tiv)o(e)f(in)o(tegers\))117 2320 y(IN)155 b Ft(a)o(rra)o(y)p 416 2320 V 15 w(of)p 468 2320 V 16 w(displacements)162 b Fs(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h (blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 2393 y(IN)155 b Ft(oldt)o(yp)q(e)450 b Fs(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2466 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2591 y Fr(int)23 b(MPI)p 245 2591 15 2 v 17 w(Type)p 358 2591 V 17 w(hindexed\(int)f(count,)h(int)h(*array)p 1092 2591 V 16 w(of)p 1156 2591 V 17 w(blocklengths,)393 2647 y(MPI)p 468 2647 V 17 w(Aint)f(*array)p 748 2647 V 17 w(of)p 813 2647 V 17 w(displacements,)e(MPI)p 1259 2647 V 17 w(Datatype)i(oldtype,) 393 2704 y(MPI)p 468 2704 V 17 w(Datatype)g(*newtype\))-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 46 46 bop 75 -100 a Fy(46)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fr(MPI)p 150 49 15 2 v 17 w(TYPE)p 263 49 V 16 w(HINDEXED\(COUNT,)22 b(ARRAY)p 781 49 V 17 w(OF)p 846 49 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1317 49 V 16 w(OF)p 1381 49 V 17 w(DISPLACEMENTS,)393 106 y(OLDTYPE,)h(NEWTYPE,)g (IERROR\))170 162 y(INTEGER)g(COUNT,)g(ARRAY)p 651 162 V 17 w(OF)p 716 162 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 162 V 17 w(OF)p 1323 162 V 17 w(DISPLACEMENTS\(*\),)170 219 y(OLDTYPE,)i (NEWTYPE,)g(IERROR)166 306 y Fy(Assume)15 b(that)g Ft(oldt)o(yp)q(e)h Fy(has)f(t)o(yp)q(e)g(map)600 411 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 418 y Fj(0)743 411 y Fi(;)8 b(disp)848 418 y Fj(0)865 411 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 418 y Fg(n)p Fh(\000)p Fj(1)1128 411 y Fi(;)g(disp)1233 418 y Fg(n)p Fh(\000)p Fj(1)1297 411 y Fy(\))p Fn(g)p Fi(;)75 517 y Fy(with)35 b(exten)o(t)g Fp(extent)p Fy(.)79 b(Let)36 b Ft(B)f Fy(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1013 517 14 2 v 15 w(of)p 1065 517 V 16 w(blo)q(cklength)i Fy(argumen)o(t)d(and)h Ft(D)g Fy(b)q(e)h(the)75 573 y Ft(a)o(rra)o(y)p 173 573 V 15 w(of)p 225 573 V 16 w(displacements)21 b Fy(argumen)o(t.)39 b(The)22 b(newly)h(created)f(datat)o(yp)q(e)f(has)h(a)f(t)o(yp)q(e)h(map)f (with)i Fi(n)14 b Fn(\001)75 598 y Ff(P)119 608 y Fe(count)p Fh(\000)p Fj(1)119 641 y Fg(i)p Fj(=0)243 630 y Ft(B)p Fy([)p Ft(i)p Fy(])h(en)o(tries:)446 735 y Fn(f)p Fy(\()p Fi(ty)r(pe)571 742 y Fj(0)589 735 y Fi(;)8 b(disp)694 742 y Fj(0)721 735 y Fy(+)j Ft(D)p Fy([)p Ft(0)p Fy(])o(\))p Fi(;)d(:::;)g Fy(\()o Fi(ty)r(pe)1048 742 y Fg(n)p Fh(\000)p Fj(1)1111 735 y Fi(;)g(disp)1216 742 y Fg(n)p Fh(\000)p Fj(1)1290 735 y Fy(+)i Ft(D)p Fy([)p Ft(0)p Fy(])o(\))p Fi(;)e(:::;)75 844 y Fy(\()p Fi(ty)r(pe)177 851 y Fj(0)196 844 y Fi(;)g(disp)301 851 y Fj(0)325 844 y Fy(+)f Ft(D)p Fy([)p Ft(0)p Fy(])e(+)i(\()p Ft(B)p Fy([)p Ft(0)p Fy(])f Fn(\000)h Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fi(;)h(:::;)g Fy(\()p Fi(ty)r(pe)1037 851 y Fg(n)p Fh(\000)p Fj(1)1099 844 y Fi(;)g(disp)1204 851 y Fg(n)p Fh(\000)p Fj(1)1275 844 y Fy(+)f Ft(D)p Fy([)p Ft(0)p Fy(])f(+)h(\()p Ft(B)p Fy([)p Ft(0)p Fy(])f Fn(\000)h Fy(1\))g Fn(\001)g Fi(extent)p Fy(\))p Fi(;)h(:::;)295 934 y Fy(\()p Fi(ty)r(pe)397 941 y Fj(0)416 934 y Fi(;)g(disp)521 941 y Fj(0)548 934 y Fy(+)j Ft(D)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])o(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1037 941 y Fg(n)p Fh(\000)p Fj(1)1099 934 y Fi(;)g(disp)1204 941 y Fg(n)p Fh(\000)p Fj(1)1278 934 y Fy(+)j Ft(D)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])o(\))p Fi(;)e(:::;)375 1024 y Fy(\()p Fi(ty)r(pe)477 1031 y Fj(0)496 1024 y Fi(;)g(disp)601 1031 y Fj(0)628 1024 y Fy(+)i Ft(D)p Fy([)p Ft(count)h Fn(\000)g Ft(1)p Fy(])e(+)i(\()p Ft(B)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fi(;)f(:::;)346 1115 y Fy(\()p Fi(ty)r(pe)448 1122 y Fg(n)p Fh(\000)p Fj(1)513 1115 y Fi(;)g(disp)618 1122 y Fg(n)p Fh(\000)p Fj(1)692 1115 y Fy(+)j Ft(D)p Fy([)p Ft(count)f Fn(\000)h Ft(1)p Fy(])e(+)i(\()p Ft(B)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)f Fi(extent)p Fy(\))p Fn(g)p Fi(:)75 1241 y Ft(Struct)47 b(MPI)p 323 1241 V 16 w(TYPE)p 456 1241 V 16 w(STRUCT)23 b Fy(is)g(the)f(most)g(general)h(constructor:)33 b(It)22 b(further)g (generalizes)i(the)75 1297 y(previous)16 b(one)f(in)h(that)f(it)g(allo)o(ws)h (eac)o(h)f(blo)q(c)o(k)h(to)e(consist)i(of)f(replications)h(of)f(a)g (di\013eren)o(t)h(datat)o(yp)q(e.)75 1402 y Ft(MPI)p 160 1402 V 16 w(TYPE)p 293 1402 V 17 w(STRUCT\(count,)c(a)o(rra)o(y)p 729 1402 V 15 w(of)p 781 1402 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1151 1402 V 14 w(of)p 1202 1402 V 16 w(displacements,)e(a)o(rra)o(y)p 1599 1402 V 15 w(of)p 1651 1402 V 15 w(t)o(yp)q(es,)i(new-)75 1459 y(t)o(yp)q(e\))117 1537 y Fs(IN)155 b Ft(count)482 b Fs(n)o(um)o(b)q(er) 18 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))g({)g(also)f(n)o(um)o(b)q(er)h(of)f (en)o(tries)905 1593 y(in)c(arra)o(ys)h Fo(a)o(rra)o(y)p 1167 1593 13 2 v 15 w(of)p 1216 1593 V 14 w(t)o(yp)q(es)p Fs(,)h Fo(a)o(rra)o(y)p 1432 1593 V 15 w(of)p 1481 1593 V 14 w(displacements)e Fs(and)g Fo(a)o(r-)905 1650 y(ra)o(y)p 959 1650 V 15 w(of)p 1008 1650 V 15 w(blo)q(cklengths)117 1727 y Fs(IN)155 b Ft(a)o(rra)o(y)p 416 1727 14 2 v 15 w(of)p 468 1727 V 16 w(blo)q(cklength)208 b Fs(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g (\(arra)o(y)g(of)f(in)o(teger\))117 1804 y(IN)155 b Ft(a)o(rra)o(y)p 416 1804 V 15 w(of)p 468 1804 V 16 w(displacements)162 b Fs(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o (teger\))117 1882 y(IN)155 b Ft(a)o(rra)o(y)p 416 1882 V 15 w(of)p 468 1882 V 16 w(t)o(yp)q(es)327 b Fs(t)o(yp)q(e)20 b(of)f(elemen)o(ts) g(in)g(eac)o(h)h(blo)q(c)o(k)f(\(arra)o(y)g(of)g(handles)g(to)905 1938 y(datat)o(yp)q(e)14 b(ob)r(jects\))117 2015 y(OUT)108 b Ft(newt)o(yp)q(e)433 b Fs(new)15 b(datat)o(yp)q(e)f(\(handle\))75 2141 y Fr(int)23 b(MPI)p 245 2141 15 2 v 17 w(Type)p 358 2141 V 17 w(struct\(int)f(count,)h(int)h(*array)p 1044 2141 V 16 w(of)p 1108 2141 V 17 w(blocklengths,)393 2197 y(MPI)p 468 2197 V 17 w(Aint)f(*array)p 748 2197 V 17 w(of)p 813 2197 V 17 w(displacements,)e(MPI)p 1259 2197 V 17 w(Datatype)i(*array)p 1635 2197 V 16 w(of)p 1699 2197 V 17 w(types,)393 2254 y(MPI)p 468 2254 V 17 w(Datatype)g(*newtype\))75 2341 y(MPI)p 150 2341 V 17 w(TYPE)p 263 2341 V 16 w(STRUCT\(COUNT,)f(ARRAY)p 733 2341 V 17 w(OF)p 798 2341 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1269 2341 V 16 w(OF)p 1333 2341 V 17 w(DISPLACEMENTS,)393 2398 y(ARRAY)p 516 2398 V 17 w(OF)p 581 2398 V 17 w(TYPES,)h(NEWTYPE,)f(IERROR\))170 2454 y(INTEGER)h(COUNT,)g(ARRAY)p 651 2454 V 17 w(OF)p 716 2454 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 2454 V 17 w(OF)p 1323 2454 V 17 w(DISPLACEMENTS\(*\),)170 2511 y(ARRAY)p 293 2511 V 17 w(OF)p 358 2511 V 17 w(TYPES\(*\),)h(NEWTYPE,)h(IERROR)166 2598 y Fy(Example:)d(Let)c Ft(t)o(yp)q(e1)g Fy(ha)o(v)o(e)f(t)o(yp)q(e)g(map) 755 2704 y Fn(f)p Fy(\()p Ft(double)q Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(8\))p Fn(g)p Fi(;)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 47 47 bop 75 -100 a Fu(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b Fy(47)75 49 y(with)12 b(exten)o(t)f(16.)18 b(Let)11 b Ft(B)h(=)g(\(2,)f(1,) g(3\))p Fy(,)g Ft(D)h(=)f(\(0,)g(16,)h(26\))p Fy(,)e(and)i Ft(T)f(=)h(\(MPI)p 1321 49 14 2 v 15 w(FLO)o(A)l(T,)f(t)o(yp)q(e1,)i(MPI)p 1711 49 V 16 w(CHAR\))p Fy(.)75 106 y(Then)j(a)f(call)h(to)e Ft(MPI)p 455 106 V 16 w(TYPE)p 588 106 V 17 w(STRUCT\(3,)h(B,)g(D,)g(T,)f (newt)o(yp)q(e\))j Fy(returns)e(datat)o(yp)q(e)g(with)g(t)o(yp)q(e)h(map,)240 212 y Fn(f)p Fy(\()p Ft(\015oat)p Fi(;)8 b Fy(0\))p Fi(;)g Fy(\()p Ft(\015oat)m Fi(;)g Fy(4\))p Fi(;)g Fy(\()p Ft(double)o Fi(;)g Fy(16\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(24\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(26\))p Fi(;)g Fy(\()p Ft(cha)o(r)m Fi(;)g Fy(27\))p Fi(;)g Fy(\()p Ft(cha)o(r)n Fi(;)g Fy(28\))p Fn(g)i Fy(:)75 318 y(t)o(w)o(o)g(copies)j(of)e Ft(MPI)p 421 318 V 15 w(FLO)o(A)l(T)h Fy(starting)f(at)g(0,)g(follo)o(w)o(ed)h(b)o(y)f (one)h(cop)o(y)f(of)g Ft(t)o(yp)q(e1)i Fy(starting)e(at)f(16,)i(follo)o(w)o (ed)75 375 y(b)o(y)i(three)g(copies)h(of)e Ft(MPI)p 520 375 V 16 w(CHAR)p Fy(,)i(starting)e(at)g(26.)19 b(\(W)l(e)14 b(assume)g(that)f(a) g(\015oat)h(o)q(ccupies)h(four)f(b)o(ytes.\))166 433 y(In)i(general,)f(let)h Ft(T)f Fy(b)q(e)h(the)f Ft(a)o(rra)o(y)p 742 433 V 14 w(of)p 793 433 V 16 w(t)o(yp)q(es)i Fy(argumen)o(t,)d(where)31 b Ft(T[i])15 b Fy(is)h(a)f(handle)h(to)464 539 y Fi(ty)r(pemap)635 546 y Fg(i)661 539 y Fy(=)d Fn(f)p Fy(\()p Fi(ty)r(pe)834 520 y Fg(i)834 550 y Fj(0)853 539 y Fi(;)8 b(disp)958 520 y Fg(i)958 550 y Fj(0)975 539 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1176 520 y Fg(i)1176 550 y(n)1197 554 y Fd(i)1207 550 y Fh(\000)p Fj(1)1251 539 y Fi(;)g(disp)1356 520 y Fg(i)1356 550 y(n)1377 554 y Fd(i)1389 550 y Fh(\000)p Fj(1)1433 539 y Fy(\))p Fn(g)p Fi(;)75 645 y Fy(with)i(exten)o(t)g Fi(extent)434 652 y Fg(i)450 645 y Fy(.)18 b(Let)10 b Ft(B)g Fy(b)q(e)h(the)g Ft(a)o(rra)o(y)p 826 645 V 14 w(of)p 877 645 V 16 w(blo)q(cklength)h Fy(argumen)o(t)d(and)i Ft(D)f Fy(b)q(e)h(the)f Ft(a)o(rra)o(y)p 1671 645 V 14 w(of)p 1722 645 V 16 w(displacements)75 702 y Fy(argumen)o(t.)19 b(Then)d(the)f (newly)h(created)f(datat)o(yp)q(e)g(has)g(a)g(t)o(yp)q(e)g(map)g(with)1399 670 y Ff(P)1443 680 y Fe(count)p Fh(\000)p Fj(1)1443 713 y Fg(i)p Fj(=0)1567 702 y Ft(B)p Fy([)p Ft(i)p Fy(])9 b Fn(\001)h Fi(n)1693 709 y Fg(i)1722 702 y Fy(en)o(tries)472 821 y Fn(f)p Fy(\()p Fi(ty)r(pe)597 802 y Fj(0)597 832 y(0)615 821 y Fi(;)e(disp)720 802 y Fj(0)720 832 y(0)748 821 y Fy(+)i Ft(D)p Fy([)p Ft(0)p Fy(])o(\))p Fi(;)e(:::;)g Fy(\()p Fi(ty)r(pe)1075 802 y Fj(0)1075 832 y Fg(n)1096 836 y Fc(0)1111 821 y Fi(;)g(disp)1216 802 y Fj(0)1216 832 y Fg(n)1237 836 y Fc(0)1263 821 y Fy(+)j Ft(D)p Fy([)p Ft(0)p Fy(])o(\))p Fi(;)d(:::;)75 931 y Fy(\()p Fi(ty)r(pe)177 913 y Fj(0)177 943 y(0)196 931 y Fi(;)g(disp)301 913 y Fj(0)301 943 y(0)325 931 y Fy(+)f Ft(D)p Fy([)p Ft(0)p Fy(])g(+)g(\()p Ft(B)p Fy([)p Ft(0)p Fy(])g Fn(\000)g Fy(1\))g Fn(\001)g Fi(extent)839 938 y Fj(0)860 931 y Fy(\))p Fi(;)h(:::;)g Fy(\()p Fi(ty)r(pe)1061 913 y Fj(0)1061 943 y Fg(n)1081 947 y Fc(0)1097 931 y Fi(;)g(disp)1202 913 y Fj(0)1202 943 y Fg(n)1223 947 y Fc(0)1247 931 y Fy(+)f Ft(D)p Fy([)p Ft(0)p Fy(])g(+)g(\()p Ft(B)p Fy([)p Ft(0)p Fy(])i Fn(\000)i Ft(1)p Fy(\))c Fn(\001)g Fi(extent)1767 938 y Fj(0)1787 931 y Fy(\))p Fi(;)h(:::;)107 1024 y Fy(\()p Fi(ty)r(pe)209 1005 y Fe(count)p Fh(\000)p Fj(1)209 1035 y(0)326 1024 y Fi(;)g(disp)431 1005 y Fe(count)n Fh(\000)p Fj(1)431 1035 y(0)556 1024 y Fy(+)j Ft(D)p Fy([)p Ft(count)f Fn(\000)h Ft(1)p Fy(])o(\))p Fi(;)d(:::;)g Fy(\()p Fi(ty)r(pe)1045 1005 y Fe(count)m Fh(\000)p Fj(1)1045 1035 y Fg(n)1066 1039 y Fb(count)m Fa(\000)p Fc(1)1153 1035 y Fh(\000)p Fj(1)1198 1024 y Fi(;)g(disp)1303 1005 y Fe(count)n Fh(\000)p Fj(1)1303 1035 y Fg(n)1324 1039 y Fb(count)n Fa(\000)p Fc(1)1412 1035 y Fh(\000)p Fj(1)1467 1024 y Fy(+)i Ft(D)p Fy([)p Ft(count)h Fn(\000)g Ft(1)p Fy(])o(\))p Fi(;)d(:::;)218 1116 y Fy(\()p Fi(ty)r(pe)320 1097 y Fe(count)q Fh(\000)p Fj(1)320 1127 y(0)437 1116 y Fi(;)g(disp)542 1097 y Fe(count)o Fh(\000)p Fj(1)542 1127 y(0)668 1116 y Fy(+)i Ft(D)p Fy([)p Ft(count)h Fn(\000)g Ft(1)p Fy(])e(+)h(\()p Ft(B)p Fy([)p Ft(count)h Fn(\000)g Ft(1)p Fy(])e Fn(\000)i Fy(1\))e Fn(\001)h Fi(extent)1525 1123 y Fe(count)r Fh(\000)p Fj(1)1643 1116 y Fy(\))p Fi(;)e(:::;)204 1209 y Fy(\()p Fi(ty)r(pe)306 1190 y Fe(count)p Fh(\000)p Fj(1)306 1220 y Fg(n)327 1224 y Fb(count)p Fa(\000)p Fc(1)417 1220 y Fh(\000)p Fj(1)461 1209 y Fi(;)g(disp)566 1190 y Fe(count)o Fh(\000)p Fj(1)566 1220 y Fg(n)587 1224 y Fb(count)o Fa(\000)p Fc(1)676 1220 y Fh(\000)p Fj(1)730 1209 y Fy(+)j Ft(D)p Fy([)p Ft(count)g Fn(\000)f Ft(1)p Fy(])g(+)g(\()p Ft(B)p Fy([)p Ft(count)h Fn(\000)f Ft(1)p Fy(])g Fn(\000)g Ft(1)p Fy(\))g Fn(\001)g Fi(extent)1588 1216 y Fe(count)q Fh(\000)p Fj(1)1706 1209 y Fy(\))p Fn(g)166 1298 y Fy(A)15 b(call)h(to)f Ft(MPI)p 439 1298 V 16 w(TYPE)p 572 1298 V 16 w(HINDEXED\()g(count,)h(B,)f(D,)g(oldt)o(yp) q(e,)h(newt)o(yp)q(e\))g Fy(is)g(equiv)m(alen)o(t)h(to)d(a)h(call)75 1355 y(to)d Ft(MPI)p 213 1355 V 16 w(TYPE)p 346 1355 V 17 w(STRUCT\()h (count,)i(B,)e(D,)f(T,)h(newt)o(yp)q(e\))p Fy(,)h(where)f(eac)o(h)g(en)o(try) g(of)g Ft(T)g Fy(is)g(equal)h(to)e Ft(oldt)o(yp)q(e)p Fy(.)75 1485 y Fq(1.13.2)49 b(Address)16 b(and)g(extent)f(functions)75 1573 y Fy(The)j(displacemen)o(ts)i(in)f(a)f(general)g(datat)o(yp)q(e)g(are)g (relativ)o(e)h(to)e(some)h(initial)i(bu\013er)e(address.)29 b Fx(Ab-)75 1630 y(solute)20 b(addresses)15 b Fy(can)i(b)q(e)g(substituted)h (for)e(these)h(displacemen)o(ts:)24 b(w)o(e)16 b(treat)g(them)h(as)f (displace-)75 1686 y(men)o(ts)e(relativ)o(e)h(to)f(\\address)g(zero",)g(the)h (start)e(of)h(the)h(address)g(space.)20 b(This)15 b(initial)h(address)f(zero) f(is)75 1742 y(indicated)i(b)o(y)d(the)h(constan)o(t)f Fo(MPI)p 668 1742 13 2 v 15 w(BOTTOM)p Fy(.)g(Th)o(us,)h(a)f(datat)o(yp)q(e)g(can)i (sp)q(ecify)g(the)f(absolute)g(address)75 1799 y(of)k(the)g(en)o(tries)g(in)h (the)f(comm)o(unication)h(bu\013er,)f(in)h(whic)o(h)g(case)f(the)g Ft(buf)h Fy(argumen)o(t)e(is)i(passed)f(the)75 1855 y(v)m(alue)e Fo(MPI)p 269 1855 V 15 w(BOTTOM)p Fy(.)166 1913 y(The)31 b(address)g(of)f(a)h (lo)q(cation)g(in)h(memory)e(can)h(b)q(e)h(found)f(b)o(y)g(in)o(v)o(oking)g (the)g(function)75 1970 y Ft(MPI)p 160 1970 14 2 v 16 w(ADDRESS)p Fy(.)75 2075 y Ft(MPI)p 160 2075 V 16 w(ADDRESS\(lo)q(cation,)16 b(address\))117 2153 y Fs(IN)155 b Ft(lo)q(cation)437 b Fs(lo)q(cation)13 b(in)h(caller)f(memory)f(\(c)o(hoice\))117 2231 y(OUT)108 b Ft(address)449 b Fs(address)15 b(of)f(lo)q(cation)f(\(in)o(teger\))75 2357 y Fr(int)23 b(MPI)p 245 2357 15 2 v 17 w(Address\(void*)f(location,)h (MPI)p 907 2357 V 17 w(Aint)g(address\))75 2445 y(MPI)p 150 2445 V 17 w(ADDRESS\(LOCATION,)e(ADDRESS,)i(IERROR\))170 2501 y()g(LOCATION\(*\))170 2558 y(INTEGER)g(ADDRESS,)g(IERROR)166 2646 y Fy(Returns)16 b(the)f(\(b)o(yte\))f(address)i(of)e Ft(lo)q(cation)p Fy(.)166 2704 y(Example:)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 48 48 bop 75 -100 a Fy(48)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)147 49 y Fr(REAL)23 b(A\(100,100\))147 106 y(INTEGER)f(I1,)i(I2,)f(DIFF)147 162 y(CALL)g(MPI_ADDRESS\(A\(1,1\),)e (I1,)j(IERROR\))147 219 y(CALL)f(MPI_ADDRESS\(A\(10,10\),)e(I2,)i(IERROR\)) 147 275 y(DIFF)g(=)h(I2)f(-)h(I1)75 332 y(!)g(The)f(value)g(of)h(DIFF)f(is)h (909*sizeofreal;)e(the)h(values)g(of)h(I1)f(and)h(I2)f(are)75 388 y(!)h(implementation)e(dependent.)189 502 y Fp(A)n(dvic)n(e)e(to)i (users.)73 b Fy(C)20 b(users)h(ma)o(y)f(b)q(e)h(tempted)g(to)f(a)o(v)o(oid)h (the)f(usage)h(of)f Ft(MPI)p 1659 502 14 2 v 16 w(ADDRESS)189 558 y Fy(and)e(rely)g(on)g(the)g(a)o(v)m(ailabilit)o(y)i(of)d(the)h(address)g (op)q(erator)f(&.)29 b(Note,)17 b(ho)o(w)o(ev)o(er,)h(that)f Fo(&)h Fp(c)n(ast-)189 614 y(expr)n(ession)c Fy(is)j(a)e(p)q(oin)o(ter,)i (not)e(an)h(address.)22 b(ANSI)17 b(C)f(do)q(es)g(not)g(require)g(that)g(the) g(v)m(alue)h(of)e(a)189 671 y(p)q(oin)o(ter)g(\(or)g(the)g(p)q(oin)o(ter)h (cast)f(to)f Fo(int)p Fy(\))h(b)q(e)h(the)g(absolute)f(address)h(of)e(the)i (ob)s(ject)e(p)q(oin)o(ted)j(at)d({)189 727 y(although)j(this)g(is)h(the)f (common)g(case.)25 b(F)l(urthermore,)17 b(referencing)h(ma)o(y)e(not)h(ha)o (v)o(e)g(a)f(unique)189 784 y(de\014nition)i(on)f(mac)o(hines)g(with)g (segmen)o(ted)g(address)g(space.)24 b(The)17 b(use)g(of)f Ft(MPI)p 1602 784 V 16 w(ADDRESS)i Fy(to)189 840 y(\\reference")11 b(C)g(v)m(ariables) h(guaran)o(tees)e(p)q(ortabilit)o(y)i(to)f(suc)o(h)g(mac)o(hines)h(as)e(w)o (ell.)20 b(\()p Fp(End)12 b(of)g(advic)n(e)189 897 y(to)k(users.)p Fy(\))166 1012 y(The)f(follo)o(wing)h(auxiliary)h(functions)f(pro)o(vide)f (useful)i(information)e(on)g(deriv)o(ed)h(datat)o(yp)q(es.)75 1118 y Ft(MPI)p 160 1118 V 16 w(TYPE)p 293 1118 V 17 w(EXTENT\(datat)o(yp)q (e,)g(extent\))117 1197 y Fs(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(\(handle\))117 1276 y(OUT)108 b Ft(extent)471 b Fs(datat)o(yp)q(e)14 b(exten)o(t)h(\(in)o(teger\))75 1402 y Fr(int)23 b(MPI)p 245 1402 15 2 v 17 w(Type)p 358 1402 V 17 w(extent\(MPI)p 615 1402 V 16 w(Datatype)f(datatype,)h(MPI)p 1156 1402 V 17 w(Aint)g(*extent\))75 1490 y(MPI)p 150 1490 V 17 w(TYPE)p 263 1490 V 16 w(EXTENT\(DATATYPE,)f(EXTENT,)h(IERROR\))170 1547 y(INTEGER)g(DATATYPE,)g(EXTENT,)g(IERROR)166 1635 y Fy(Returns)16 b(the)f(exten)o(t)g(of)g(a)f(datat)o(yp)q(e)h({)g(where)g(exten)o(t)g(is)h (as)f(de\014ned)h(in)g(Eq.)f(1.1)f(on)h(page)h(41.)75 1740 y Ft(MPI)p 160 1740 14 2 v 16 w(TYPE)p 293 1740 V 17 w(SIZE\(datat)o(yp)q(e,) f(size\))117 1819 y Fs(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(\(handle\))117 1898 y(OUT)108 b Ft(size)519 b Fs(datat)o(yp)q(e)14 b(size)h(\(in)o(teger\))75 2024 y Fr(int)23 b(MPI)p 245 2024 15 2 v 17 w(Type)p 358 2024 V 17 w(size\(MPI)p 567 2024 V 16 w(Datatype)g(datatype,)f(MPI)p 1108 2024 V 17 w(Aint)h(*size\))75 2113 y(MPI)p 150 2113 V 17 w(TYPE)p 263 2113 V 16 w(SIZE\(DATATYPE,)f(SIZE,)i (IERROR\))170 2169 y(INTEGER)f(DATATYPE,)g(SIZE,)g(IERROR)166 2257 y Fy(Returns)15 b(the)h(n)o(um)o(b)q(er)f(of)g(b)o(ytes)f(o)q(ccupied)j (b)o(y)e(en)o(tries)h(in)g(the)f(datat)o(yp)q(e)f({)h(i.e.)20 b(the)15 b(sum)g(of)g(the)75 2314 y(sizes)h(of)f(the)g(datat)o(yp)q(e)g (elemen)o(ts.)75 2419 y Ft(MPI)p 160 2419 14 2 v 16 w(TYPE)p 293 2419 V 17 w(COUNT\(datat)o(yp)q(e,)i(count\))117 2499 y Fs(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(\(handle\))117 2577 y(OUT)108 b Ft(count)482 b Fs(datat)o(yp)q(e)14 b(coun)o(t)g(\(in)o (teger\))75 2704 y Fr(int)23 b(MPI)p 245 2704 15 2 v 17 w(Type)p 358 2704 V 17 w(count\(MPI)p 591 2704 V 16 w(Datatype)g(datatype,)f(int)i (*count\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 49 49 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(49)75 49 y Fr(MPI)p 150 49 15 2 v 17 w(TYPE)p 263 49 V 16 w(COUNT\(DATATYPE,)22 b(COUNT,)h(IERROR\))170 106 y(INTEGER)g(DATATYPE,)g (COUNT,)g(IERROR)166 192 y Fy(Returns)16 b(the)f(n)o(um)o(b)q(er)g(of)g(en)o (tries)h(in)g(the)f(datat)o(yp)q(e.)75 316 y Fq(1.13.3)49 b(Lo)o(w)o(er-b)q (ound)17 b(and)g(upp)q(er-b)q(ound)g(ma)o(rk)n(ers)75 402 y Fy(It)e(is)g(often)g(con)o(v)o(enien)o(t)g(to)g(de\014ne)h(explicitly)h(the)e (lo)o(w)o(er)g(b)q(ound)h(and)f(upp)q(er)h(b)q(ound)f(of)g(a)g(t)o(yp)q(e)g (map,)75 459 y(and)20 b(o)o(v)o(erwrite)g(the)g(de\014nition)i(giv)o(en)f(b)o (y)f(Equation)g(1.1)g(on)g(page)g(41.)34 b(This)21 b(allo)o(ws)f(to)g (de\014ne)h(a)75 515 y(datat)o(yp)q(e)12 b(that)g(has)h(\\holes")g(at)f(its)h (b)q(eginning)h(or)f(its)g(end;)g(or)g(a)f(datat)o(yp)q(e)g(with)h(en)o (tries)g(that)f(extend)75 572 y(ab)q(o)o(v)o(e)k(the)g(upp)q(er)g(b)q(ound)h (or)f(b)q(elo)o(w)g(the)g(lo)o(w)o(er)g(b)q(ound.)23 b(Examples)16 b(of)g(suc)o(h)g(a)g(usage)g(are)f(pro)o(vided)75 628 y(in)h(section)g (1.13.7.)j(T)l(o)d(ac)o(hiev)o(e)g(this,)g(w)o(e)f(add)h(t)o(w)o(o)e (additional)j(\\pseudo-datat)o(yp)q(es")e Ft(MPI)p 1706 628 14 2 v 16 w(LB)p Fy(,)g(and)75 685 y Ft(MPI)p 160 685 V 16 w(UB)i Fy(that)e(can)i(b)q(e)g(used,)g(resp)q(ectiv)o(ely)l(,)h(to)e(mark)f (the)i(lo)o(w)o(er)f(b)q(ound)h(or)f(the)h(upp)q(er)g(b)q(ound)g(of)f(a)75 741 y(datat)o(yp)q(e.)j(These)c(pseudo-datat)o(yp)q(es)g(o)q(ccup)o(y)g(no)f (space)h(\()p Fi(extent)p Fy(\()p Ft(MPI)p 1357 741 V 17 w(LB)p Fy(\))d(=)h Fi(extent)p Fy(\()p Ft(MPI)p 1734 741 V 17 w(UB)p Fy(\))g(=)75 798 y(0\).)18 b(They)c(do)f(not)f(a\013ect)g(the)h(size)h(or)f (coun)o(t)g(of)f(a)h(datat)o(yp)q(e,)f(and)h(do)g(not)g(a\013ect)f(the)h(the) g(con)o(ten)o(t)g(of)f(a)75 854 y(message)i(created)h(with)h(this)f(datat)o (yp)q(e.)k(Ho)o(w)o(ev)o(er,)14 b(they)h(do)g(a\013ect)f(the)h(de\014nition)h (of)f(the)g(exten)o(t)f(of)75 911 y(a)e(datat)o(yp)q(e)g(and,)h(therefore,)g (a\013ect)f(the)g(outcome)h(of)f(a)g(replication)i(of)f(this)g(datat)o(yp)q (e)f(b)o(y)g(a)g(datat)o(yp)q(e)75 967 y(constructor.)166 1024 y(Example:)27 b(Let)19 b Ft(D)f(=)h(\(-3,)f(0,)h(6\))p Fy(;)g Ft(T)f(=)h(\(MPI)p 990 1024 V 15 w(LB,)f(MPI)p 1173 1024 V 16 w(INT,)g(MPI)p 1378 1024 V 16 w(UB\))p Fy(,)g(and)h Ft(B)f(=)h(\(1,)f(1,)h (1\))p Fy(.)75 1080 y(Then)e(a)e(call)i(to)f Ft(MPI)p 459 1080 V 16 w(TYPE)p 592 1080 V 16 w(STRUCT\(3,)g(B,)g(D,)g(T,)f(t)o(yp)q(e1\))i Fy(creates)e(a)h(new)g(datat)o(yp)q(e)g(that)f(has)h(an)75 1137 y(exten)o(t)f(of)g(9)g(\(from)f(-3)i(to)e(5\),)h(and)g(con)o(tains)h(an) f(in)o(teger)h(at)e(displacemen)o(t)j(0.)j(This)c(is)g(the)f(datat)o(yp)q(e) 75 1193 y(de\014ned)g(b)o(y)g(the)f(sequence)h Fn(f)p Ft(\(lb,)f(-3\),)f (\(int,)i(0\),)e(\(ub,)h(6\))p Fn(g)g Fy(.)19 b(If)c(this)f(t)o(yp)q(e)g(is)h (replicated)h(t)o(wice)e(b)o(y)g(a)g(call)75 1250 y(to)f Ft(MPI)p 214 1250 V 16 w(TYPE)p 347 1250 V 16 w(CONTIGUOUS\(2,)i(t)o(yp)q(e1,)f(t)o (yp)q(e2\))g Fy(then)g(the)g(newly)g(created)g(t)o(yp)q(e)f(can)h(b)q(e)g (describ)q(ed)75 1306 y(b)o(y)f(the)g(sequence)h Fn(f)p Ft(\(lb,)f(-3\),)g (\(int,)g(0\),)f(\(int,9\),)h(\(ub,)h(15\))p Fn(g)d Fy(.)20 b(\(En)o(tries)12 b(of)h(t)o(yp)q(e)g Ft(lb)g Fy(or)g Ft(ub)h Fy(can)f(b)q(e)h(deleted)75 1363 y(if)i(they)f(are)g(not)g(at)f(the)h(end-p)q (oin)o(ts)i(of)e(the)g(datat)o(yp)q(e.\))166 1419 y(In)h(general,)f(if)476 1522 y Fi(T)6 b(y)r(pemap)12 b Fy(=)h Fn(f)p Fy(\()p Fi(ty)r(pe)849 1529 y Fj(0)867 1522 y Fi(;)8 b(disp)972 1529 y Fj(0)990 1522 y Fy(\))p Fi(;)g(:::;)f Fy(\()p Fi(ty)r(pe)1190 1529 y Fg(n)p Fh(\000)p Fj(1)1253 1522 y Fi(;)h(disp)1358 1529 y Fg(n)p Fh(\000)p Fj(1)1421 1522 y Fy(\))p Fn(g)p Fi(;)75 1625 y Fy(then)16 b(the)f Fx(lo)o(w)o(er)h(b)q(ound)g Fy(of)f Fi(T)6 b(y)r(pemap)15 b Fy(is)g(de\014ned)i(to)e(b)q(e)227 1759 y Fi(l)q(b)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))k(=)544 1687 y Ff(\()598 1730 y Fy(min)674 1737 y Fg(j)699 1730 y Fi(disp)783 1737 y Fg(j)1142 1730 y Fy(if)16 b(no)f(en)o(try)g(has)g(basic)h(t)o(yp)q(e)f Ft(lb)598 1787 y Fy(min)q Fn(f)p Fi(disp)781 1794 y Fg(j)826 1787 y Fy(:)27 b Fi(ty)r(pe)950 1794 y Fg(j)981 1787 y Fy(=)13 b Ft(lb)p Fn(g)41 b Fy(otherwise)166 1892 y(Similarly)l(,)17 b(the)e Fx(upp)q(er)j(b)q(ound)e Fy(of)e Fi(T)6 b(y)r(pemap)15 b Fy(is)h(de\014ned)h(to)d(b)q(e)191 2026 y Fi(ub)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))11 b(=)520 1954 y Ff(\()575 1997 y Fy(max)659 2004 y Fg(j)684 1997 y Fi(disp)768 2004 y Fg(j)795 1997 y Fy(+)g Fi(siz)r(eof)5 b Fy(\()p Fi(ty)r(pe)1073 2004 y Fg(j)1091 1997 y Fy(\))56 b(if)16 b(no)f(en)o(try)g(has)g(basic)h(t)o(yp)q (e)f Ft(ub)575 2054 y Fy(max)o Fn(f)p Fi(disp)766 2061 y Fg(j)811 2054 y Fy(:)27 b Fi(ty)r(pe)935 2061 y Fg(j)966 2054 y Fy(=)13 b Ft(ub)q Fn(g)66 b Fy(otherwise)166 2159 y(Then)441 2217 y Fi(extent)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))14 b(=)e Fi(ub)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))j Fn(\000)i Fi(l)q(b)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))i(+)i Fi(\017)75 2301 y Fy(If)19 b Fi(ty)r(pe)208 2308 y Fg(i)242 2301 y Fy(requires)h(alignmen)o(t)g(to)e(a)h (b)o(yte)g(address)g(that)g(is)h(a)f(m)o(ultiple)i(of)d Fi(k)1460 2308 y Fg(i)1474 2301 y Fy(,)i(then)f Fi(\017)h Fy(is)g(the)f(least)75 2358 y(nonnegativ)o(e)d(incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p Fy(\()p Fi(T)6 b(y)r(pemap)p Fy(\))16 b(to)e(the)h(next)h(m)o (ultiple)h(of)d(max)1794 2365 y Fg(i)1815 2358 y Fi(k)1839 2365 y Fg(i)1853 2358 y Fy(.)166 2414 y(The)g(formal)f(de\014nitions)j(giv)o (en)e(for)f(the)h(v)m(arious)g(datat)o(yp)q(e)f(constructors)g(apply)i(no)o (w,)e(with)h(the)75 2471 y(amended)i(de\014nition)h(of)e Fx(exten)o(t)p Fy(.)166 2604 y Fl(Discussion:)52 b Fs(The)15 b(follo)o(wing)c(additional)h (functions)i(w)o(ere)h(suggested:)575 b Fk(\()166 2654 y Fo(MPI)p 243 2654 13 2 v 14 w(TYPE)p 365 2654 V 14 w(LB\()15 b(datat)o(yp)q(e,)f (displacement\))75 2704 y Fs(Returns)h(the)f(lo)o(w)o(er)g(b)q(ound)g(of)f (the)h(t)o(yp)q(e)h(map)d(of)h(datat)o(yp)q(e.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 50 50 bop 75 -100 a Fy(50)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fo(MPI)p 243 49 13 2 v 14 w(TYPE)p 365 49 V 14 w(UB\()e(datat)o(yp)q(e,)g(displacement\))75 99 y Fs(Returns)h(the)f(upp)q(er)h(b)q(ound)f(of)f(the)i(t)o(yp)q(e)f(map)f (of)g(datat)o(yp)q(e.)166 156 y Fo(MPI)p 243 156 V 14 w(MO)o(VE\(from)o(buf,) e(fromt)o(yp)q(e,)g(fromcount,)g(tobuf,)j(tot)o(yp)q(e,)g(to)q(count\))75 212 y Fs(Lo)q(cal)f(mo)o(v)o(e)e(of)i(data.)18 b(Essen)o(tially)13 b(equiv)n(alen)o(t)f(to)h Fo(MPI)p 982 212 V 14 w(SENDRECV)g Fs(with)g Fo(source)h(=)g(dest)p Fs(,)f(but)h(could)f(b)q(e)h(de-)75 269 y(\014ned)d(to)f(b)q(e)h(lenien)o(t)g(w.r.t.)e(t)o(yp)q(e)i(matc)o(hing.) k(It)c(could)f(b)q(e)h(then)g(used)g(for)f(pac)o(king)g(\(with)g Fo(tot)o(yp)q(e)h(=)g(MPI)p 1738 269 V 14 w(BYTE)p Fs(\))75 325 y(or)j(unpac)o(king)f(\(with)h Fo(from)o(t)o(yp)q(e)e(=)i(MPI)p 726 325 V 15 w(BYTE)p Fs(\).)75 532 y Fq(1.13.4)49 b(Comm)o(it)13 b(and)k(free)75 618 y Fy(A)g(datat)o(yp)q(e)f(ob)s(ject)h(has)g(to)f(b)q(e)h Fx(committed)h Fy(b)q(efore)f(it)g(can)g(b)q(e)h(used)g(in)f(a)g(comm)o (unication.)26 b(The)75 675 y(system)20 b(ma)o(y)g(\\compile")i(at)e(commit)g (time)h(an)g(in)o(ternal)g(represen)o(tation)g(for)f(the)h(datat)o(yp)q(e)f (that)75 731 y(facilitates)c(comm)o(unication,)g(e.g.)k(c)o(hange)15 b(from)g(a)g(compacted)g(represen)o(tation)h(to)e(a)h(\015at)g(represen-)75 788 y(tation)i(of)f(the)h(datat)o(yp)q(e,)g(and)g(select)g(the)g(most)g(con)o (v)o(enien)o(t)g(transfer)f(mec)o(hanism.)26 b(A)17 b(committed)75 844 y(datat)o(yp)q(e)e(can)g(still)i(b)q(e)e(used)h(as)f(a)g(argumen)o(t)f (in)i(datat)o(yp)q(e)f(constructors.)166 901 y(Note:)28 b(the)20 b(commit)f(op)q(eration)h(commits)f(the)h(datat)o(yp)q(e,)f(i.e.,)i(the)e (formal)g(description)i(of)e(a)75 957 y(comm)o(unication)i(bu\013er,)h(not)e (the)h(con)o(ten)o(t)f(of)g(that)g(bu\013er.)36 b(Th)o(us,)22 b(after)e(a)g(datat)o(yp)q(e)g(has)h(b)q(een)75 1014 y(committed,)d(it)h(can) f(b)q(e)h(rep)q(eatedly)g(reused)g(to)e(comm)o(unicate)h(the)h(c)o(hanging)f (con)o(ten)o(t)g(of)f(a)h(bu\013er)75 1070 y(or,)c(indeed,)j(the)e(con)o(ten) o(t)g(of)g(di\013eren)o(t)g(bu\013ers,)g(with)g(di\013eren)o(t)h(starting)e (addresses.)75 1174 y Ft(MPI)p 160 1174 14 2 v 16 w(TYPE)p 293 1174 V 17 w(COMMIT\(datat)o(yp)q(e\))117 1252 y Fs(INOUT)62 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(that)g(is)g(committed)e (\(handle\))75 1377 y Fr(int)23 b(MPI)p 245 1377 15 2 v 17 w(Type)p 358 1377 V 17 w(commit\(MPI)p 615 1377 V 16 w(Datatype)f (*datatype\))75 1463 y(MPI)p 150 1463 V 17 w(TYPE)p 263 1463 V 16 w(COMMIT\(DATATYPE,)g(IERROR\))170 1520 y(INTEGER)h(DATATYPE,)g(IERROR) 166 1607 y Fy(There)15 b(is)h(no)f(need)h(to)f(commit)g(basic)h(datat)o(yp)q (es;)e(they)h(are)g(\\pre-committed".)75 1711 y Ft(MPI)p 160 1711 14 2 v 16 w(TYPE)p 293 1711 V 17 w(FREE\(datat)o(yp)q(e\))117 1788 y Fs(INOUT)62 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(that)g(is)g(freed)h(\(handle\))75 1913 y Fr(int)23 b(MPI)p 245 1913 15 2 v 17 w(Type)p 358 1913 V 17 w(free\(MPI)p 567 1913 V 16 w(Datatype)g(*datatype\))75 2000 y(MPI)p 150 2000 V 17 w(TYPE)p 263 2000 V 16 w(FREE\(DATATYPE,)f(IERROR\))170 2056 y(INTEGER)h(DATATYPE,)g(IERROR)166 2143 y Fy(Marks)16 b(the)h(datat)o(yp)q(e)f(ob)s(ject)g(asso)q(ciated)h(with)g Ft(datat)o(yp)q(e)i Fy(for)d(deallo)q(cation.)26 b(The)17 b(ob)s(ject)f(will) 75 2200 y(b)q(e)g(deallo)q(cated)h(after)d(an)o(y)i(p)q(ending)h(comm)o (unication)f(that)e(uses)i(this)g(ob)s(ject)f(completes,)g(at)g(whic)o(h)75 2256 y(p)q(oin)o(t)21 b Ft(datat)o(yp)q(e)g Fy(b)q(ecomes)g(n)o(ull.)36 b(It)20 b(is)h(erroneous)f(to)f(use)i(a)f(datat)o(yp)q(e)f(as)h(argumen)o(t)f (to)h(an)g(MPI)75 2313 y(function)c(after)e(it)i(w)o(as)e(freed.)166 2369 y(Example:)75 2534 y Fr(INTEGER)23 b(type1,)g(type2)75 2591 y(CALL)g(MPI_TYPE_CONTIGUOUS\(5,)e(MPI_REAL,)i(type1,)g(ierr\))409 2647 y(!)h(new)f(type)h(object)f(created)75 2704 y(CALL)g (MPI_TYPE_COMMIT\(type1,)e(ierr\))1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 51 51 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(51)409 49 y Fr(!)24 b(now)f(type1)g(can)h(be)f(used)h(for)f (communication)75 106 y(type2)g(=)h(type1)409 162 y(!)g(type2)f(can)g(be)h (used)f(for)h(communication)409 219 y(!)g(\(it)f(is)h(a)g(handle)f(to)g(same) h(object)f(as)g(type1\))75 275 y(CALL)g(MPI_TYPE_VECTOR\(3,)f(5,)h(4,)h (MPI_REAL,)f(type1,)g(ierr\))409 332 y(!)h(new)f(uncommitted)g(type)g(object) g(created)75 388 y(CALL)g(MPI_TYPE_COMMIT\(type1,)e(ierr\))409 444 y(!)j(now)f(type1)g(can)h(be)f(used)h(anew)f(for)h(communication)166 541 y Fy(F)l(reeing)19 b(a)f(datat)o(yp)q(e)f(do)q(es)i(not)f(a\013ect)f(an)o (y)h(other)g(datat)o(yp)q(e)f(that)h(w)o(as)f(built)j(from)d(the)i(freed)75 598 y(datat)o(yp)q(e.)34 b(The)20 b(system)g(should)h(b)q(eha)o(v)o(e)g(as)e (if)i(a)f(separate)f(cop)o(y)h(of)g(the)g(datat)o(yp)q(e)g(w)o(as)f(created) 75 654 y(whenev)o(er)d(this)f(datat)o(yp)q(e)g(w)o(as)f(incorp)q(orated)i(in) g(a)f(new)g(deriv)o(ed)i(datat)o(yp)q(e.)166 787 y Fl(Implemen)o(tati)o(on)12 b(note:)166 837 y Fs(The)e(implemen)o(tati)o(on)d(ma)o(y)g(k)o(eep)j(a)f (reference)j(coun)o(t)e(of)f(activ)o(e)g(comm)o(unications)d(that)k(use)g (the)g(datat)o(yp)q(e,)75 886 y(in)15 b(order)i(to)f(decide)h(when)f(to)g (free)h(it.)24 b(Also,)15 b(one)h(ma)o(y)e(implemen)o(t)g(constructors)k(of)d (deriv)o(ed)h(datat)o(yp)q(es)h(so)75 936 y(that)e(they)h(k)o(eep)g(p)q(oin)o (ters)g(to)f(their)h(datat)o(yp)q(e)f(argumen)o(ts,)f(rather)j(then)e(cop)o (ying)g(them.)21 b(In)16 b(suc)o(h)g(case,)g(one)75 986 y(needs)e(to)e(k)o (eep)h(trac)o(k)f(of)g(activ)o(e)g(datat)o(yp)q(e)g(de\014nition)g (references)j(in)d(order)h(to)f(kno)o(w)g(when)h(a)f(datat)o(yp)q(e)g(ob)r (ject)75 1036 y(can)i(b)q(e)h(freed.)75 1239 y Fq(1.13.5)49 b(Use)16 b(of)h(general)f(datat)o(yp)q(es)g(in)h(comm)n(unication)75 1325 y Fy(Handles)g(to)f(deriv)o(ed)h(datat)o(yp)q(es)f(can)h(b)q(e)f(passed) h(to)f(a)g(comm)o(unication)h(call)g(wherev)o(er)f(a)g(datat)o(yp)q(e)75 1381 y(argumen)o(t)11 b(is)h(required.)19 b(A)12 b(call)g(of)f(the)h(form)e Ft(MPI)p 947 1381 14 2 v 16 w(SEND\(buf,)j(count,)g(datat)o(yp)q(e)g(,)f (...\))p Fy(,)e(where)i Ft(count)i Fi(>)75 1438 y Ft(1)p Fy(,)e(is)h(in)o (terpreted)g(as)g(if)g(the)f(call)i(w)o(as)d(passed)i(a)f(new)h(datat)o(yp)q (e)f(whic)o(h)h(is)g(the)g(concatenation)g(of)f Ft(count)75 1494 y Fy(copies)k(of)f Ft(datat)o(yp)q(e)p Fy(.)21 b(Th)o(us,)166 1551 y Ft(MPI)p 251 1551 V 16 w(SEND\(buf,)15 b(count,)i(datat)o(yp)q(e,)f (dest,)g(tag,)f(comm)m(\))d Fy(is)k(equiv)m(alen)o(t)h(to)75 1648 y Fr(MPI_TYPE_CONTIGUOUS\(count,)j(datatype,)j(newtype\))75 1704 y(MPI_TYPE_COMMIT\(newtype\))75 1761 y(MPI_SEND\(buf,)f(1,)i(newtype,)e (dest,)i(tag,)f(comm\))166 1858 y Fy(The)14 b(same)g(applies)h(to)f(all)h (other)e(comm)o(unication)i(functions)g(that)e(ha)o(v)o(e)g(a)h Ft(count)h Fy(and)f Ft(datat)o(yp)q(e)75 1914 y Fy(argumen)o(t.)166 1970 y(Supp)q(ose)k(that)e(a)g(a)h(send)g(op)q(eration)g Ft(MPI)p 920 1970 V 16 w(SEND\(buf,)h(count,)g(datat)o(yp)q(e.)26 b(dest,)18 b(tag,)f(comm)m(\))c Fy(is)75 2027 y(executed,)j(where)f Ft(datat)o(yp)q(e)i Fy(ha)o(v)o(e)e(t)o(yp)q(e)g(map)600 2130 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 2137 y Fj(0)743 2130 y Fi(;)8 b(disp)848 2137 y Fj(0)865 2130 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 2137 y Fg(n)p Fh(\000)p Fj(1)1128 2130 y Fi(;)g(disp)1233 2137 y Fg(n)p Fh(\000)p Fj(1)1297 2130 y Fy(\))p Fn(g)p Fi(;)166 2224 y Fy(with)20 b(exten)o(t)f Fi(extent)i Fy(\(empt)o(y)e(en)o(tries)g (deleted\).)34 b(Then)20 b(the)g(send)g(bu\013er)f(used)h(b)o(y)g(the)f(send) 75 2281 y(op)q(eration)c(consists)h(of)f Fi(n)10 b Fn(\001)g Ft(count)16 b Fy(en)o(tries,)f(where)h(en)o(try)f Fi(i)9 b Fn(\001)h Fi(n)h Fy(+)f Fi(j)17 b Fy(is)f(at)f(lo)q(cation)653 2384 y Fi(addr)746 2391 y Fg(i;j)796 2384 y Fy(=)e Ft(buf)h Fy(+)d Fi(extent)g Fn(\001)f Fi(i)g Fy(+)g Fi(disp)1280 2391 y Fg(j)166 2478 y Fy(and)15 b(has)h(t)o(yp)q(e)f Fi(ty)r(pe)522 2485 y Fg(j)540 2478 y Fy(,)g(for)f Fi(i)f Fy(=)g(0)p Fi(;)8 b(:::;)g Ft(count)h Fn(\000)h Fy(1)15 b(and)h Fi(j)f Fy(=)e(0)p Fi(;)8 b(:::;)g(n)f Fn(\000)k Fy(1.)20 b(These)15 b(en)o(tries)h(need)g(not) 75 2534 y(b)q(e)g(con)o(tiguous,)f(nor)g(distinct;)h(their)f(order)g(can)h(b) q(e)g(arbitrary)l(.)166 2591 y(The)g(v)m(ariable)h(stored)e(at)g(address)g Fi(addr)879 2598 y Fg(i;j)933 2591 y Fy(in)h(the)g(calling)h(program)e (should)h(b)q(e)h(of)e(a)g(t)o(yp)q(e)h(that)75 2647 y(matc)o(hes)d Fi(ty)r(pe)333 2654 y Fg(j)351 2647 y Fy(,)h(where)g(t)o(yp)q(e)g(matc)o (hing)g(is)g(de\014ned)i(as)d(in)i(section)f(1.5.)19 b(The)14 b(message)f(sen)o(t)h(con)o(tains)75 2704 y Fi(n)c Fn(\001)g Ft(count)17 b Fy(en)o(tries,)e(where)g(en)o(try)g Fi(i)10 b Fn(\001)g Fi(n)g Fy(+)g Fi(j)18 b Fy(has)d(t)o(yp)q(e)g Fi(ty)r(pe)1100 2711 y Fg(j)1118 2704 y Fy(.)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 52 52 bop 75 -100 a Fy(52)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fy(Similarly)l(,)f(supp)q(ose)f (that)e(a)h(receiv)o(e)h(op)q(eration)f Ft(MPI)p 1100 49 14 2 v 16 w(RECV\(buf,)h(count,)g(datat)o(yp)q(e,)h(source,)e(tag,)75 106 y(comm)m(,)f(status\))17 b Fy(is)e(executed,)h(where)g Ft(datat)o(yp)q(e)g Fy(has)g(t)o(yp)q(e)f(map)600 258 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 265 y Fj(0)743 258 y Fi(;)8 b(disp)848 265 y Fj(0)865 258 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 265 y Fg(n)p Fh(\000)p Fj(1)1128 258 y Fi(;)g(disp)1233 265 y Fg(n)p Fh(\000)p Fj(1)1297 258 y Fy(\))p Fn(g)p Fi(;)166 398 y Fy(with)k(exten)o(t)f Fi(extent)i Fy(\(empt)o(y)e(en)o(tries)i (deleted\).)19 b(Then)12 b(the)g(receiv)o(e)h(bu\013er)f(used)g(for)f(this)h (receiv)o(e)75 455 y(op)q(eration)j(consists)h(of)e(the)h Fi(n)c Fn(\001)e Ft(count)16 b Fy(en)o(tries,)f(where)h(en)o(try)e Fi(i)c Fn(\001)f Fi(n)i Fy(+)f Fi(j)17 b Fy(is)f(at)e(lo)q(cation)i Ft(buf)e Fy(+)c Fi(extent)h Fn(\001)75 511 y Fi(i)d Fy(+)h Fi(disp)227 518 y Fg(j)259 511 y Fy(and)15 b(has)f(t)o(yp)q(e)h Fi(ty)r(pe)613 518 y Fg(j)630 511 y Fy(.)20 b(If)15 b(the)f(incoming)i (message)e(consists)h(of)f Fi(k)i Fy(elemen)o(ts,)e(then)h(w)o(e)g(m)o(ust)75 567 y(ha)o(v)o(e)f Fi(k)g Fn(\024)f Fi(n)d Fn(\001)f Ft(count)p Fy(;)15 b(the)g Fi(i)9 b Fn(\001)g Fi(n)h Fy(+)f Fi(j)s Fy(-th)15 b(elemen)o(t)g(of)g(the)f(message)h(should)h(ha)o(v)o(e)e(a)g(t)o(yp)q(e)h (that)f(matc)o(hes)75 624 y Fi(ty)r(pe)159 631 y Fg(j)177 624 y Fy(.)166 690 y(Note)d(that)g(t)o(yp)q(e)h(matc)o(hing)f(is)h(de\014ned)h (according)f(to)f(the)h(t)o(yp)q(e)f(signature)h(of)f(the)h(corresp)q(onding) 75 746 y(datat)o(yp)q(es,)f(i.e.,)h(the)f(sequence)i(of)e(basic)h(t)o(yp)q(e) g(comp)q(onen)o(ts,)f(and)h(do)q(es)g(not)f(dep)q(end)i(on)e(the)g(datat)o (yp)q(e)75 803 y(de\014nition.)166 869 y(Example:)75 1011 y Fr(...)75 1067 y(CALL)23 b(MPI_TYPE_CONTIGUOUS\()e(2,)j(MPI_REAL,)f(type2,)g (...\))75 1124 y(CALL)g(MPI_TYPE_CONTIGUOUS\()e(4,)j(MPI_REAL,)f(type4,)g (...\))75 1180 y(CALL)g(MPI_TYPE_CONTIGUOUS\()e(2,)j(type2,)f(type22,)g (...\))75 1237 y(...)75 1293 y(CALL)g(MPI_SEND\()g(a,)h(4,)f(MPI_REAL,)g (...\))75 1349 y(CALL)g(MPI_SEND\()g(a,)h(2,)f(type2,)g(...\))75 1406 y(CALL)g(MPI_SEND\()g(a,)h(1,)f(type22,)g(...\))75 1462 y(CALL)g(MPI_SEND\()g(a,)h(1,)f(type4,)g(...\))75 1519 y(...)75 1575 y(CALL)g(MPI_RECV\()g(a,)h(4,)f(MPI_REAL,)g(...\))75 1632 y(CALL)g(MPI_RECV\()g(a,)h(2,)f(type2,)g(...\))75 1688 y(CALL)g(MPI_RECV\()g (a,)h(1,)f(type22,)g(...\))75 1745 y(CALL)g(MPI_RECV\()g(a,)h(1,)f(type4,)g (...\))166 1877 y Fy(F)l(rom)14 b(a)h(t)o(yp)q(e)h(matc)o(hing)f(viewp)q(oin) o(t)h(an)o(y)f(of)g(the)g(sends)h(legally)g(matc)o(hes)f(an)o(y)g(of)g(the)g (receiv)o(es.)166 1943 y(A)22 b(datat)o(yp)q(e)f(ma)o(y)g(sp)q(ecify)i(o)o(v) o(erlapping)g(en)o(tries.)40 b(If)22 b(suc)o(h)h(datat)o(yp)q(e)e(is)h(used)h (in)f(a)g(receiv)o(e)75 1999 y(op)q(eration,)17 b(i.e.,)f(if)h(the)f(some)g (part)g(of)g(the)g(receiv)o(e)i(bu\013er)e(is)h(o)o(v)o(erwritten)f(more)g (than)g(once)h(b)o(y)f(the)75 2056 y(receiv)o(e)g(op)q(eration,)f(then)h(the) f(outcome)g(of)g(the)g(receiv)o(e)h(is)g(unde\014ned.)166 2122 y(Supp)q(ose,)j(again,)f(that)f Ft(MPI)p 683 2122 V 15 w(RECV\(buf,)j(count,) f(datat)o(yp)q(e,)g(dest,)g(tag,)f(comm)m(,)d(status\))k Fy(is)f(exe-)75 2178 y(cuted,)d(where)h Ft(datat)o(yp)q(e)h Fy(has)e(t)o(yp)q(e)g(map)600 2328 y Fn(f)p Fy(\()p Fi(ty)r(pe)725 2335 y Fj(0)743 2328 y Fi(;)8 b(disp)848 2335 y Fj(0)865 2328 y Fy(\))p Fi(;)g(:::;)g Fy(\()p Fi(ty)r(pe)1066 2335 y Fg(n)p Fh(\000)p Fj(1)1128 2328 y Fi(;)g(disp)1233 2335 y Fg(n)p Fh(\000)p Fj(1)1297 2328 y Fy(\))p Fn(g)p Fi(:)166 2468 y Fy(The)k(receiv)o(ed)g(message)f(need)h(not)f (\014ll)i(all)g(the)e(receiv)o(e)h(bu\013er,)g(nor)f(do)q(es)h(it)g(need)g (to)f(\014ll)i(a)e(n)o(um)o(b)q(er)75 2525 y(of)i(lo)q(cations)h(whic)o(h)g (is)f(a)g(m)o(ultiple)i(of)e Fi(n)p Fy(;)h(an)o(y)f(n)o(um)o(b)q(er)g Fi(k)i Fy(of)d(basic)i(elemen)o(ts)g(can)f(b)q(e)h(receiv)o(ed,)h(where)75 2581 y(0)d Fn(\024)h Fi(k)h Fn(\024)f Ft(count)e Fn(\001)f Fi(n)p Fy(.)166 2647 y(The)k(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i(receiv)o (ed)g(can)f(b)q(e)g(retriev)o(ed)g(from)f Ft(status)j Fy(using)e(the)g(query) g(function)75 2704 y Ft(MPI)p 160 2704 V 16 w(GET)p 264 2704 V 17 w(ELEMENTS)p Fy(:)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 53 53 bop 75 -100 a Fu(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b Fy(53)75 49 y Ft(MPI)p 160 49 14 2 v 16 w(GET)p 264 49 V 17 w(ELEMENTS\()15 b(status,)h(datat)o(yp)q(e,)h(count\))117 127 y Fs(IN)155 b Ft(status)476 b Fs(return)15 b(status)g(of)e(receiv)o(e)i (op)q(eration)f(\(Status\))117 202 y(IN)155 b Ft(datat)o(yp)q(e)424 b Fs(datat)o(yp)q(e)14 b(used)h(b)o(y)f(receiv)o(e)h(op)q(eration)f (\(handle\))117 278 y(OUT)108 b Ft(count)482 b Fs(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h(basic)f(elemen)o(ts)g(\(in)o(teger\))75 402 y Fr(int)23 b(MPI)p 245 402 15 2 v 17 w(Get)p 334 402 V 17 w(elements\(MPI)p 639 402 V 16 w(Status)g(status,)f(MPI)p 1084 402 V 17 w(Datatype)h(datatype,)g(int)g(*count\))75 489 y(MPI)p 150 489 V 17 w(GET)p 239 489 V 17 w(ELEMENTS\(STATUS,)e(DATATYPE,)i (COUNT,)g(IERROR\))170 545 y(INTEGER)g(STATUS\(MPI)p 604 545 V 16 w(STATUS)p 764 545 V 17 w(SIZE\),)g(DATATYPE,)f(COUNT,)h(IERROR)166 632 y Fy(The)13 b(previously)h(used)f(function)h Ft(MPI)p 834 632 14 2 v 16 w(GET)p 938 632 V 16 w(COUNT)g Fy(has)f(a)f(di\013eren)o(t)h(b) q(eha)o(vior:)19 b(it)13 b(returns)g(the)75 688 y(n)o(um)o(b)q(er)g(of)f (\\top)g(elemen)o(ts")h(receiv)o(ed.)21 b(In)13 b(the)g(previous)g(example,)h Ft(MPI)p 1368 688 V 16 w(GET)p 1472 688 V 16 w(COUNT)g Fy(ma)o(y)e(return)75 745 y(an)o(y)i(in)o(teger)h(v)m(alue)g Fi(k)q Fy(,)g(where)f(0)f Fn(\024)g Fi(k)g Fn(\024)g Ft(count)r Fy(.)19 b(If)c Ft(MPI)p 1047 745 V 16 w(GET)p 1151 745 V 16 w(COUNT)h Fy(returns)e Fi(k)q Fy(,)g(then)h(the)g(n)o(um)o(b)q(er)75 801 y(of)f(basic)h(elemen)o(ts) f(receiv)o(ed)i(\(and)e(the)g(v)m(alue)h(returned)g(b)o(y)f Ft(MPI)p 1226 801 V 16 w(GET)p 1330 801 V 16 w(ELEMENTS)p Fy(\))g(is)h Fi(n)8 b Fn(\001)g Fi(k)q Fy(.)19 b(If)c(the)75 858 y(n)o(um)o(b)q(er)j(of)f (basic)h(elemen)o(ts)g(receiv)o(ed)h(is)f(not)f(a)h(m)o(ultiple)h(of)e Fi(n)p Fy(,)h(i.e.)27 b(if)18 b(the)g(receiv)o(e)g(op)q(eration)g(has)75 914 y(not)f(receiv)o(ed)i(an)f(in)o(tegral)g(n)o(um)o(b)q(er)g(of)f Ft(datat)o(yp)q(e)j Fy(\\copies",)e(then)g Ft(MPI)p 1360 914 V 16 w(GET)p 1464 914 V 17 w(COUNT)g Fy(returns)g(the)75 971 y(v)m(alue)e Fo(MPI)p 269 971 13 2 v 15 w(UNDEFINED)p Fy(.)166 1027 y(Example:)75 1134 y Fr(...)75 1191 y(CALL)23 b(MPI_TYPE_CONTIGUOUS\(2,) e(MPI_REAL,)i(Type2,)g(ierr\))75 1247 y(CALL)g(MPI_COMMIT\(Type2,)f(ierr\))75 1304 y(...)75 1360 y(CALL)h(MPI_RANK\(comm,)f(rank,)h(ierr\))75 1417 y(IF\(rank.EQ.0\))147 1473 y(THEN)218 1530 y(CALL)g(MPI_SEND\(a,)g(2,)g (MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))218 1586 y(CALL)g(MPI_SEND\(a,)g(3,)g (MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))147 1643 y(ELSE)218 1699 y(CALL)g(MPI_RECV\(a,)g(2,)g(Type2,)g(0,)h(0,)g(comm,)f(stat,)g(ierr\))218 1755 y(CALL)g(MPI_GET_COUNT\(stat,)f(Type2,)h(i,)g(ierr\))119 b(!)24 b(returns)f(i=1)218 1812 y(CALL)g(MPI_GET_ELEMENTS\(stat,)e(Type2,)i (i,)h(ierr\))47 b(!)24 b(returns)f(i=2)218 1868 y(CALL)g(MPI_RECV\(a,)g(2,)g (Type2,)g(0,)h(0,)g(comm,)f(stat,)g(ierr\))218 1925 y(CALL)g (MPI_GET_COUNT\(stat,)f(Type2,)h(i,)g(ierr\))119 b(!)24 b(returns)f (i=MPI_UNDEFINED)218 1981 y(CALL)g(MPI_GET_ELEMENTS\(stat,)e(Type2,)i(i,)h (ierr\))47 b(!)24 b(returns)f(i=3)75 2038 y(END)g(IF)166 2145 y Fy(The)15 b(function)h Ft(MPI)p 522 2145 14 2 v 16 w(GET)p 626 2145 V 17 w(ELEMENTS)f Fy(can)g(also)g(b)q(e)g(used)h(after)e(a)h(prob)q (e)g(to)g(\014nd)h(the)f(n)o(um)o(b)q(er)75 2201 y(of)20 b(elemen)o(ts)h(in)g (the)f(prob)q(ed)h(message.)34 b(Note)20 b(that)f(the)h(t)o(w)o(o)f (functions)i Ft(MPI)p 1507 2201 V 16 w(GET)p 1611 2201 V 17 w(COUNT)g Fy(and)75 2258 y Ft(MPI)p 160 2258 V 16 w(GET)p 264 2258 V 17 w(ELEMENTS)15 b Fy(return)g(the)g(same)g(v)m(alues)i(when)e(they)h (are)e(used)i(with)g(basic)g(datat)o(yp)q(es.)189 2365 y Fp(R)n(ationale.)38 b Fy(The)14 b(extension)g(giv)o(en)g(to)f(the)h(de\014nition)h(of)e Ft(MPI)p 1303 2365 V 16 w(GET)p 1407 2365 V 16 w(COUNT)i Fy(seems)e(natural:) 189 2421 y(one)j(w)o(ould)h(exp)q(ect)h(this)f(function)g(to)f(return)h(the)f (v)m(alue)i(of)e(the)h Ft(count)h Fy(argumen)o(t,)e(when)h(the)189 2478 y(receiv)o(e)e(bu\013er)f(is)h(\014lled.)22 b(Often)15 b(times)f Ft(datat)o(yp)q(e)i Fy(represen)o(ts)f(a)f(basic)h(unit)g(of)f (data)g(one)g(w)o(an)o(ts)189 2534 y(to)19 b(transfer;)j(e.g.,)e(a)g(record)g (in)h(an)f(arra)o(y)f(of)h(records)g(\(structures\).)34 b(One)21 b(should)g(b)q(e)g(able)189 2591 y(to)f(\014nd)i(out)f(ho)o(w)f(man)o(y)h (comp)q(onen)o(ts)g(where)g(receiv)o(ed)i(without)e(b)q(othering)h(to)e (divide)j(b)o(y)189 2647 y(the)18 b(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(in)f (eac)o(h)g(comp)q(onen)o(t.)30 b(Ho)o(w)o(ev)o(er,)18 b(on)h(other)f(o)q (ccasions,)h Ft(datat)o(yp)q(e)189 2704 y Fy(is)h(used)h(to)f(de\014ned)h(a)f (complex)h(la)o(y)o(out)f(of)g(data)f(in)i(the)g(receiv)o(er)g(memory)l(,)f (and)h(do)q(es)f(not)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 54 54 bop 75 -100 a Fy(54)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)189 49 y Fy(represen)o(t)22 b(a)g(basic)h(unit)g(of)f(data)g(for)f(transfers.)41 b(In)23 b(suc)o(h)f(cases,)i(one)e(needs)i(to)d(use)i(the)189 106 y(function)16 b Ft(MPI)p 452 106 14 2 v 15 w(GET)p 555 106 V 17 w(ELEMENTS)p Fy(.)f(\()p Fp(End)h(of)g(r)n(ationale.)p Fy(\))166 300 y Fl(Implemen)o(tati) o(on)i(note:)58 b Fs(The)19 b(curren)o(t)i(de\014nition)e(implies)e(that)i(a) g(receiv)o(e)h(cannot)g(c)o(hange)f(the)75 350 y(v)n(alue)d(of)g(storage)h(b) q(et)o(w)o(een)h(the)f(en)o(tries)g(de\014ned)h(to)e(comp)q(ose)g(the)h(comm) o(unication)c(bu\013er.)27 b(In)17 b(particular,)75 400 y(the)c(de\014nition) g(implies)e(that)i(padding)f(space)i(in)e(a)h(structure)i(should)d(not)h(b)q (e)g(mo)q(di\014ed)f(when)h(suc)o(h)h(structure)75 450 y(is)f(copied)g(from)f (one)h(pro)q(cess)i(to)e(another.)18 b(This)13 b(w)o(ould)f(prev)o(en)o(ts)j (the)f(ob)o(vious)e(optimization)e(of)j(cop)o(ying)f(the)75 500 y(structure,)i(together)f(with)e(the)i(padding,)e(as)h(one)g(con)o (tiguous)f(blo)q(c)o(k.)18 b(The)12 b(implemen)o(tatio)o(n)d(is)j(free)h(to)e (do)h(this)75 549 y(optimization)h(when)j(it)f(do)q(es)i(not)e(impact)f(the)i (outcome)f(of)g(the)h(computation.)22 b(The)16 b(user)g(can)g(\\force")g (this)75 599 y(optimization)11 b(b)o(y)j(explicitly)f(including)g(padding)g (as)h(part)g(of)f(the)h(message.)75 818 y Fq(1.13.6)49 b(Co)o(rrect)15 b(use)h(of)g(addresses)75 908 y Fy(Successiv)o(ely)23 b(declared)f(v)m (ariables)h(in)e(C)g(or)g(F)l(ortran)e(are)i(not)g(necessarily)h(stored)f(at) f(con)o(tiguous)75 965 y(lo)q(cations.)i(Th)o(us,)16 b(care)g(m)o(ust)f(b)q (e)h(exercised)i(that)d(displacemen)o(ts)i(do)f(not)f(cross)h(from)f(one)h(v) m(ariable)75 1021 y(to)e(another.)20 b(Also,)15 b(in)g(mac)o(hines)h(with)f (segmen)o(ted)g(address)g(space,)g(addresses)g(are)g(not)f(unique)j(and)75 1078 y(address)11 b(arithmetic)h(has)g(some)f(p)q(eculiar)i(prop)q(erties.)19 b(Th)o(us,)12 b(use)g(of)f Fx(addresses)p Fy(,)f(i.e.)20 b(displacemen)o(ts) 75 1134 y(relativ)o(e)c(to)e(the)i(start)e(address)29 b Fo(MPI)p 733 1134 13 2 v 14 w(BOTTOM)p Fy(,)14 b(has)h(to)g(b)q(e)h(restricted.)166 1193 y(V)l(ariables)g(b)q(elong)f(to)f(the)g(same)g Fx(sequen)o(tial)k (storage)c Fy(if)h(they)f(b)q(elong)i(to)e(the)g(same)g(arra)o(y)l(,)g(to)75 1250 y(the)k(same)g Ft(COMMON)g Fy(blo)q(c)o(k)h(in)g(F)l(ortran,)e(or)h(to)f (the)h(same)g(structure)f(in)i(C.)f(V)l(alid)h(addresses)f(are)75 1306 y(de\014ned)f(recursiv)o(ely)f(as)f(follo)o(ws:)131 1425 y(1.)22 b(The)g(function)g Ft(MPI)p 558 1425 14 2 v 16 w(ADDRESS)h Fy(returns)e(a)h(v)m(alid)h(address,)h(when)e(passed)g(as)f(argumen)o(t)g(a) 189 1481 y(v)m(ariable)16 b(of)f(the)g(calling)i(program.)131 1585 y(2.)22 b(The)d Ft(buf)h Fy(argumen)o(t)f(of)g(a)g(comm)o(unication)h (function)g(ev)m(aluates)h(to)d(a)h(v)m(alid)i(address,)f(when)189 1641 y(passed)15 b(as)g(argumen)o(t)f(a)h(v)m(ariable)i(of)e(the)g(calling)i (program.)131 1745 y(3.)22 b(If)15 b Ft(v)g Fy(is)h(a)f(v)m(alid)i(address,)d (and)i Ft(i)f Fy(is)g(an)h(in)o(teger,)f(then)g Ft(v+i)g Fy(is)h(a)f(v)m (alid)i(address,)e(pro)o(vided)h Ft(v)f Fy(and)189 1801 y Ft(v+i)g Fy(are)g(in)h(the)f(same)g(sequen)o(tial)h(storage.)131 1905 y(4.)22 b(If)15 b Ft(v)g Fy(is)h(a)f(v)m(alid)i(address)e(then)29 b Fo(MPI)p 820 1905 13 2 v 15 w(BOTTOM)13 b(+)h(v)h Fy(is)h(a)f(v)m(alid)i (address.)166 2024 y(A)22 b(correct)g(program)g(uses)g(only)h(v)m(alid)i (addresses)d(to)g(iden)o(tify)i(the)e(lo)q(cations)h(of)f(en)o(tries)h(in)75 2080 y(comm)o(unication)16 b(bu\013ers.)k(F)l(urthermore,)14 b(if)h Ft(u)h Fy(and)f Ft(v)g Fy(are)g(t)o(w)o(o)f(v)m(alid)j(addresses,)e (then)g(the)g(\(in)o(teger\))75 2137 y(di\013erence)g Ft(u)f(-)g(v)g Fy(can)g(b)q(e)g(computed)g(only)h(if)f(b)q(oth)g Ft(u)g Fy(and)g Ft(v)g Fy(are)f(in)i(the)f(same)f(sequen)o(tial)i(storage;)e(no)75 2193 y(other)i(arithmetic)h(op)q(erations)f(can)g(b)q(e)h(meaningfully)h (executed)g(on)e(addresses.)166 2252 y(The)41 b(ab)q(o)o(v)o(e)20 b(rules)h(imp)q(ose)g(no)f(constrain)o(ts)g(in)h(the)f(use)h(of)f(deriv)o(ed) h(datat)o(yp)q(es,)g(as)f(long)g(as)-1903 b Fm(\))75 2308 y Fy(they)18 b(are)f(used)h(to)f(de\014ne)h(a)g(comm)o(unication)g(bu\013er)f (that)g(is)h(wholly)h(con)o(tained)f(within)h(the)e(same)75 2365 y(sequen)o(tial)23 b(storage.)38 b(Ho)o(w)o(ev)o(er,)22 b(the)g(construction)g(of)f(a)h(comm)o(unication)g(bu\013er)g(that)f(con)o (tains)75 2421 y(v)m(ariables)f(that)f(are)g(not)g(within)h(the)f(same)g (sequen)o(tial)h(storage)e(m)o(ust)h(ob)q(ey)g(certain)h(restrictions.)75 2478 y(Basically)l(,)g(a)e(comm)o(unication)h(bu\013er)f(with)g(v)m(ariables) i(that)d(are)h(not)g(within)h(the)f(same)g(sequen)o(tial)75 2534 y(storage)g(can)h(b)q(e)h(used)g(only)g(b)o(y)f(sp)q(ecifying)i(in)f (the)g(comm)o(unication)g(call)g Ft(buf)g(=)g(MPI)p 1651 2534 14 2 v 16 w(BOTTOM)p Fy(,)75 2591 y Ft(count)i(=)g(1)p Fy(,)g(and)f(using)h (a)f Ft(datat)o(yp)q(e)i Fy(argumen)o(t)d(where)i(all)g(displacemen)o(ts)h (are)d(v)m(alid)j(\(absolute\))75 2647 y(addresses.)d(A)15 b(datat)o(yp)q(e)f(with)h(displacemen)o(ts)h(that)e(are)h(absolute)g (addresses)g(not)f(within)i(the)f(same)75 2704 y(sequen)o(tial)h(storage)e (can)i(only)f(b)q(e)h(constructed)g(using)g Ft(MPI)p 1139 2704 V 15 w(TYPE)p 1271 2704 V 17 w(STRUCT)p Fy(.)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 55 55 bop 75 -100 a Fu(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b Fy(55)189 49 y Fp(A)n(dvic)n(e)15 b(to)i(users.)39 b Fy(It)16 b(is)f(not)g(exp)q(ected)i(that)d(MPI)h(implemen)o(tations)i(will)f(b)q(e)g (able)g(to)f(detect)189 106 y(erroneous,)22 b(\\out)f(of)g(b)q(ound")h (displacemen)o(ts)h({)f(unless)g(those)f(o)o(v)o(er\015o)o(w)g(the)g(user)h (address)189 162 y(space)15 b({)h(since)g(the)g(MPI)g(call)g(ma)o(y)f(not)g (kno)o(w)g(the)h(exten)o(t)f(of)g(the)h(arra)o(ys)e(and)i(records)f(in)i(the) 189 219 y(host)d(program.)19 b(\()p Fp(End)d(of)g(advic)n(e)g(to)h(users.)p Fy(\))166 412 y Fl(Implemen)o(tati)o(on)d(note:)41 b Fs(There)16 b(is)f(no)g(need)h(to)f(distinguish)g(\(absolute\))h(addresses)h(and)e (\(relativ)o(e\))75 468 y(displacemen)o(ts)h(on)h(a)g(mac)o(hine)e(with)i (con)o(tiguous)f(address)i(space:)25 b Fo(MPI)p 1274 468 13 2 v 15 w(BOTTOM)16 b Fs(is)h(zero,)h(and)e(b)q(oth)h(ad-)75 524 y(dresses)d(and)e(displacemen)o(ts)g(are)g(in)o(tegers.)18 b(On)13 b(mac)o(hines)e(where)i(the)f(distinction)g(is)g(required,)g (addresses)j(are)75 581 y(recognized)g(as)f(expressions)h(that)f(in)o(v)o (olv)o(e)f Fo(MPI)p 852 581 V 14 w(BOTTOM)p Fs(.)75 790 y Fq(1.13.7)49 b(Examples)75 877 y Fy(The)15 b(follo)o(wing)h(examples)g(illustrate)h(the)e (use)g(of)g(deriv)o(ed)h(datat)o(yp)q(es.)75 1002 y Ft(First)f(example:)43 b Fy(Send)16 b(and)g(receiv)o(e)g(a)e(section)i(of)f(a)g(3D)g(arra)o(y)l(.) 218 1113 y Fr(REAL)23 b(a\(100,100,100\),)f(e\(9,9,9\))218 1169 y(INTEGER)h(oneslice,)g(twoslice,)f(threeslice,)h(sizeofreal,)f(myrank,) h(ierr)218 1226 y(MPI_STATUS)g(status)75 1339 y(C)143 b(extract)23 b(the)g(section)g(a\(1:17:2,)g(3:11,)g(2:10\))75 1395 y(C)143 b(and)23 b(store)h(it)f(in)h(e\(*,*,*\).)218 1508 y(CALL)f (MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))218 1621 y(CALL)j (MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 1734 y(C)119 b(create)23 b(datatype)g(for)g(a)h(1D)g(section)218 1790 y(CALL)f(MPI_TYPE_VECTOR\()f(9,)i(1,)f(2,)h(MPI_REAL,)f(oneslice,)f (ierr\))75 1903 y(C)119 b(create)23 b(datatype)g(for)g(a)h(2D)g(section)218 1960 y(CALL)f(MPI_TYPE_HVECTOR\(9,)f(1,)h(100*sizeofreal,)f(oneslice,)h (twoslice,ierr\))75 2073 y(C)119 b(create)23 b(datatype)g(for)g(the)h(entire) f(section)218 2129 y(CALL)g(MPI_TYPE_HVECTOR\()f(9,)i(1,)f (100*100*sizeofreal,)e(twoslice,)i(1,)767 2186 y(threeslice,)f(ierr\))218 2299 y(CALL)h(MPI_TYPE_COMMIT\()f(threeslice,)h(ierr\))218 2355 y(CALL)g(MPI_SENDRECV\(a\(1,3,2\),)e(1,)j(threeslice,)e(myrank,)h(0,)h (e,)f(9*9*9,)648 2411 y(MPI_REAL,)f(myrank,)h(0,)h(MPI_COMM_WORLD,)e(status,) h(ierr\))75 2536 y Ft(Second)17 b(example:)42 b Fy(Cop)o(y)15 b(the)g(\(strictly\))g(lo)o(w)o(er)g(triangular)g(part)g(of)g(a)g(matrix.)218 2647 y Fr(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 2704 y(INTEGER)47 b(disp\(100\),)22 b(blocklen\(100\),)g(ltype,)h(myrank,)g(ierr)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 56 56 bop 75 -100 a Fy(56)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)218 49 y Fr(MPI_STATUS)23 b(status)75 162 y(C)119 b(copy)23 b(lower)h(triangular)e(part)h(of)h(array)f(a)75 219 y(C)119 b(onto)23 b(lower)h(triangular)e(part)h(of)h(array)f(b)218 332 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))75 444 y(C)119 b(compute)23 b(start)g(and)h(size)f(of)h(each)f(column)218 501 y(DO)h(i=1,)f(100)266 557 y(disp\(i\))g(=)h(100*\(i-1\))e(+)i(i)266 614 y(block\(i\))f(=)g(100-i)218 670 y(END)h(DO)75 783 y(C)119 b(create)23 b(datatype)g(for)g(lower)h(triangular)e(part)218 840 y(CALL)h(MPI_TYPE_INDEX\()f(100,)i(block,)f(disp,)g(MPI_REAL,)f(ltype,)i (ierr\))218 953 y(CALL)f(MPI_TYPE_COMMIT\(ltype,)e(ierr\))218 1009 y(CALL)i(MPI_SENDRECV\()g(a,)g(1,)h(ltype,)f(myrank,)g(0,)g(b,)h(1,)552 1065 y(ltype,)f(myrank,)g(0,)h(MPI_COMM_WORLD,)e(status,)h(ierr\))75 1192 y Ft(Third)15 b(example:)43 b Fy(T)l(ransp)q(ose)15 b(a)g(matrix)218 1306 y Fr(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 1362 y(INTEGER)g(row,)g (xpose,)g(sizeofreal,)g(myrank,)g(ierr)218 1419 y(MPI_STATUS)g(status)75 1532 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 1645 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))218 1758 y(CALL)j(MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 1870 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row)218 1927 y(CALL)f(MPI_TYPE_VECTOR\()f(100,)h(1,)h(100,)f(MPI_REAL,)g(row,)g(ierr\))75 2040 y(C)119 b(create)23 b(datatype)g(for)g(matrix)g(in)h(row-major)f(order) 218 2096 y(CALL)g(MPI_TYPE_HVECTOR\()f(100,)h(1,)h(sizeofreal,)e(row,)i (xpose,)f(ierr\))218 2209 y(CALL)g(MPI_TYPE_COMMIT\()f(xpose,)h(ierr\))75 2322 y(C)119 b(send)23 b(matrix)h(in)f(row-major)g(order)g(and)g(receive)g (in)h(column)f(major)g(order)218 2379 y(CALL)g(MPI_SENDRECV\()g(a,)g(1,)h (xpose,)f(myrank,)g(0,)g(b,)h(100*100,)457 2435 y(MPI_REAL,)e(myrank,)h(0,)h (MPI_COMM_WORLD,)e(status,)h(ierr\))166 2547 y Fy(Another)15 b(approac)o(h)g(to)f(the)i(transp)q(ose)f(problem:)218 2647 y Fr(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 2704 y(INTEGER)47 b(disp\(2\),)23 b(blocklen\(2\),)f(type\(2\),)h(row,)g(row1,)g(sizeofreal) 1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 57 57 bop 75 -100 a Fu(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b Fy(57)218 49 y Fr(INTEGER)47 b(myrank,)23 b(ierr)218 106 y(MPI_STATUS)g(status)218 219 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d (myrank\))75 332 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 444 y(CALL)g(MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 557 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row)218 614 y(CALL)f(MPI_TYPE_VECTOR\()f(100,)h(1,)h(100,)f(MPI_REAL,)g(row,)g(ierr\))75 727 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row,)f(with)h(the)f(extent)g (of)h(one)f(real)g(number)218 783 y(disp\(1\))g(=)h(0)218 840 y(disp\(2\))f(=)h(sizeofreal)218 896 y(type\(1\))47 b(=)24 b(row)218 953 y(type\(2\))47 b(=)24 b(MPI_UB)218 1009 y(blocklen\(1\))46 b(=)24 b(1)218 1065 y(blocklen\(2\))46 b(=)24 b(1)218 1122 y(CALL)f(MPI_TYPE_STRUCT\()f(2,)i(blocklen,)e(disp,)i(type,)f(row1,)g(ierr\)) 218 1235 y(CALL)g(MPI_TYPE_COMMIT\()f(row1,)h(ierr\))75 1348 y(C)119 b(send)23 b(100)h(rows)f(and)h(receive)f(in)g(column)g(major)g(order) 218 1404 y(CALL)g(MPI_SENDRECV\()g(a,)g(100,)g(row1,)h(myrank,)f(0,)g(b,)h (100*100,)457 1461 y(MPI_REAL,)e(myrank,)h(0,)h(MPI_COMM_WORLD,)e(status,)h (ierr\))75 1581 y Ft(F)o(ourth)15 b(example:)43 b Fy(manipulate)17 b(an)e(arra)o(y)f(of)g(structures.)75 1687 y Fr(struct)23 b(Partstruct)147 1744 y({)147 1800 y(int)95 b(class;)47 b(/*)23 b(particle)g(class)g(*/)147 1857 y(double)g(d[6];)71 b(/*)23 b(particle)g(coordinates)f(*/)147 1913 y(char)71 b(b[7];)g(/*)23 b(some)g(additional)g(information)f(*/)147 1970 y(};)75 2083 y(struct)h(Partstruct)94 b(particle[1000];)75 2195 y(int)429 b(i,)24 b(dest,)f(rank;)75 2252 y(MPI_Comm)118 b(comm;)75 2421 y(/*)24 b(build)f(datatype)g(describing)f(structure)h(*/)75 2534 y(MPI_Datatype)f(Particletype;)75 2591 y(MPI_Datatype)g(type[3])h(=)h ({MPI_INT,)e(MPI_DOUBLE,)h(MPI_CHAR};)75 2647 y(int)238 b(blocklen[3])23 b(=)g({1,)h(6,)f(7};)75 2704 y(MPI_Aint)118 b(disp[3];)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 58 58 bop 75 -100 a Fy(58)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 162 y Fr(/*)24 b(compute)e(displacements)h (of)g(structure)g(components)f(*/)75 275 y(MPI_Address\()g(particle,)h (disp\);)75 332 y(MPI_Address\()f(particle[0].d,)g(disp+1\);)75 388 y(MPI_Address\()g(particle[0].b,)g(disp+2\);)75 444 y(for)h(\(i=0;)h(i)f (<3;)h(i++\))f(disp[i])g(-=)h(disp[0];)75 557 y(MPI_Type_struct\()e(3,)h (blocklen,)g(disp,)g(type,)g(&Particletype\);)147 670 y(/*)g(If)h(compiler)f (does)g(padding)g(in)g(mysterious)g(ways,)147 727 y(the)g(following)g(may)g (be)h(safer)f(*/)75 840 y(MPI_Datatype)f(type1[4])h(=)h({MPI_INT,)e (MPI_DOUBLE,)h(MPI_CHAR,)f(MPI_UB};)75 896 y(int)238 b(blocklen1[4])22 b(=)i({1,)g(6,)f(7,)h(1};)75 953 y(MPI_Aint)118 b(disp1[4];)75 1065 y(/*)24 b(compute)e(displacements)h(of)g(structure)g(components)f(*/)75 1178 y(MPI_Address\()g(particle,)h(disp1\);)75 1235 y(MPI_Address\()f (particle[0].d,)g(disp1+1\);)75 1291 y(MPI_Address\()g(particle[0].b,)g (disp1+2\);)75 1348 y(MPI_Address\()g(particle+1,)h(disp1+3\);)75 1404 y(for)g(\(i=0;)h(i)f(<4;)h(i++\))f(disp1[i])g(-=)g(disp1[0];)75 1517 y(/*)h(build)f(datatype)g(describing)f(structure)h(*/)75 1630 y(MPI_Type_struct\()f(4,)h(blocklen1,)g(disp1,)g(type1,)g (&Particletype\);)409 1799 y(/*)h(4.1:)266 1856 y(send)f(the)h(entire)f (array)g(*/)75 1969 y(MPI_Type_commit\()f(&Particletype\);)75 2025 y(MPI_Send\()h(particle,)f(1000,)h(Particletype,)g(dest,)g(tag,)g (comm\);)409 2195 y(/*)h(4.2:)266 2251 y(send)f(only)h(the)f(entries)g(of)g (class)h(zero)f(particles,)266 2307 y(preceded)g(by)g(the)h(number)f(of)g (such)h(entries)f(*/)75 2420 y(MPI_Datatype)f(Zparticles;)70 b(/*)24 b(datatype)f(describing)f(all)i(particles)791 2477 y(with)f(class)g(zero)h(\(needs)f(to)g(be)h(recomputed)791 2533 y(if)f(classes)g(change\))g(*/)75 2590 y(MPI_Datatype)f(Ztype;)75 2703 y(MPI_Aint)118 b(zdisp[1000];)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 59 59 bop 75 -100 a Fu(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b Fy(59)75 49 y Fr(int)23 b(zblock[1000],)g(j,)g(k;)75 106 y(int)g(zzblock[2])g(=)h({1,1};)75 162 y(MPI_Aint)118 b(zzdisp[2];)75 219 y(MPI_Datatype)22 b(zztype[2];)75 332 y(/*)i(compute)e(displacements)h (of)g(class)g(zero)h(particles)e(*/)75 388 y(j)i(=)f(0;)75 444 y(for\(i=0;)g(i)g(<)h(1000;)f(i++\))123 501 y(if)g (\(particle[i].class==0\))194 557 y({)194 614 y(zdisp[j])g(=)h(i;)194 670 y(zblock[j])f(=)h(1;)194 727 y(j++;)194 783 y(})75 896 y(/*)g(create)f(datatype)f(for)i(class)f(zero)g(particles)47 b(*/)75 953 y(MPI_Type_indexed\()22 b(j,)h(zblock,)g(zdisp,)g(Particletype,)f (&Zparticles\);)75 1065 y(/*)i(prepend)e(particle)h(count)g(*/)75 1122 y(MPI_Address\(&j,)f(zzdisp\);)75 1178 y(MPI_Address\(particle,)f (zzdisp+1\);)75 1235 y(zztype[0])i(=)g(MPI_INT;)75 1291 y(zztype[1])g(=)g (Zparticles;)75 1348 y(MPI_Type_struct\(2,)e(zzblock,)i(zzdisp,)g(zztype,)g (&Ztype\);)75 1461 y(MPI_Type_commit\()f(&Ztype\);)75 1517 y(MPI_Send\()h(MPI_BOTTOM,)f(1,)i(Ztype,)f(dest,)g(tag,)g(comm\);)242 1686 y(/*)h(A)f(probably)g(more)g(efficient)g(way)h(of)f(defining)g (Zparticles)f(*/)75 1799 y(/*)i(consecutive)e(particles)h(with)g(index)g (zero)g(are)h(handled)f(as)g(one)h(block)f(*/)75 1856 y(j=0;)75 1912 y(for)g(\(i=0;)h(i)f(<)h(1000;)f(i++\))123 1969 y(if)g (\(particle[i].index==0\))170 2025 y({)170 2082 y(for)h(\(k=i+1;)f(\(k)g(<)h (1000\)&&\(particle[k].index)d(==)i(0\))h(;)f(k++\);)170 2138 y(zdisp[j])g(=)h(i;)170 2195 y(zblock[j])f(=)h(k-i;)170 2251 y(j++;)170 2307 y(i)g(=)g(k;)170 2364 y(})75 2420 y(MPI_Type_indexed\()e(j,)h (zblock,)g(zdisp,)g(Particletype,)f(&Zparticles\);)457 2590 y(/*)h(4.3:)314 2646 y(send)g(the)g(first)h(two)f(coordinates)f(of)i(all)f (entries)g(*/)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 60 60 bop 75 -100 a Fy(60)613 b Fu(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fr(MPI_Datatype)22 b(Allpairs;)142 b(/*)24 b(datatype)f(for)g(all)g(pairs)h(of)f(coordinates)g (*/)75 162 y(MPI_Aint)g(sizeofentry;)75 275 y(MPI_Type_extent\()f (Particletype,)g(sizeofentry\);)194 388 y(/*)i(sizeofentry)e(can)i(also)f(be) h(computed)e(by)i(subtracting)e(the)i(address)266 444 y(of)f(particle[0])g (from)g(the)h(address)f(of)g(particle[1])f(*/)75 557 y(MPI_Type_hvector\()g (1000,)h(2,)g(sizeofentry,)g(MPI_REAL,)f(&Allpairs\);)75 614 y(MPI_Type_commit\()g(&Allpairs\);)75 670 y(MPI_Send\()h(particle.d,)f(1,)i (Allpairs,)e(dest,)h(tag,)h(comm\);)218 783 y(/*)g(an)f(alternative)g (solution)f(to)i(4.3)g(*/)75 896 y(MPI_Datatype)e(Onepair;)71 b(/*)23 b(datatype)g(for)h(one)f(pair)g(of)h(coordinates,)e(with)695 953 y(the)i(extent)f(of)h(one)f(particle)g(entry)g(*/)75 1009 y(MPI_Aint)g(disp2[3];)75 1065 y(MPI_Datatype)f(type2[3])h(=)h({MPI_LB,)e (MPI_DOUBLE,)h(MPI_UB};)75 1122 y(int)g(blocklen2[3])g(=)g({1,)h(2,)f(1};)75 1235 y(MPI_Address\()f(particle,)h(disp2\);)75 1291 y(MPI_Address\()f (particle[0].d,)g(disp2+1\);)75 1348 y(MPI_Address\()g(particle+1,)h (disp2+2\);)75 1404 y(for)g(\(i=0;)h(i<2;)f(i++\))g(disp2[i])g(-=)h (disp2[0];)75 1517 y(MPI_Type_struct\()e(3,)h(blocklen2,)g(disp2,)g(type2,)g (&Onepair\);)75 1574 y(MPI_Type_commit\()f(&Onepair\);)75 1630 y(MPI_Send\()h(particle[0].d,)f(1000,)h(Onepair,)g(dest,)g(tag,)g(comm\);)75 1807 y Ft(Fifth)18 b(example:)43 b Fy(same)17 b(arra)o(y)g(manipulations)i (as)f(in)g(the)g(previous)g(example,)h(but)f(using)h(absolute)75 1863 y(addresses)c(in)h(datat)o(yp)q(es.)75 1970 y Fr(struct)23 b(Partstruct)147 2026 y({)147 2083 y(int)g(class;)147 2139 y(double)g(d[6];)147 2195 y(char)g(b[7];)147 2252 y(};)75 2365 y(struct)g(Partstruct)g(particle[1000];)337 2478 y(/*)h(build)f(datatype)g (describing)g(first)g(array)g(entry)g(*/)75 2591 y(MPI_Datatype)f (Particletype;)75 2647 y(MPI_Datatype)g(type[3])h(=)h({MPI_INT,)e (MPI_DOUBLE,)h(MPI_CHAR};)75 2704 y(int)238 b(block[3])23 b(=)h({1,)f(6,)h (7};)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 61 61 bop 75 -100 a Fu(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b Fy(61)75 49 y Fr(MPI_Aint)118 b(disp[3];)75 162 y(MPI_Address\()22 b(particle,)h(disp\);)75 219 y(MPI_Address\()f(particle[0].d,)g(disp+1\);)75 275 y(MPI_Address\()g(particle[0].b,)g(disp+2\);)75 332 y(MPI_Type_struct\()g (3,)h(block,)g(disp,)g(type,)h(&Particletype\);)75 444 y(/*)g(Particletype)e (describes)g(first)i(array)f(entry)g(--)h(using)f(absolute)147 501 y(addresses)f(*/)505 614 y(/*)h(5.1:)361 670 y(send)h(the)f(entire)g (array)g(*/)75 783 y(MPI_Type_commit\()f(&Particletype\);)75 840 y(MPI_Send\()h(MPI_BOTTOM,)f(1000,)h(Particletype,)f(dest,)h(tag,)h (comm\);)481 1009 y(/*)f(5.2:)290 1065 y(send)g(the)h(entries)e(of)i(class)f (zero,)290 1122 y(preceded)g(by)g(the)h(number)f(of)g(such)h(entries)e(*/)75 1235 y(MPI_Datatype)g(Zparticles,)h(Ztype;)75 1348 y(MPI_Aint)g(zdisp[1000]) 75 1404 y(int)g(zblock[1000],)g(i,)g(j,)h(k;)75 1461 y(int)f(zzblock[2])g(=)h ({1,1};)75 1517 y(MPI_Datatype)e(zztype[2];)75 1574 y(MPI_Aint)118 b(zzdisp[2];)75 1686 y(j=0;)75 1743 y(for)23 b(\(i=0;)h(i)f(<)h(1000;)f (i++\))123 1799 y(if)g(\(particle[i].index==0\))170 1856 y({)170 1912 y(for)h(\(k=i+1;)f(\(k)g(<)h(1000\)&&\(particle[k].index)d(=)i(0\))h(;)g (k++\);)170 1969 y(zdisp[j])f(=)h(i;)170 2025 y(zblock[j])f(=)h(k-i;)170 2082 y(j++;)170 2138 y(i)g(=)g(k;)170 2195 y(})75 2251 y(MPI_Type_indexed\()e (j,)h(zblock,)g(zdisp,)g(Particletype,)f(&Zparticles\);)75 2307 y(/*)i(Zparticles)e(describe)h(particles)f(with)i(class)f(zero,)g(using) 147 2364 y(their)g(absolute)g(addresses*/)75 2477 y(/*)h(prepend)e(particle)h (count)g(*/)75 2533 y(MPI_Address\(&j,)f(zzdisp\);)75 2590 y(zzdisp[1])h(=)g(MPI_BOTTOM;)75 2646 y(zztype[0])g(=)g(MPI_INT;)75 2703 y(zztype[1])g(=)g(Zparticles;)-32 46 y Fw(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 62 62 bop 75 -100 a Fy(62)608 b Fu(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fr(MPI_Type_struct\(2,)21 b(zzblock,)i(zzdisp,)g(zztype,)g(&Ztype\);)75 162 y(MPI_Type_commit\()f (&Ztype\);)75 219 y(MPI_Send\()h(MPI_BOTTOM,)f(1,)i(Ztype,)f(dest,)g(tag,)g (comm\);)75 395 y Ft(Sixth)16 b(example:)43 b Fy(handling)17 b(of)e(unions.)75 501 y Fr(union)23 b({)147 558 y(int)118 b(ival;)147 614 y(float)70 b(fval;)218 671 y(})24 b(u[1000])75 784 y(int)119 b(utype;)75 897 y(/*)24 b(All)f(entries)g(of)h(u)f(have)h(identical)e(type;)h (variable)147 953 y(utype)g(keeps)g(track)g(of)h(their)f(current)g(type)g(*/) 75 1066 y(MPI_Datatype)70 b(type[2];)75 1122 y(int)286 b(blocklen[2])22 b(=)i({1,1};)75 1179 y(MPI_Aint)166 b(disp[2];)75 1235 y(MPI_Datatype)70 b(mpi_utype[2];)75 1292 y(MPI_Aint)166 b(i,j;)75 1405 y(/*)24 b(compute)e(an)i(MPI)g(datatype)e(for)i(each)f(possible)g(union)g(type;)147 1461 y(assume)g(values)g(are)g(left-aligned)f(in)i(union)f(storage.)g(*/)75 1574 y(MPI_Address\()f(u,)i(&i\);)75 1631 y(MPI_Address\()e(u+1,)i(&j\);)75 1687 y(disp[0])f(=)h(0;)f(disp[1])g(=)h(j-i)75 1743 y(type[1])f(=)h(MPI_UB;) 75 1856 y(type[0])f(=)h(MPI_INT;)75 1913 y(MPI_Type_struct\(2,)d(blocklen,)i (disp,)g(type,)g(&mpi_utype[0]\);)75 2026 y(type[0])g(=)h(MPI_FLOAT;)75 2082 y(MPI_Type_struct\(2,)d(blocklen,)i(disp,)g(type,)g(&mpi_utype[1]\);)75 2195 y(for\(i=0;)g(i<2;)g(i++\))g(MPI_Type_commit\(&mpi_utype[i])o(\);)75 2308 y(/*)h(actual)f(communication)f(*/)75 2421 y(MPI_Send\(u,)g(1000,)i (mpi_utype[utype],)d(dest,)i(tag,)h(comm\);)1967 46 y Fw(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 20 11:34:19 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id LAA14815; Thu, 20 Jan 1994 11:34:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA19657; Thu, 20 Jan 1994 11:34:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 20 Jan 1994 11:34:43 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from Phoenix.ERC.MsState.Edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id LAA19622; Thu, 20 Jan 1994 11:34:41 -0500 Received: from Athena.ERC.MsState.Edu by Phoenix.ERC.MsState.Edu (4.1/6.0s-FWP); id AA26540; Thu, 20 Jan 94 10:39:34 CST From: "Nathan E. Doss" Received: by Athena.ERC.MsState.Edu (4.1/6.0c-FWP); id AA17948; Thu, 20 Jan 94 10:37:54 CST Message-Id: <9401201637.AA17948@Athena.ERC.MsState.Edu> Subject: Proposal for explicit datatype packing and unpacking To: mpi-pt2pt@CS.UTK.EDU Date: Thu, 20 Jan 1994 10:37:54 -0600 (CST) X-Mailer: ELM [version 2.4 PL17] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 7504 Dear All, The remainder of this message proposes a small addition to the the datatype section of the point-to-point chapter. Nathan, Tony, Puri, and Kishore =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Proposal for explicit packing and unpacking of datatypes -------------------------------------------------------- Current Status MPI does not provide for the explicit "packing" and "unpacking" of messages. MPI provides an interface where the user can say what needs to be sent or received, but does not have to specify how it is packed or unpacked. This is a good thing, but we lacks the ability for the user to specify ``when'' a message should be packed or unpacked (i.e. data conversions and scatter/gather). Currently, MPI must gather the data every time a send is performed and scatter the data on every receive (and in heterogeneous environments, perform any neccessary data conversions on every send and receive). MPI provides the ability to pack by sending a message to yourself and receiving it using MPI_BYTE, however, in heterogeneous environments, the MPI_TYPE_SIZE function may not give the correct size of a message receieved from a remote node (in MPI_BYTE format) since the size is given in terms of the local data representation. As an example, assume we have a datatype that consists of every other integer from an array of 20 integers and wish to send this information to some number of other processors. We want to avoid doing any unnecessary data conversions and gathering. The following code should work in homogeneous environments but is not guaranteed to work in heterogeneous environments: int a[20]; MPI_Datatype type; int size; int rank; char *buffer; MPI_Status status; MPI_Comm_rank ( MPI_COMM_WORLD, &rank ); MPI_Type_vector ( 10, 1, 2, MPI_INT, &type ); MPI_Type_size ( type, &size ); buffer = (char *) malloc ( size ); MPI_Sendrecv ( a, 1, type, rank, 0, buffer, size, MPI_BYTE, rank, 0, MPI_COMM_WORLD, &status ); for (each dest) MPI_Send ( buffer, size, MPI_BYTE, dest, 0, MPI_COMM_WORLD ); This does not necessarily work since the size returned by MPI_TYPE_SIZE may not be big enough to hold the datatype elements after data conversions have been applied to it. Another problem with this example is that in implementations that check to see that the sent type map is equivalent to the received type map, the MPI_SENDRECV call would generate an error or warning (depending on the implementation) that the types do not match. In the latest point to point draft (dated Jan 4, 1994), an MPI_MOVE function is proposed which would be used in place of the MPI_SENDRECV call used in the previous example. The draft states that this function is "[e]ssentially equivalent to MPI_SENDRECV with source = dest, but could be defined to be lenient w.r.t. type matching. It could then be used for packing (with totype = MPI_BYTE) or unpacking (with fromtype = MPI_BYTE)." This does remove the problem caused by MPI_SENDRECV performing type checking, however MPI_MOVE as proposed could not take the place of MPI_SENDRECV in a heterogeneous implementation. MPI_MOVE as proposed appears to only convert between types as represented on the local machine. This function may be convenient for converting/copying locally, but would not be useful for packing/unpacking in a heterogeneous environment. Proposal We propose a function that gives the size of the "packed" data in relationship to a communicator, MPI_TYPE_COMM_SIZE. In a heterogeneous implementation, the decision to apply data conversions to messages may depend on the members of the communicator. For example, one communicator may contain only homogeneous nodes with no conversions necessary while another communicator may consist of heterogeneous nodes. The size of the packed data in these two cases may be different if the determination of whether or not to pack data is based on the type of nodes in a communicator. This allows the user to allocate a contiguous buffer into which a message may be received with no data conversions. MPI_TYPE_COMM_SIZE ( datatype, comm, size ) IN datatype datatype (handle) IN comm communicator (handle) OUT size type size after conversions (integer) The following MPI_TYPE_PACK and MPI_TYPE_UNPACK functions can be used to pack (unpack) buffers with respect to the given communicator. Like the proposed MPI_MOVE, these functions are lenient with respect to type checking. Unlike MPI_MOVE, they are specifically for packing and unpacking messages and perform the necessary data conversions. MPI_TYPE_PACK ( comm, oldbuffer, count, oldtype, newbuffer) IN comm communicator (handle) IN oldbuffer buffer to convert from IN count count of old buffer items IN oldtype MPI type of oldbuffer OUT newbuffer packed buffer after gather & data conversions MPI_TYPE_UNPACK ( comm, oldbuffer, newbuffer, count, newtype) IN comm communicator (handle) IN oldbuffer packed buffer IN newbuffer unpacked buffer after scatter & data conversions IN count count of buffer items IN newtype MPI type of newbuffer The messages to be packed or unpacked are assumed to be of type MPI_BYTE. The following illustrates how the previously given example can be written using these proposed functions: int a[20]; MPI_Datatype type; int size; int rank; char *buffer; MPI_Status status; MPI_Comm_rank ( MPI_COMM_WORLD, &rank ); MPI_Type_vector ( 10, 1, 2, MPI_INT, &type ); MPI_Type_comm_size ( type, MPI_COMM_WORLD, &size ); buffer = (char *) malloc ( size ); MPI_Type_pack ( MPI_COMM_WORLD, a, 1, type, buffer ); for (each dest) MPI_Send ( buffer, size, MPI_BYTE, dest, 0, MPI_COMM_WORLD ); If type matching is performed, then the a type map consisting entirely of MPI_BYTEs must be used by each receive. Note The draft may want to specify that if type matching is done, that the MPI_BYTE type can match any other type. If this is so, then the argument for MPI_TYPE_PACK and MPI_TYPE_UNPACK is weakened. Advantages o These functions provide a clear way for programmers to separate the gather (scatter) and data conversion operations from the communication operation. For example, when forwarding messages, tere is no need to unpack (scatter and apply data conversions) and then repack (gather and apply data conversions). o There is no way in MPI to find the size of a datatype after data conversions are applied. MPI_TYPE_COMM_SIZE provides this. o By providing MPI_TYPE_PACK and MPI_TYPE_UNPACK, MPI can provide an explicit means for users to pack and unpack buffers in a heterogeneously clean manner. Disadvantages o On some platforms it may be more efficient to do the gather repetitively than to pack and then perform multiple sends. Thus, these use of these functions may decrease rather than increase o The MPI_TYPE_PACK and MPI_TYPE_UNPACK functions are not strictly necessary since MPI_SENDRECV can be used in a similar fashion. From owner-mpi-pt2pt@CS.UTK.EDU Thu Feb 3 12:34:14 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id MAA13140; Thu, 3 Feb 1994 12:34:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id MAA23440; Thu, 3 Feb 1994 12:32:35 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 3 Feb 1994 12:32:26 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id MAA23382; Thu, 3 Feb 1994 12:31:33 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 2911; Thu, 03 Feb 94 12:31:25 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 5131; Thu, 3 Feb 1994 12:30:36 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Thu, 03 Feb 94 12:30:15 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA27361; Thu, 3 Feb 1994 12:30:21 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9402031730.AA27361@snir.watson.ibm.com> To: mpi-pt2pt@CS.UTK.EDU Subject: current pt2pt draft -- pack/unpack functions added at the end Reply-To: snir@watson.ibm.com Date: Thu, 03 Feb 94 12:30:20 -0500 -:) -:) -:) -:) -:) %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: pt2pt942.dvi %%Pages: 72 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 5 118 df<3C7EE2C0C0C0E27E3C07097F880A>99 D110 D<3C7EE7C3C3C3E77E3C08097F880B>I<303030FCFC30303030323E1C070C808B08> 116 DI E /Fc 2 50 df<1E007F806180C0C0C0C0C0C0C0 C0C0C0C0C0C0C061807F801E000A0D7E8C0E>48 D<18F8F81818181818181818FFFF080D7D8C0E >I E /Fd 1 106 df<04040000000038FCD8181832333E3C080F7E8E0B>105 D E /Fe 5 118 df<0F803FE078607000E000E000E000E000E000702078603FE00F800B0D7F8C 0E>99 D110 D<0F003FC070E070E0E070E070E070E070E07070E079E03FC00F000C0D7F8C0F>I<38383838FF FF3838383838383838393F1E08117F900B>116 DI E /Ff 2 81 df<00001C00003C0000F80001E00003C00007 80000F00000E00001E00003C00003C00003C000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 0000780000780000780000780000780000F00000F00000F00001E00001E00003C0000380000700 000E00001C0000780000E00000E000007800001C00000E000007000003800003C00001E00001E0 0000F00000F00000F0000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 00007800007800003C00003C00003C00001E00000E00000F000007800003C00001E00000F80000 3C00001C167C7B8121>40 D80 D E /Fg 4 111 df<60F0F07030303060E040040A7E830A>59 D<030003800300000000000000 00001C003E006600C600CC000C000C0018001980318033003F001C00091480930C>105 D<0060007000600000000000000000038007C00CE018C018C000C000C001800180018001800300 0300030003006600EE00FC0078000C1A81930E>I<387C007CFE006F8600CF0600CE06000E0600 0C06001C0C00180CC01818C0181980301F80300E00120D808C15>110 D E /Fh 2 4 df0 D<040006000600C660F7E03F801F003F80F7E0C6 600600060004000B0D7E8D11>3 D E /Fi 27 123 df<007E01FE07800E001E003C003C007800 7FF87FF8F000F000F00070007000700038001C180FF807E00F147E9312>15 D<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C181838306040060E7C840D>I62 D<0001FC000007FF0000 1F07C0003C03E000F001E001E000F003C000F007C00078078000780F0000781F0000781E000078 3E0000783E0000787C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001E0F80003 E0F80003C0F80007C07800078078000F007C001F003C003E003E007C001E00F0000F83E00007FF 800001FC00001D217E9F23>79 D<0FFFFFFC1FFFFFF81E03C0781803C0383803C0183007801830 0780186007801860078030C00F0030C00F0030000F0000000F0000001E0000001E0000001E0000 001E0000003C0000003C0000003C0000003C00000078000000780000007800000078000000F000 0000F0000000F0000001F000007FFFC0007FFFC0001E1F7F9E1B>84 D<00F18003FDC0078F800E 07801C07803C07803C0700780700780700780700F00E00F00E00F00E00F00E18F01C30F03C3070 7C3078FC603FCFC00F078015147E9318>97 D<07803F803F000700070007000E000E000E000E00 1C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01EE01EE03CE03CE038607071 E03FC01F0010207E9F14>I<0000780003F80003F00000700000700000700000E00000E00000E0 0000E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700 F00E00F00E00F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015207E9F18>100 D<007C01FE07870E031E033C033C06781C7FF87FC0F000F000F000F000700070027007381E1FF8 07E010147E9315>I<00007C0000FE00019E00039E00030C000700000700000700000700000E00 000E00000E0000FFF001FFF0000E00001C00001C00001C00001C00001C00003800003800003800 00380000380000700000700000700000700000700000E00000E00000E00000E00000C00001C000 31C000798000F300007E00003C000017297E9F16>I<001E30007FB800F1F001C0F00380F00780 F00780E00F00E00F00E00F00E01E01C01E01C01E01C01E01C01E03801E07800E0F800F1F8007FF 0001E700000700000700000E00000E00300E00781C00F078007FF0003FC000151D809316>I<00 7000F000F00060000000000000000000000000000007001F8039C031C061C061C0C38003800380 070007000E000E000E181C301C301C601CE00FC007000D1F7F9E10>105 D<0000C00001E00001E00001C0000000000000000000000000000000000000000000003E00007F 0000C380018380030380030380060700000700000700000700000E00000E00000E00000E00001C 00001C00001C00001C0000380000380000380000380000700000700030700078E000F3C0007F80 003E00001328819E13>I<01E0000FE0000FC00001C00001C00001C00003800003800003800003 80000700000700000701C00707E00E0C300E18700E30F00E60F01CC0E01F80001F80001FE00038 F000383800383800381C307038607038607038407018C0E01F80600F0014207E9F18>I<03C01F C01F8003800380038007000700070007000E000E000E000E001C001C001C001C00380038003800 38007000700070007180E300E300E300E6007E003C000A207E9F0E>I<0F03E03E001F8FF0FF00 31DC39C38061F03B038061E01E01C061C01C01C0C3C03C03800380380380038038038003803803 800700700700070070070007007007000700700E0C0E00E00E180E00E01C180E00E01C300E00E0 0C701C01C00FE00C00C0078026147F9328>I<0F07C0001F8FE00031D8700061F0380061E03800 61C03800C3C070000380700003807000038070000700E0000700E0000700E0000701C1800E01C3 000E0383000E0386000E018E001C01FC000C00F00019147F931B>I<007C0001FF000383800F01 C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E 003838001FF00007C00013147E9316>I<03C1E007E7F80C7E3C187C1C18781E18701E30E01E00 E01E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E0007 00000700000E00000E00000E00000E00001C0000FFC000FFC000171D819317>I<0F0F001F9F80 31F1C061E1C061E3C061C3C0C3C3800380000380000380000700000700000700000700000E0000 0E00000E00000E00001C00000C000012147F9315>114 D<007C01FE0387070F070F070E0F0007 8007F803FC01FE001E000F700FF00EF00EE01C70383FF01FC010147E9315>I<00C000E001C001 C001C001C003800380FFF8FFF807000700070007000E000E000E000E001C001C001C001C183830 3830386038E01FC00F000D1C7F9B10>I<0780301FC03838C07030E07060E07061C070C1C0E003 80E00380E00380E00701C00701C00701C00701C3070386070386070786070F8C03F9F801F0F018 147F931A>I<07C3C00FE7E0187C703038706038F06038F0C070E000700000700000700000E000 00E00000E00070E030F1C060F1C060E1C0C0E3E1807E7F003C3E0014147E931A>120 D<0780301FC03838C07030E07060E07061C070C1C0E00380E00380E00380E00701C00701C00701 C00701C0070380070380070780070F8003FF0001F700000700000700000E003C0C003C1C003838 0030F0001FE0000F8000151D7F9316>I<01E06003F0C007FFC00E1F800C0300000600000C0000 180000300000600000C0000180000300000600C00C01801803803F87007FFE0061FC00C0700013 147E9315>I E /Fj 4 62 df<0FC01FE0387070386018E01CE01CE01CE01CE01CE01CE01CE01C E01C6018703838701FE00FC00E137F9211>48 D<06001E00FE00EE000E000E000E000E000E000E 000E000E000E000E000E000E000E00FFE0FFE00B137D9211>I<1F003FC061E0C0E0E070E07000 70007000E000C001C0030006000C30183030707FE0FFE0FFE00C137E9211>I<7FFFE0FFFFF000 0000000000000000000000000000000000FFFFF07FFFE0140A7E8B19>61 D E /Fk 2 42 df<0000200000000000700000000000F00000000001E00000000001C000000000 03C0000000000780000000000FFFFFFFF0001FFFFFFFF8007FFFFFFFF800F80000000003E00000 00001FC000000000FF0000000000FE00000000001F800000000007E00000000001F00000000000 7800000000003FFFFFFFF8001FFFFFFFF8000FFFFFFFF00007800000000003C00000000001C000 00000001E00000000000F00000000000700000000000200000002D1D7D9934>40 D<0000002000000000007000000000007800000000003C00000000001C00000000001E00000000 000F00007FFFFFFF8000FFFFFFFFC000FFFFFFFFF00000000000F800000000003E00000000001F C00000000007F80000000003F8000000000FC0000000003F00000000007C0000000000F000FFFF FFFFE000FFFFFFFFC0007FFFFFFF80000000000F00000000001E00000000001C00000000003C00 000000007800000000007000000000002000002D1D7D9934>I E /Fl 8 104 df0 D<70F8F8F87005057C8D0D>I<018001C001800180 C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C0018010147D9417>3 D<07E01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E01012 7D9317>15 D<000000C0000003C000000F0000003C000000F0000003C000000F0000001C000000 78000001E00000078000001E00000078000000E0000000780000001E0000000780000001E00000 00780000001C0000000F00000003C0000000F00000003C0000000F00000003C0000000C0000000 00000000000000000000000000000000000000000000000000FFFFFFC0FFFFFFC01A247C9C23> 20 D91 D<001F003F00F800E001C001C001C001C001C001C001C0 01C001C001C001C001C001C001C001C003800700FE00F800FE000700038001C001C001C001C001 C001C001C001C001C001C001C001C001C001C001C000E000F8003F001F102D7DA117>102 DI E /Fm 1 16 df<07E01FF83FFC7FFE7FFEFFFF FFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E010107E9115>15 D E /Fn 2 42 df<0001C0000000000380000000000380000000000700000000000E00000000000E000000 00001FFFFFFF80007FFFFFFF8000FFFFFFFF8001E0000000000780000000003F0000000000FC00 000000003F000000000007800000000001E00000000000FFFFFFFF80007FFFFFFF80001FFFFFFF 80000E00000000000E000000000007000000000003800000000003800000000001C00000002919 7D9630>40 D<000001C00000000000E00000000000E00000000000700000000000380000000000 380000FFFFFFFC0000FFFFFFFF0000FFFFFFFF800000000003C00000000000F000000000007E00 000000001F80000000007E0000000000F00000000003C000FFFFFFFF8000FFFFFFFF0000FFFFFF FC0000000000380000000000380000000000700000000000E00000000000E00000000001C00000 29197D9630>I E /Fo 56 122 df<01F1C003F1C007F1C00701C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFF1C0FFF1C0FFF1C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0121D809C16>13 D<00F0000001F8000003FC0000039C0000070E0000070E0000070E0000070E0000070E0000071E 0000071C0000073C00000778000003F0040003E0070007C00F000FC00E001FC00E003DE01E0078 E01C0070F03C00E0703800E0387800E03CF000E01FE000E00FC00070078000781FE0C03FFFFFC0 1FF8FFC007E03F001A1F7E9D1F>38 D<00E001C0038007000E000E001C001C0038003800380070 00700070007000E000E000E000E000E000E000E000E000E000E000E000E0007000700070007000 3800380038001C001C000E000E000700038001C000E00B2A7E9E10>40 DI<0006000000060000000600000006000000060000000600000006000000060000000600 0000060000000600000006000000060000FFFFFFE0FFFFFFE00006000000060000000600000006 00000006000000060000000600000006000000060000000600000006000000060000000600001B 1C7E9720>43 D<7878787838307060E005097D830C>II58 D<7FFFFFC0FFFFFFE0000000000000000000 0000000000000000000000000000000000000000000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<001C0000003E0000003E0000002E0000006700000067000000E7800000C7800000C3800001C3 C0000183C0000181C0000381E0000381E0000700F0000700F0000600F0000E0078000FFFF8000F FFF8001FFFFC001C003C0018003C0038001E0038001E0070001F0070000F0070000F00E0000780 191D7F9C1C>65 DI<003FC000FFF003FFF007C0700F8000 1E00003E00003C0000780000780000780000F00000F00000F00000F00000F00000F00000F00000 F00000F000007800007800007800003C00003E00001E00000F800807C07803FFF800FFF0003F80 151F7D9D1B>III< FFFF80FFFF80FFFF80F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000 FFFF00FFFF00FFFF00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000 F00000F00000F00000111D7C9C18>I<003F8001FFF003FFF807C0F80F00181E00003E00003C00 00780000780000780000F00000F00000F00000F00000F00000F00000F007F8F007F8F007F87800 387800387800383C00383E00381E00380F003807C07803FFF801FFF0003F80151F7D9D1C>III75 DIII<00 3F000001FFE00003FFF00007C0F8000F807C001E001E003E001F003C000F007800078078000780 78000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007 C078000780780007807C000F803C000F003E001F001F003E000F807C0007C0F80003FFF00001FF E000003F00001A1F7E9D1F>II82 D<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F00000F800 007C00007F00003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C00003C00003 C00003C0C00780E00780FC1F007FFE001FFC0007F000121F7E9D17>IIIII89 D<7FFFF07FFFF07FFFF00001E00003C0 0007C0000780000F80001F00001E00003E00003C0000780000F80000F00001F00001E00003C000 07C0000780000F80001F00001E00003E00003C0000780000FFFFF0FFFFF0FFFFF0141D7E9C19> II93 D<0FC03FF07FF87038401C001C001C00FC0FFC3F FC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 DI<07E00FF81FFC3C1C70047000E000E000E000E000E000E000700070 043C1C1FFC0FF807E00E127E9112>I<000E000E000E000E000E000E000E000E000E000E000E0F 8E1FEE3FFE7C3E700E700EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E9C 15>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF807E0 0E127E9112>I<00FC01FC03FC07000E000E000E000E000E000E000E00FFE0FFE0FFE00E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E1D809C0D>I<03C3C00FFFC0 1FFFC01C3800381C00381C00381C00381C00381C001C38001FF8001FF0003BC000380000380000 1FFC001FFF003FFF80700780E001C0E001C0E001C0F003C07C0F803FFF001FFE0007F800121B7F 9115>III107 DIII<03F0000FFC001FFE003C0F00780780700380E001C0 E001C0E001C0E001C0E001C0F003C07003807807803C0F001FFE000FFC0003F00012127F9115> II114 D<1FC03FF07FF0F030E000E000 F0007F003FC01FE000F0003800388038F078FFF07FE01FC00D127F9110>I<1C001C001C001C00 1C001C00FFE0FFE0FFE01C001C001C001C001C001C001C001C001C001C001C001C201FF00FF007 C00C187F970F>III<7003807807003C0E001C1C000E1C0007380003F00001E0 0001C00001E00003F0000738000E18000E1C001C0E00380700700380F003C01212809113>120 DI E /Fp 3 86 df<70F8F8F87005057D840C> 58 D<01FFFF0003FFFFC0003C01E0003C00F0003C00F0003C007000780070007800F0007800F0 007801E000F003C000F0078000FFFF0000FFFE0001E01F0001E0078001E003C001E003C003C003 C003C003C003C003C003C003C00780078007800F0007801E0007807C00FFFFF800FFFFE0001C1C 7E9B1F>66 D<7FF87FE0FFF8FFE00F000E000F000C000F000C000F000C001E0018001E0018001E 0018001E0018003C0030003C0030003C0030003C00300078006000780060007800600078006000 F000C000F000C000F000C000F0018000F0018000F003000070060000700C0000383800001FF000 0007C000001B1D7D9B1C>85 D E /Fq 28 122 df<001FE00000FFF00003F0380007C07C000780 FC000F80FC000F80FC000F8078000F8000000F8000000F800000FFFFFC00FFFFFC000F807C000F 807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C00 0F807C000F807C000F807C007FE1FF807FE1FF80191D809C1B>12 D<0030006001C00380030007 000E001E001C003C003C003800780078007800F800F000F000F000F000F000F000F000F000F000 F80078007800780038003C003C001C001E000E0007000300038001C0006000300C297D9E13>40 DI<387CFEFEFE7C3807077D860D>46 D<387CFEFEFE7C380000000038 7CFEFEFE7C3807127D910D>58 D<0FF03FFC703EF03FF83FF83FF83F203E007C00F800F001C001 C0018001800180018001800000000000000000038007C00FE00FE00FE007C00380101D7D9C17> 63 D68 D73 D78 D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07 C0FC07C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F00 7C1F007C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00007E01803E03801F07000FFE00 03F80011127E9115>99 D<000FF0000FF00001F00001F00001F00001F00001F00001F00001F000 01F00001F003F1F00FFFF01F07F03E01F07C01F07C01F0FC01F0FC01F0FC01F0FC01F0FC01F0FC 01F07C01F07C01F03E03F01F0FF00FFDFE03F1FE171D7E9C1B>I<01FC000FFF001F0F803E07C0 7C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC0000FC00007C00007E00603E00C01F81C00FFF00 01FC0013127F9116>I<003F0001FF8003E7C007C7C00F87C00F83800F80000F80000F80000F80 000F8000FFF800FFF8000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80000F80000F80000F80007FF8007FF800121D809C0F>I<03F0F00FFFF83E1F383C0F307C 0F807C0F807C0F807C0F803C0F003E1F001FFC0033F0003000007000003800003FFF003FFFC01F FFE03FFFF07801F8F00078F00078F000787800F03E03E01FFFC003FE00151B7F9118>II<1E003F003F007F003F003F001E0000000000000000000000FF00 FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E >I108 DII<01FC000FFF801F 07C03E03E07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F87C01F07C01F03E03E01F 07C00FFF8001FC0015127F9118>II<03E0600FF8E01F0DE03E07E07E03E07C03 E0FC03E0FC03E0FC03E0FC03E0FC03E0FC03E07C03E07E03E03E07E03F0FE00FFBE007E3E00003 E00003E00003E00003E00003E00003E0001FFC001FFC161A7E9119>II<1FD83FF87038E018E018F000FF807FE07FF01FF807FC007CC01CC0 1CE01CF038FFF0CFC00E127E9113>I<030003000300070007000F000F003F00FFFCFFFC1F001F 001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913>II121 D E /Fr 26 122 df<7FF0FFE0FFE00C037D8A10>45 D<70F8F8F0E005057B840E>I<0000070000000700 00000F0000000F0000001F0000003F0000003F0000006F0000006F000000CF000000CF0000018F 0000038F0000030F0000060F0000060F00000C0F80000C07800018078000180780003FFF80007F FF800060078000C0078000C00780018007800180078003000780070007800F0007807FC07FF8FF C07FF81D207E9F22>65 D<01FFFFFE01FFFFFC001E003C001E001C001E001C003C000C003C000C 003C000C003C001800780C1800780C1800780C0000781C0000F0380000FFF80000FFF80000F038 0001E0300001E0300001E0303001E0306003C0006003C0006003C000C003C001C0078001800780 03800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69 D<01FFFF0001FFFFC0001E01E000 1E00F0001E0078003C0078003C0078003C0078003C0078007800F0007800F0007801E0007803C0 00F00F0000FFFE0000FFF80000F03C0001E01E0001E00E0001E00F0001E00F0003C01E0003C01E 0003C01E0003C01E0007803C0007803C1807803C1807803C30FFF81E30FFF00FE0000007C01D20 7D9E21>82 D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00 F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F 700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207B9F15>I<007E0001FF00 0383800F07801E07801C07003C0200780000780000780000F00000F00000F00000F00000F00000 700200700700381E001FF80007E00011147C9315>I<0000780003F80003F00000700000700000 700000E00000E00000E00000E00001C00001C000F1C003FDC0078F800E07801C07803C07803C07 00780700780700780700F00E00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F07 8015207C9F17>I<007C0001FF000783000F01801E01803C01803C0300780E007FFC007FE000F0 0000F00000F00000F000007000007002007807003C1E001FF80007E00011147C9315>I<0000F8 0001FC0003BC00033C000718000700000700000E00000E00000E00000E00000E0001FFE001FFE0 001C00001C00001C00003800003800003800003800003800007000007000007000007000007000 00700000E00000E00000E00000E00001C00001C00001C0000180003380007B8000F300007E0000 3C00001629829F0E>I<003C6000FF7001E3E00381E00701E00F01E00F01C01E01C01E01C01E01 C03C03803C03803C03803C03803C07003C0F001C1F001E3F000FFE0003CE00000E00000E00001C 00001C00301C00783800F0F0007FE0003F8000141D7E9315>I<006000F000F000E00000000000 0000000000000000000F001F80318031C063806380C3800700070007000E000E000E001C301C60 1C6038C018C01F800F000C1F7D9E0E>105 D<01E0000FE0000FC00001C00001C00001C0000380 000380000380000380000700000700000703C00707E00E0C600E10E00E21E00E61E01CC1C01F80 001F00001FC00039E0003870003870003838607070C07070C07070C0703180E03F00601E001320 7D9F15>107 D<03C01FC01F8003800380038007000700070007000E000E000E000E001C001C00 1C001C0038003800380038007000700070007180E300E300E300E6007E003C000A207C9F0C>I< 1E07C0F8003F1FE1FC0033B8730E0063E076070063C03C07006380380700C780780E000700700E 000700700E000700700E000E00E01C000E00E01C000E00E01C000E00E038601C01C038C01C01C0 38C01C01C071801C01C031803803803F001801801E0023147D9325>I<1E07C03F1FE033B87063 E07063C038638038C780700700700700700700700E00E00E00E00E00E00E01C31C01C61C01C61C 038C1C018C3801F81800F018147D931A>I<007C0001FF000383800F01C01E01C01C01E03C01E0 7801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E003838001FF00007C000 13147C9317>I<03C1E007E7F8067E3C0C7C1C0C781E0C701E18E01E00E01E00E01E00E01E01C0 3C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E000700000700000E00000E00 000E00000E00001C0000FFC000FFC000171D809317>I<1E0F003F3F8033F1C063C1C063C3C063 83C0C783800700000700000700000E00000E00000E00000E00001C00001C00001C00001C000038 000018000012147D9313>114 D<00FC03FE07070E0F0E0F0E0E1E000F800FF007F803FC003E00 1E701EF01CF01CE03860703FE01F8010147D9313>I<018001C0038003800380038007000700FF F0FFF00E000E000E000E001C001C001C001C0038003800380038307060706070C071803F001E00 0C1C7C9B0F>I<0F00601F80703180E031C0E06380E06380E0C381C00701C00701C00701C00E03 800E03800E03800E038C0E07180E07180E0F180E1F3007F3F003E1E016147D9318>I<0F01C01F 83C03183E031C1E06380E06380E0C380C00700C00700C00700C00E01800E01800E01800E03000E 03000E06000E06000F0C0007F80001E00013147D9315>I<0787800FCFC018F8E03070E06071E0 6071E0C0E1C000E00000E00000E00001C00001C00001C00071C060F380C0F380C0E38180C7C300 7CFE00387C0013147D9315>120 D<0F00601F80703180E031C0E06380E06380E0C381C00701C0 0701C00701C00E03800E03800E03800E03800E07000E07000E0F000E1F0007FE0003EE00000E00 000E00001C00781C0078380070700060E0003FC0001F0000141D7D9316>I E /Fs 45 122 df45 DI<00C001C007C0 FFC0FFC0FBC003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003 C003C003C003C003C003C003C003C003C0FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E001E06001E02001E00001E00001 E00001E00003C00003C0000780000780000F00001E00003C0000780000F00001E00001C0000380 000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001F FF003E0F003807807003C02003C02003C00003C00003C00003C0000780000780000F00001E0003 FC0003F80003FE00000F000007800003C00003C00001E00001E00001E00001E00001E08001E0C0 03C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00 006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00 3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00000F00 000F00000F00000F0015217FA018>I<3FFF803FFF803FFF803C00003C00003C00003C00003C00 003C00003C00003C00003C00003CF8003FFE003FFF003F0F803E07803C03C03803C00001E00001 E00001E00001E00001E00001E00001E04003C04003C0E003C07007807C1F003FFE000FFC0003F0 0013227EA018>I<007E0001FF0003FF0007C1000F00001E00001E00003C00003C000078000078 000078F800F3FE00F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F001E0F001E0F001E0F0 01E07001E07001E07801E07803C03803C03C03801C07801F0F000FFE0007FC0001F00013237EA1 18>II<01F00007FC000FFE 001E0F003C07803C07807803C07803C07803C07803C07803C03803803C07801E0F000F1E0007FC 0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F001E07803 C07803C03C07803E0F801FFF0007FC0001F00013237EA118>I<001F0000001F0000003F800000 3F8000003B8000007BC0000073C0000071C00000F1E00000F1E00000E0E00001E0F00001E0F000 01C0F00003C0780003C078000380780007803C0007803C0007003C000F001E000F001E000FFFFE 001FFFFF001FFFFF001C000F003C0007803C00078038000780780003C0780003C0700003C0F000 01E0F00001E0E00001E01B237EA220>65 D<000FF000007FFC0000FFFF0001F01F0003C0070007 8002000F0000001E0000003E0000003C0000003C000000780000007800000078000000F0000000 F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000078000000780000 00780000003C0000003C0000003E0000001E0000000F0000800780018003C0038001F00F8000FF FF00007FFC00000FF00019257DA31F>67 DII76 DII<001FC000007FF00001FFFC0003 F07E0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000F0780000F0780000F0 70000070F0000078F0000078F0000078F0000078F0000078F0000078F0000078F0000078F00000 78780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F0007C00F800F8007C0 1F0003F07E0001FFFC00007FF000001FC0001D257DA324>I82 D<00FF0003FFC007FFF00F83F01E00F03C0020 3C00007800007800007800007800007800007C00003C00003F00001FC0000FFC0007FF0001FF80 003FC00007E00001F00000F00000F8000078000078000078000078000078400078E000F0F000F0 F801E07F07C03FFF800FFF0001FC0015257EA31B>III<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780F F00FF00FF00FF80F7C3F7FFF3FFF1F8F10167E9517>97 DI<01FC0007FF000FFF801F07803C01807800007800 00700000F00000F00000F00000F00000F00000F000007800007800007800403C00C01F07C00FFF C007FF8001FC0012167E9516>I<0003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F0 03C0F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003 E3C012237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0 FFFFC0F00000F00000F000007000007800007800003C00801F07800FFF8007FF0001F80012167E 9516>I<003F00FF01FF03C1038007800780078007800780078007800780FFF8FFF8FFF8078007 8007800780078007800780078007800780078007800780078007800780078007800780102380A2 0F>I<01F0F807FFF80FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F00 1FFE001FFC0039F0003800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078 F00078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>I105 D<00F000F000F000F000000000000000000000000000000000000000F000F000F000F000F000F0 00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000 F000F080F0E3E0FFE07FC01F000C2D83A20D>IIIII<01FC0007FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F0 0078F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F95 18>II 114 D<07F01FFE3FFF3C0F7803780078007C003F003FF01FF80FFC01FE001F000F000F400FE00F F81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F00 0F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12>I< F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF83F7F FF7FEF1F8F10167C9519>III<7801F07C01E03E03C01E07C00F0780 078F0007DE0003FC0001FC0000F80000700000F80001FC0003DC00039E00078F000F07801E0780 1E03C03C01E07800F0F800F81516809516>II E /Ft 80 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F870000000000070F8F8 F870051C779B18>33 D<03C00007E0000FF0001E78001C38001C38001C38001C38001C7BF01CF3 F01EE3F00FE7800FC7000F87000F0F001F0E003F8E007B9E0073DC00F1DC00E1FC00E0F800E078 70F0F87071FE707FFFF03FCFE01F03C0141C7F9B18>38 D<007000F001F003C007800F001E001C 003C003800780070007000F000E000E000E000E000E000E000E000E000F0007000700078003800 3C001C001E000F00078003C001F000F000700C24799F18>40 D<6000F00078003C001E000F0007 80038003C001C001E000E000E000F00070007000700070007000700070007000F000E000E001E0 01C003C0038007800F001E003C007800F00060000C247C9F18>I<01C00001C00001C00001C000 C1C180F1C780F9CF807FFF001FFC0007F00007F0001FFC007FFF00F9CF80F1C780C1C18001C000 01C00001C00001C00011147D9718>I<00600000F00000F00000F00000F00000F00000F00000F0 007FFFC0FFFFE0FFFFE07FFFC000F00000F00000F00000F00000F00000F00000F0000060001314 7E9718>I<3C7E7F7F7F3F0F0E1E7CF870080C788518>I<7FFF00FFFF80FFFF807FFF0011047D8F 18>I<78FCFCFCFC780606778518>I<000300000780000F80000F80000F00001F00001F00003E00 003E00003C00007C00007C0000F80000F80000F00001F00001F00003E00003E00007C00007C000 0780000F80000F80001F00001F00001E00003E00003E00007C00007C0000780000F80000F80000 F0000060000011247D9F18>I<01F00007FC000FFE001F1F001C07003803807803C07001C07001 C0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803 C03803801C07001F1F000FFE0007FC0001F000131C7E9B18>I<01800380078007800F803F80FF 80FB80638003800380038003800380038003800380038003800380038003800380038003807FFC FFFE7FFC0F1C7B9B18>I<07F8001FFE003FFF007C0F807003C0F001E0F000E0F000E0F000E000 00E00000E00001E00001C00003C0000780000F00001E00003C0000780000F00003E00007C0000F 80001E00E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007C0F807803C0 7801C03001C00001C00003C0000780000F8003FF0003FE0003FF00000F800003C00001C00001E0 0000E00000E0F000E0F001E0F001C0F003C07C0F803FFF001FFE0007F800131C7E9B18>I<001F 00003F00007F0000770000F70001E70001C70003C7000787000707000E07001E07003C07003807 00780700F00700FFFFF8FFFFF8FFFFF8000700000700000700000700000700000700007FF000FF F8007FF0151C7F9B18>I<1FFF803FFF803FFF8038000038000038000038000038000038000038 00003800003BFC003FFE003FFF003E07801803C00001E00000E00000E06000E0F000E0F001E0E0 01C0F003C07C0F803FFF001FFE0007F800131C7E9B18>I<007E0001FF0007FF800FC3C01F03C0 3C03C0380180780000700000700000F3F800EFFE00FFFF00FE0F80F803C0F001C0F001E0F000E0 F000E0F000E07000E07001E07801C03C03C01E0F800FFF0007FE0003F800131C7E9B18>II<03F8000FFC001FFE003E0F00780780F003C0E001C0E001C0E0 01E0E001E0E001E0F001E07803E03E0FE01FFFE00FFEE003F8E00001E00001C00001C00003C030 0380780780780F00783E003FFC001FF8000FE000131C7E9B18>57 D<78FCFCFCFC780000000000 00000078FCFCFCFC780614779318>I<3C7E7E7E7E3C0000000000000000387C7E7E7E3E0E1E3C 78F060071A789318>I<000300000F80001F80003F0000FE0001FC0003F00007E0001FC0003F80 007E0000FC0000FC00007E00003F80001FC00007E00003F00001FC0000FE00003F00001F80000F 8000030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FF FFE0FFFFE07FFFC0130C7E9318>I<600000F80000FC00007E00003F80001FC00007E00003F000 01FC0000FE00003F00001F80001F80003F0000FE0001FC0003F00007E0001FC0003F80007E0000 FC0000F8000060000011187D9918>I<00700000F80000F80000D80000D80001DC0001DC0001DC 00018C00038E00038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF 800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<01FCE003FEE007FFE00F07E01E03E03C01E07800E07000E07000E0F00000E00000E000 00E00000E00000E00000E00000E00000E00000F000007000E07000E07800E03C01E01E01C00F07 C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C 00E01C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C 00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>III<01F9C007FFC00FFFC01F0FC01C03C03C03C07801C070 01C07001C0F00000E00000E00000E00000E00000E00000E00FF0E01FF0E00FF0F001C07001C070 03C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C0141C7E9B18>I<7F07F0FF8FF87F07F0 1C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C0 1C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F 9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C07001C0F001C 1E001C3C001C38001C78001CF0001DF0001DF8001FF8001FBC001F1C001E1E001E0E001C0F001C 07001C07801C03801C03C01C01C07F03F0FF87F87F03F0151C7F9B18>75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E00707FFF F0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D81C0 1D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C01C39C0 1C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18> I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F007FFF 003FFE000FF800111C7D9B18>II<0FF8003FFE007FFF00780F00 700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380 E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00001C00001E00 000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03801C03C01C01 C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03801C03801C03 801C03801C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07F3801FFF803F FF807C1F80700780F00380E00380E00380E00000F000007800003F00001FF0000FFE0001FF0000 1F800003C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FFFF00E7FC0013 1C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E07038007000007000007000007000 007000007000007000007000007000007000007000007000007000007000007000007000007000 00700007FF0007FF0007FF00151C7F9B18>IIII<7F8FE07F9FE07F8FE00E07000F0700070E00078E 00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00038E 00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>II<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C00 00780000700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E0 7000E0FFFFE0FFFFE0FFFFE0131C7E9B18>II93 D<7FFF00FFFF80FFFF807FFF0011047D7F18 >95 D<1FE0003FF8007FFC00783E00300F0000070000070001FF000FFF003FFF007F0700780700 F00700E00700E00700F00F00783F007FFFF03FFBF00FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000EFF800FFFC00FE3E00F80 F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFF C00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300780000700000F00000E0 0000E00000E00000E00000F000007000007801C03C01C03F07C01FFF8007FF0001FC0012147D93 18>I<001F80003F80001F8000038000038000038000038000038003F3800FFB801FFF803E1F80 780F80700780F00780E00380E00380E00380E00380E00380E00780F00780700780780F803E3F80 1FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F00780780700380F003C0E001 C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01FFF8007FF0001FC001214 7D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C0007FFFC0FFFFC0FFFFC001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F303C0F00380700380700 3807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00001FFF001FFFC03FFFE0 7801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF8003FE00151F7F9318> I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000EFF800FFFC00FE1E00F80 E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3 FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C00003800000000000000000000000 00007FC000FFC0007FC00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C00 3800000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C003C6038F078FFF07FE03FC00E277E9C18>II<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFFFC001F1F1C 001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C 1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E3F00FEFF80 7FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001 C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE 0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00380E00380E 00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00000E00000E 00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<03F3800FFB801FFF80 3E1F80780F80700780F00780E00380E00380E00380E00380E00380E00380F00780700780780F80 3E1F801FFF800FFB8003F380000380000380000380000380000380000380000380003FF8003FF8 003FF8151E7E9318>I<7F87E0FF9FF87FBFF803FC7803F03003E00003C00003C00003C0000380 000380000380000380000380000380000380000380007FFE00FFFF007FFE0015147F9318>I<0F F7003FFF007FFF00F81F00E00700E00700F007007C00007FF0001FFC0007FE00001F00600780E0 0380F00380F00780FC0F00FFFF00FFFE00E7F80011147D9318>I<018000038000038000038000 0380007FFFC0FFFFC0FFFFC0038000038000038000038000038000038000038000038000038000 0380400380E00380E00381E003C3C001FFC000FF80007E0013197F9818>I<7E07E0FE0FE07E07 E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01 E00F07E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E 0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000 700015147F9318>II<7F8F F07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F80001DC00039E 00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E 03800E0380070380070700070700038700038600038E0001CE0001CE0000CC0000CC0000DC0000 780000780000780000700000700000700000F00000E00079E0007BC0007F80003F00001E000015 1E7F9318>I<3FFFF07FFFF07FFFF07001E07003C0700780000F00001E00007C0000F80001F000 03E0000780000F00701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0003F E0007FE000FC0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0001E0007FC000FF8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000FC00007FE0003FE00007E013247E9F18>I<7C0000FF8000FF C00007E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 F000007FC0003FE0003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00007E000FFC000FF80007C000013247E9F18>125 D E /Fu 73 124 df<003F0F0000FFBF8003C3F3C00703E3C00703C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 7F87FC007F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C00703C00E01800E00000E00 000E00000E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D809C17>I<003FC000FFC003 C3C00703C00701C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0FFFFC00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF87F CFF8151D809C17>I<003F03F00000FFCFF80003C0FC1C000701F03C000701F03C000E00E01800 0E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC00FFFFFFFC000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF807FC7FCFF80211D 809C23>I<7070F8F8FCFCFCFC7C7C0C0C0C0C0C0C181818183030606040400E0D7F9C15>34 D<70F8FCFC7C0C0C0C1818306040060D7D9C0C>39 D<00C00180030006000E000C001C00180038 00300030007000700060006000E000E000E000E000E000E000E000E000E000E000E000E0006000 60007000700030003000380018001C000C000E0006000300018000C00A2A7D9E10>II<018001C0018001806186F99F7DBE1FF807E007E01FF87DBEF99F61860180 018001C0018010127E9E15>I<0006000000060000000600000006000000060000000600000006 0000000600000006000000060000000600000006000000060000FFFFFFE0FFFFFFE00006000000 060000000600000006000000060000000600000006000000060000000600000006000000060000 00060000000600001B1C7E9720>I<70F0F8F8781818183030706040050D7D840C>II<70F8F8F87005057D840C>I<00030003000700060006000E000C001C00180018 00380030003000700060006000E000C000C001C001800380030003000700060006000E000C000C 001C001800180038003000700060006000E000C000C00010297E9E15>I<03C00FF01C38381C38 1C700E700E700EF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00F700E700E700E 381C381C1C380FF007E0101D7E9B15>I<030007003F00FF00C700070007000700070007000700 070007000700070007000700070007000700070007000700070007000700FFF8FFF80D1C7C9B15 >I<07C01FF03878603C601EF01EF80FF80FF80F700F000F000E001E001C003C0078007000E001 C0038007000E030C03180330067FFEFFFEFFFE101C7E9B15>I<6000007FFF807FFF807FFF0060 0300C00600C00C00C00C0000180000300000300000600000600000C00000C00001C00001C00003 8000038000038000038000078000078000078000078000078000078000078000030000111D7E9B 15>55 D<03C00FF01C38381C781C700EF00EF00EF00FF00FF00FF00FF00F700F701F781F383F1F EF0FCF000E000E000E301C781C7838703030F03FC00F80101D7E9B15>57 D<70F8F8F870000000000000000070F8F8F87005127D910C>I<70F8F8F8700000000000000000 70F0F8F8781818183030706040051A7D910C>I<7FFFFFC0FFFFFFE00000000000000000000000 000000000000000000000000000000000000000000FFFFFFE07FFFFFC01B0C7E8F20>61 D<0FE03FF8703C601CF01EF01EF01E001E003C007800E001C00180038003000300030003000300 0200000000000000000007000F800F800F8007000F1D7E9C14>63 D<00060000000F0000000F00 00000F0000001F8000001F8000001F8000001F80000033C0000033C0000033C0000061E0000061 E0000061E00000C0F00000C0F00000C0F000018078000180780001FFF80003FFFC0003003C0003 003C0006001E0006001E0006001E001F001F00FFC0FFF0FFC0FFF01C1D7F9C1F>65 DI<001F808000FFE18003F0338007801B800F000F801E0007801C 0003803C000380780003807800018070000180F0000180F0000000F0000000F0000000F0000000 F0000000F0000000F00000007000018078000180780001803C0001801C0003001E0003000F0006 0007800C0003F0380000FFF000001F8000191E7E9C1E>III< FFFFF8FFFFF80F00780F00380F00180F001C0F000C0F000C0F030C0F030C0F03000F03000F0700 0FFF000FFF000F07000F03000F03000F03000F03000F00000F00000F00000F00000F00000F0000 FFF800FFF800161C7E9B1B>I<001F808000FFE18003F0338007801B800F000F801E0007801C00 03803C000380780003807800018070000180F0000180F0000000F0000000F0000000F0000000F0 000000F000FFF0F000FFF07000078078000780780007803C0007801C0007801E0007800F000780 07800F8003F0398000FFF080001FC0001C1E7E9C21>III76 DII<003F800000FFE00003E0F80007803C000E000E001E00 0F003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F00001E0F0 0001E0F00001E0F00001E0F00001E0780003C0780003C0780003C03C0007803C0007801E000F00 0F001E0007803C0003E0F80000FFE000003F80001B1E7E9C20>I I82 D<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F00000F0 00007C00007FC0003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001C0C0 01C0E00180E00380F00300FC0E00CFFC0083F800121E7E9C17>I<7FFFFFC07FFFFFC0780F03C0 700F01C0600F00C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F000003FFFC0003FFFC001B1C7F9B1E>II87 D<7FF0FFC07FF0FFC007C03E0003C0380003E0300001E0700001F0600000F0C0000079C000007D 8000003F0000001F0000001F0000000F0000000F8000001F8000003BC0000033E0000071E00000 61F00000C0F80001C0780001807C0003003C0007001E000F801F00FFE0FFF0FFE0FFF01C1C7F9B 1F>II<7FFFF07FFFF07E01E07803E07003C0700780600780600F00601F00601E00003C0000 7C0000780000F00000F00001E00003E03003C0300780300F80300F00701E00701E00603C00E07C 01E07807E0FFFFE0FFFFE0141C7E9B19>I<08081818303060606060C0C0C0C0C0C0F8F8FCFCFC FC7C7C38380E0D7B9C15>92 D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE001FFE 003E0E00780E00F00E00F00E60F00E60F01E60783E603FFFC01F878013127F9115>97 DI<03F00FF81E3C383C78187000F000F000F000F000F000 F000780078063C061E0C0FF803E00F127F9112>I<001F80001F80000380000380000380000380 00038000038000038000038000038003E3800FFB801E0F80380780780380700380F00380F00380 F00380F00380F00380F003807003807803803807801E1F800FFBF007E3F0141D7F9C17>I<03E0 0FF01C38381C781E700EFFFEFFFEF000F000F000F000700078063C061E0C0FF803E00F127F9112 >I<007801FC039E071E0E0C0E000E000E000E000E000E00FFE0FFE00E000E000E000E000E000E 000E000E000E000E000E000E000E000E007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3D C0381C00781E00781E00781E00781E00381C003C3C003FF00037E0007000007000003000003FFC 001FFF003FFF80700780E001C0E001C0E001C0E001C07003803C0F001FFE0007F800121C7F9215 >II<18003C007C003C0018000000000000000000000000 00FC00FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D 7F9C0C>I<01C003E003E003E001C00000000000000000000000000FE00FE000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0F1C0F1C07F803E 000B25839C0D>IIIII<03F0000FFC001E1E00380700780780700380F003C0F003C0F003 C0F003C0F003C0F003C07003807807803807001E1E000FFC0003F00012127F9115>II<03E1800FF9801E1F803C0780780780780380F00380F00380F00380F00380F00380F00380 7803807807803C07801E1F800FFB8007E380000380000380000380000380000380000380001FF0 001FF0141A7F9116>II<1F903FF07070E030E030E030F8007F803FE00FF000F8C038C0 38E038E038F070DFE08FC00D127F9110>I<0C000C000C000C000C001C001C003C00FFE0FFE01C 001C001C001C001C001C001C001C001C301C301C301C301C301E600FC007800C1A7F9910>IIII<7F8FF07F8FF00F0780070600038E0001DC0001D80000F00000700000780000F80001 DC00038E00030E000607000F0380FF8FF8FF8FF81512809116>II<7FFC7FFC78 38707060F060E061C063C00380070C0F0C0E0C1C1C3C1838187078FFF8FFF80E127F9112>II E /Fv 74 123 df<0000F000F8F001F8F003F8F00780000700000F 00000F00000F00000F00000F00000F00000F0000FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F 00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F 00F0142180A018>12 D<00F8F001F8F003F8F00780F00700F00F00F00F00F00F00F00F00F00F00 F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00 F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F01420809F18>I<007000E0 01C00380078007000E001E001E003C003C003C0078007800780078007000F000F000F000F000F0 00F000F000F000F000F000F000F000700078007800780078003C003C003C001E001E000E000700 0780038001C000E000700C2E7EA112>40 DI< 018001C001800180C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C001 8010147DA117>I<00060000000600000006000000060000000600000006000000060000000600 0000060000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF00006 000000060000000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000000600001C207D9A23>I<787878781830306060E0050A7D 830D>III<00030003000700060006000E 000C000C001C0018001800380030003000700060006000E000C000C001C0018001800180038003 0003000700060006000E000C000C001C0018001800380030003000700060006000E000C000C000 102D7DA117>I<03F00007F8001FFE001E1E003C0F00380700780780780780700380F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07807 807807807807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>I<00C001C00FC0FF C0FFC0F3C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I<07F0000FFC001FFE00383F00700F0060 0780E00780E003C04003C04003C00003C00003C00003C0000780000780000F00000E00001C0000 3C0000780000E00001C0000380000700000E00001C0000380000700000FFFFC0FFFFC0FFFFC012 1F7E9E17>I<03F0000FFC001FFE003C1F00780F00300780200780000780000780000780000F00 000F00003E0003FC0003F80003FC00001E00000F000007800003800003C00003C00003C00003C0 8003C0C003C0C00780700F807C1F003FFE000FFC0003F00012207E9E17>I<003E00003E00005E 00005E0000DE0001DE00019E00039E00039E00079E00071E000F1E000E1E001E1E003C1E003C1E 00781E00781E00F01E00FFFFF0FFFFF0FFFFF0001E00001E00001E00001E00001E00001E00001E 00001E00141E7F9D17>I<7FFF007FFF007FFF0078000078000078000078000078000078000078 000079F0007FFC007FFE007F1F007C07007C07807807800003C00003C00003C00003C00003C000 03C00003C0400780600780F00F007C1E003FFC001FF80007E000121F7E9D17>I<007C0001FE00 03FE000782000F00001E00003C00003C000078000078000078F800F3FC00F7FE00FE1F00FC0F80 F80780F80780F003C0F003C0F003C0F003C0F003C07003C07803C07803C07807803807803C0F00 1E1E000FFE0007F80001F00012207E9E17>I<03F0000FFC001FFE003E1F003C0F007807807807 807807807807807807807807803C0F001E1E000FFC0007F8000FFC001F3E003C0F007807807807 80F003C0F003C0F003C0F003C0F003C0F003C07807807C0F803E1F001FFE000FFC0003F0001220 7E9E17>56 D<03F00007F8000FFC001E1E003C0F00780700780780F00780F00380F003C0F003C0 F003C0F003C0F003C0F003C07807C07807C07C0FC03E1FC01FFBC00FF3C007C780000780000780 000700000F00001E00201E00307C007FF8003FF0000FC00012207E9E17>II61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1E00000E1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007803C000780780007C0 780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C0000780000 780000780000F00000F00000F00000F00000F00000F00000F00000F00000780000780000780000 7C00003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FF F801FFFE03E03E07800E0F00001E00003E00003C00007C0000780000780000780000F00000F000 00F00000F00000F00000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E00 1E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E>III75 DIII<003F000000FFC00003 FFF00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800078078000780 F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007 C07800078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FF F00000FFC000003F00001A227DA021>II<003F000000FFC00003FFF00007E1F8000F807C001F003E001E001E003C000F003C000F00 780007807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003 C0F00003C0F00003C0F00003C07800078078000780781E07803C0F0F003C0F8F001E079E001F03 FE000F83FC0007E1F80003FFF00000FFF800003F780000007C0000003E0000001E0000001F0000 000F801A277DA021>II<01FC0007 FF800FFFC01F03C03C00C03C00007800007800007800007800007800007C00003C00003F00001F E0000FFC0007FE0001FF00003F800007C00003C00003E00001E00001E00001E00001E00001E000 01C0C003C0F007C0FC0F807FFF001FFE0003F80013227EA019>IIIII<780007807C000F003E001F001E001E 000F003C000F807C000780780003C0F00003E1F00001F1E00000F3C000007FC000007F8000003F 0000001F0000001E0000003F0000007F8000007FC00000F3C00001F1E00001E0F00003C0F80007 C0780007803C000F003E001F001E001E000F003C000F807C0007C0780003C0F00003E01B207F9F 1E>IIII93 D<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3FEF 1F8F10147E9316>97 DI<03F00FFC 1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E0F1FFF0FFE03F010147E 9314>I<0007800007800007800007800007800007800007800007800007800007800007800007 8007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F00780F00780F00780F007 80F00780780780780F803E1F801FFF800FF78007C78011207E9F17>I<03F0000FFC001FFE003E 1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000F000007000007800003C01003E 07001FFF0007FE0001F80011147F9314>I<007E01FE03FE078007000F000F000F000F000F000F 000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00780F00780F00780F00780F0078 0F003C1E003E3E001FFC003FF80033E0003000003800003FFE003FFF801FFFC03FFFE07803F0F0 00F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F9317>III<01E001E001E001E00000000000000000 000000000000000001E001E001E001E001E001E001E001E001E001E001E001E001E001E001E001 E001E001E001E001E001E001E001E001E001E0C3C0FFC0FF803F000B29839F0C>IIIII<01F8 0007FE001FFF803F0FC03C03C07801E07801E0F000F0F000F0F000F0F000F0F000F0F000F07801 E07801E03C03C03F0FC01FFF8007FE0001F80014147F9317>II<03C7800FF7801FFF803E1F807C0F80780780780780F00780F00780F00780F00780F00780 F00780F807807807807C0F803E1F801FFF800FF78007C780000780000780000780000780000780 000780000780000780000780111D7E9317>II<07F01FFC3FFC780C78007800 78007C003FC01FF00FF803F8007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E001E00 1E001E001E001E00FFF0FFF0FFF01E001E001E001E001E001E001E001E001E001E001E001E001E 001E201FF00FF007C00C1A7F9910>IIII<7801E07C03C03E07801E0F000F0F00079E 0003FC0003F80001F80000F00001F00001F80003FC00079E000F0F000E0F001E07803C03C07801 E0F801F01414809315>II<7FFF7FFF7FFF003E003C0078 00F800F001E003E007C007800F001F001E003C007C00FFFFFFFFFFFF10147F9314>I E /Fw 35 90 df<7FE07FE0FFC00B037E8A0F>45 D<3078F8787005057C840D>I<007E0001FF00 03C3800701C00E01C00E01E01C00E01C01E03C01E03C01E07801E07801E07801E07801E07801E0 F003C0F003C0F003C0F003C0F003C0F00380F00780E00780E00700E00F00E00E00701E00701C00 3878003FF0000FC000131F7C9D17>48 D<000C001C00FC0FF80F38003800380038003800700070 007000700070007000E000E000E000E000E000E001C001C001C001C001C001C003C07FFEFFFE0F 1E7C9D17>I<003F0000FFC001C1E00300F00600780600780F007C0F807C0F807C0F0078060078 0000F80000F00001F00001E00003C0000780000F00001C0000380000700000C000018000030060 0600601C00C03800C07FFFC0FFFF80FFFF80161E7E9D17>I<007F0001FFC00383E00701E00700 F00F00F00F01F00F01F00001E00001E00003C0000780000F0000FE0000F800000E000007000007 800003C00003C00003C03007C07807C0F807C0F807C0F00F80C00F00601E00383C001FF80007E0 00141F7D9D17>I<0001C00003C00003C0000780000F80001B80003B8000738000638000C70001 87000307000707000E07000C0700180E00300E00600E00E00E00FFFFF0FFFFF0001C00001C0000 1C00001C00001C00001C00003C0003FFC003FFC0141E7D9D17>I<03007003FFF003FFE003FF80 03FE000600000600000600000600000600000600000C7C000DFF000F87800E03800C01C00C01C0 0001E00001E00001E00001E07003C0F003C0F003C0F00380C00780E00700600E00383C001FF000 0FC000141F7D9D17>I<000F80007FC000F0E001C0E00381E00701E00E00C01E00001C00003C00 0038000079FC007BFE007E07007C0380F80380F803C0F003C0F003C0F003C0F003C0F00780E007 80E00780E00700E00F00700E00701C003838001FF00007C000131F7C9D17>I<3000003FFFE07F FFE07FFFC0600180E00300C00600C00C0000180000180000300000600000C00001C00001800003 80000300000700000600000E00000E00001E00001C00001C00003C00003C00003C000078000078 0000780000300000131F799D17>I<003F0000FF8001C1E00380E00700700E00700E00700E0070 0E00E00F00E00F81C00FC38007F70003FC0001FC0003FF000F3F801C0F803807C03803C07001C0 7001C0E001C0E001C0E001C0E003807003007007003C1C001FF80007E000141F7D9D17>I<007E 0001FF000383800701C00E01C01E01C01C01E03C01E03C01E07801E07801E07801E07801E07803 E07803E03807C03807C01C0FC01FFBC00FE380000780000780000700000E00600E00F01C00F038 00E07000E0E0007FC0003F0000131F7C9D17>I<0000180000003800000038000000780000007C 000000FC000000FC000001BC000001BC0000033C0000033E0000061E0000061E00000C1E00000C 1E0000181E0000181F0000300F0000300F0000600F00007FFF0000FFFF0000C00F000180078001 800780030007800300078006000780060007801F0007C0FFC07FFCFFC07FFC1E207E9F22>65 D<07FFFF0007FFFFC0003C01E0003C01F0007800F8007800F8007800F8007800F8007800F80078 00F000F001F000F003E000F007C000F00F8000FFFE0000FFFF8001E007C001E003E001E003E001 E001E001E001E001E001F003C001E003C003E003C003E003C007C003C007C003C00F8007C03F00 7FFFFE00FFFFF0001D1F7E9E20>I<0003F808000FFE18003F0738007801F801F000F803C000F0 07C00070078000700F0000701F0000701E0000703E0000603C0000607C0000007C0000007C0000 007C000000F8000000F8000000F8000000F8000000F80000C0780000C0780001807C0001803C00 03003C0003001E0006001E000C000F80180007E0700001FFC000007F00001D217B9F21>I<07FF FF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E0078001E0078001E0078001F00 F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E0003E01E0003E 01E0003C01E0007C03C0007803C000F003C000F003C001E003C003C003C00F8007C03F007FFFFC 00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C00780078003800780038007800 38007800380078003800780C3000F0183000F0180000F0180000F0380000FFF80000FFF80001E0 700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C0006003C000E003 C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07FFFFF807FFFFF8003C00F800 3C0078007800380078003800780038007800380078003800780C3000F0183000F0180000F01800 00F0380000FFF80000FFF80001E0700001E0300001E0300001E0300001E0300001E0000003C000 0003C0000003C0000003C0000003C0000003C0000007C000007FFE0000FFFE00001D1F7E9E1E> I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C00078078000380F0000381F00 00381E0000383E0000303E0000307C0000007C0000007C0000007C000000F8000000F8000000F8 007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C03C0003C01E0003C00F0007C0 07800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FFC007FFC7FFC0003C00780000 3C007800007800F000007800F000007800F000007800F000007800F000007800F00000F001E000 00F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E003C00001E003C00001E003C0 0001E003C00001E003C00001E003C00003C007800003C007800003C007800003C007800003C007 800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22>I<07FFE007FFE0003C0000 3C0000780000780000780000780000780000780000F00000F00000F00000F00000F00000F00001 E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003C00007C000FF FC00FFFC00131F7F9E10>I<07FFE0FFE007FFE0FFE0003C003E00003C00380000780070000078 00E00000780180000078030000007806000000780C000000F018000000F030000000F060000000 F1F0000000F3F0000000F778000001EC78000001F878000001F03C000001E03C000001E01E0000 01E01E000003C00F000003C00F000003C00F000003C007800003C007800003C003C00007C007E0 007FFC1FFC00FFFC3FFC00231F7E9E23>75 D<07FFF00007FFF000003C0000003C000000780000 007800000078000000780000007800000078000000F0000000F0000000F0000000F0000000F000 0000F0000001E0000001E0000001E0000001E0018001E0018001E0030003C0030003C0030003C0 070003C0060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>I<07FC0000FFC0 07FC0001FFC0003E0001F800003E00037800006E0003F000006E0006F000006E0006F000006E00 0CF0000067000CF00000670018F00000C70019E00000C70031E00000C70031E00000C70061E000 00C38061E00000C380C1E000018380C3C00001838183C00001838183C0000181C303C0000181C3 03C0000181C603C0000301C60780000301CC0780000301CC0780000300F80780000300F8078000 0700F00780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07FC03FFC007FC03 FFC0003E007C00003E003800006F003000006F003000006F003000006780300000678030000063 C0300000C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F060000180F0C00001 8078C000018078C00001803CC00001803CC00001803CC00003001F800003001F800003000F8000 03000F800003000F800007000780000F800700007FF0030000FFF0030000221F7E9E22>I<0003 F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F0000F01F0000F01E 0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001F0 F80001F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E001E003C000F0078 0007C1F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0003C01F0007800 F0007800F8007800F8007800F8007800F8007800F800F001F000F001F000F001E000F003C000F0 0F8000FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C0000003C0000003 C0000003C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F>I<0003F80000 1FFE00003C1F0000F0078001E003C003C001E0078001E00F8001F00F0000F01F0000F01E0000F8 3E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001F0F80001 F0F80003E0780003E0780003C0780007C07C1E07803C3F0F003C619E001E61BC000FC0F80007E1 F00003FFC04000FEC0400000E0400000E0800000E1800000FF800000FF000000FF0000007E0000 003C001D297B9F23>I<07FFFC0007FFFF00003C07C0003C03E0007801E0007801F0007801F000 7801F0007801F0007801E000F003E000F003C000F0078000F01F0000FFFC0000FFF00001E03800 01E03C0001E01C0001E01C0001E01E0001E01E0003C03E0003C03E0003C03E0003C03E0003C03E 0603C03E0607C03E0C7FFC1F18FFFC0FF8000003E01F207E9E21>I<003F04007FCC01E0FC0380 7C03003C0700380600180E00180E00180E00180E00000F00000F00000FE00007FE0003FF8001FF C0007FE00007E00001E00000E00000F00000F06000E06000E06000E06000E07001C0700180F803 80FE0F00C7FC0081F80016217D9F19>I<1FFFFFF81FFFFFF81E03C0F83803C038380780383007 803870078018600780186007803860078030C00F0030000F0000000F0000000F0000000F000000 0F0000001E0000001E0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000 003C0000003C0000003C0000007C00001FFFF0003FFFF0001D1F7B9E21>IIII<7FF803FF80FFF8 03FF0007C000F80007C000E00003C000C00003E001C00001E003800001F003000000F006000000 F80E000000F80C00000078180000007C300000003C700000003E600000001EC00000001F800000 000F800000000F000000000F000000000F000000000E000000001E000000001E000000001E0000 00001E000000001E000000001C000000003C00000003FFE0000007FFE00000211F7B9E22>89 D E /Fx 25 122 df<60F0F0703030306060C040040B7D830B>44 D<03000700FF00FF00070007 0007000700070007000700070007000700070007000700070007000700070007000700FFF0FFF0 0C197D9813>49 D<0070007000F000F001F003F00370067006700C701C701870307030706070E0 70FFFFFFFF0070007000700070007007FF07FF10197F9813>52 D<07801FE0387030306038E018 E018E01CE01CE01CE01CE01C603C703C307C3FDC1F9C00180018003830307870786071C03F801F 000E1A7E9813>57 D69 D<0FFF0FFF007800780078007800780078007800780078 007800780078007800780078007800787078F878F878F878F0F070E03FC00F80101B7F9914>74 D80 D<07C21FF6383E701E600EE00EE006E006E006F00078007F803FF01FF80FFC 01FE001E000F0007C007C007C007E007E006F00EFC1CCFF883E0101C7E9A15>83 D86 D<3F80FFC0F0E0F070607000700FF03FF07870E070E070E073E07370F37FFE3E3C10107E8F13> 97 DI<007E00007E00000E00000E00000E00000E00000E00000E00000E00000E 0007CE001FFE003C1E00700E00700E00E00E00E00E00E00E00E00E00E00E00E00E00700E00701E 003C3E001FFFC007CFC0121A7F9915>100 D<07C01FF038387018701CFFFCFFFCE000E000E000 E0007000300C3C180FF007E00E107F8F11>I<00F801FC03BC073C0E180E000E000E000E000E00 FFC0FFC00E000E000E000E000E000E000E000E000E000E000E000E007FE07FE00E1A80990C>I< FC0000FC00001C00001C00001C00001C00001C00001C00001C00001C00001CF8001FFC001F1E00 1E0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E00FF9FC0FF9FC0 121A7F9915>104 D<18003C003C001800000000000000000000000000FC00FC001C001C001C00 1C001C001C001C001C001C001C001C001C00FF80FF80091A80990A>I108 DII<07E01FF8381C700E6006E007E007E007E007 E007E007700E700E3C3C1FF807E010107F8F13>II114 D<1F207FE070E0E060E060F0007F003FC01FE0 01F0C070C070E070F0E0FFC08F800C107F8F0F>I<0C000C000C000C001C001C003C00FFC0FFC0 1C001C001C001C001C001C001C001C601C601C601C601C600FC007800B177F960F>I121 D E /Fy 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F80 1F000B107F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF 80FF8009107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FF C0FFC00B107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F 801F000B107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001F E01FE00B107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F 803F000B107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F 801F000B107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C 001C0008000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060 C03F801F000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C071 807F003E000B107F8F0F>I E /Fz 32 122 df<000FE0FE00007FF3FF0000F87F8F8001E0FF1F 8003E0FE1F8007C0FE1F8007C07E0F0007C03E000007C03E000007C03E000007C03E000007C03E 0000FFFFFFF000FFFFFFF00007C03E000007C03E000007C03E000007C03E000007C03E000007C0 3E000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007C03E000007 C03E000007C03E000007C03E00003FF9FFE0003FF9FFE0002120809F1E>11 D45 D<07F8001FFE00381F80780F80FC0FC0FC0FC0FC0FC078 0FC0301F80001F00003E00007C0000700000E00000E00000C00000C00000C00000C00000C00000 C00000000000000000000000000001C00007F00007F00007F00007F00007F00001C00012207D9F 19>63 D<0000E000000000E000000001F000000001F000000001F000000003F800000003F80000 0007FC00000007FC0000000FFE0000000CFE0000000CFE000000187F000000187F000000307F80 0000303F800000703FC00000601FC00000601FC00000C01FE00000C00FE00001FFFFF00001FFFF F000018007F000030003F800030003F800060003FC00060001FC000E0001FE00FFE01FFFE0FFE0 1FFFE0231F7E9E28>65 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E 3F00000E3F00000E7F0000067E0000067E000006FE000000FE000000FE000000FE000000FE0000 00FE000000FE0000007E0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E0 003803F0007000FE01C0003FFF800007FC001F1F7D9E26>67 D 70 D<03FC080FFF381E03F83C00F8780078780038F80038F80018FC0018FC0000FF0000FFF800 7FFF007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC0000FCC0007CC0007CC0007C E00078E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83 D<7FFFFFFC7FFFFFFC7C0FE07C70 0FE01C600FE00C600FE00CE00FE00EC00FE006C00FE006C00FE006C00FE006000FE000000FE000 000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE0 00000FE000000FE000000FE000000FE000000FE00007FFFFC007FFFFC01F1E7E9D24>I<07FC00 1FFF003F0F803F07C03F03E03F03E00C03E00003E001FFE00FFFE03F83E07E03E07C03E0F803E0 F803E0F803E0FC07E07E0DE03FF9FE07E07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC0000 FC0000FC0000FC0000FC00007C00007E00003E00603F00C01F81C007FF0001FC0013147E9317> I<0007F80007F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F801F8 F80FFEF81F83F83E01F87E00F87C00F87C00F8FC00F8FC00F8FC00F8FC00F8FC00F8FC00F87C00 F87C00F87E00F83E01F81F07F80FFEFF03F8FF18207E9F1D>I<01FE0007FF801F83E03F01F07E 00F07E00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007E00003E00183F00380F 807007FFE000FF8015147F9318>I<001F8000FFC001F3E003E7E003C7E007C7E007C3C007C000 07C00007C00007C00007C000FFFC00FFFC0007C00007C00007C00007C00007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C0003FFC003FFC0013207F9F10> I<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800FFF0019FC 001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C70003EF0001EF0001EF0001E7800 3C78003C3F01F80FFFE001FF00171E7F931A>II<1C003F007F007F007F003F001C00000000000000000000000000FF00FF001F001F 001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B217EA00E> I107 DIII< 01FF0007FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007E 7C007C7C007C3E00F83E00F81F83F007FFC001FF0017147F931A>II<01F81807FE381F87783F01F83E01F87E00F87C00F8FC00F8FC00F8FC00F8FC00F8FC 00F8FC00F87C00F87E00F87E00F83F01F81F87F80FFEF803F8F80000F80000F80000F80000F800 00F80000F80000F80007FF0007FF181D7E931C>II<0FE63FFE701E600EE006E006F800FFC07FF83FFC1FFE03FE001FC007C0 07E007F006F81EFFFCC7F010147E9315>I<01800180018003800380038007800F803F80FFFCFF FC0F800F800F800F800F800F800F800F800F800F800F860F860F860F860F8607CC03F801F00F1D 7F9C14>IIIIII E /FA 79 124 df<000FC3E0007FEFF000E07C7801C0F8780380F830070070 00070070000700700007007000070070000700700007007000FFFFFF80FFFFFF80070070000700 700007007000070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381E00381E00701E0070000070000070000070000070000070000FFFF E0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E003 80E00700E00700E00700E00700E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E07F E7FE7FE7FE1720809F19>I<000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700 F00F00070070000007007000000700700000070070000007007000000700700000FFFFFFFF00FF FFFFFF000700700700070070070007007007000700700700070070070007007007000700700700 070070070007007007000700700700070070070007007007000700700700070070070007007007 0007007007007FE3FE3FF07FE3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06 180C180C381C3018603040200F0E7E9F17>34 D<0078000000FC00000186000003860000030300 00070300000703000007030000070300000706000007060000070C00000718000003B81FFC03F0 1FFC03E003E003C0018003C0030007C003000EE006001CE0060038700C0078780C0070381800F0 1C3000F01E3000F00E6000F007C000F003C0187801C0187807E0303E1E78701FF83FE007E00F80 1E227EA023>38 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D>I<006000C0018003000700 06000C001C00180038003800300070007000700060006000E000E000E000E000E000E000E000E0 00E000E000E000E0006000600070007000700030003800380018001C000C000600070003000180 00C000600B2E7DA112>II<018001C0018001 80C183E187F99F7DBE1FF807E007E01FF87DBEF99FE187C1830180018001C0018010147DA117> I<0006000000060000000600000006000000060000000600000006000000060000000600000006 00000006000000060000000600000006000000060000FFFFFFF0FFFFFFF0000600000006000000 060000000600000006000000060000000600000006000000060000000600000006000000060000 0006000000060000000600001C207D9A23>I<70F8FCFC7C0C0C0C181838306040060E7C840D>I< FFC0FFC0FFC00A037F8A0F>I<70F8F8F87005057C840D>I<00030003000700060006000E000C00 0C001C0018001800380030003000700060006000E000C000C001C0018001800180038003000300 0700060006000E000C000C001C0018001800380030003000700060006000E000C000C000102D7D A117>I<03F0000FFC001E1E001C0E00380700780780700380700380700380F003C0F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07003807003807003 807807803807001C0E001E1E000FFC0003F000121F7E9D17>I<018003801F80FF80E380038003 800380038003800380038003800380038003800380038003800380038003800380038003800380 03800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E00300F00600F80C00780F007C0F807C0 F803C0F803C02003C00007C0000780000780000F00000E00001C0000380000700000E00001C000 0380000300000600C00C00C01800C03001807FFF80FFFF80FFFF80121E7E9D17>I<03F0000FFC 001C1E00300F00780F80780F80780780380F80000F80000F00000F00001E00003C0003F80003F0 00001C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780600F80700F 003C1E001FFC0003F000121F7E9D17>I<000E00000E00001E00003E00003E00006E0000EE0000 CE00018E00038E00030E00060E000E0E000C0E00180E00380E00300E00600E00E00E00FFFFF0FF FFF0000E00000E00000E00000E00000E00000E00000E0000FFE000FFE0141E7F9D17>I<380300 3FFF003FFE003FF8003FE00030000030000030000030000030000030000031F00037FC003E0E00 3C07003807803003800003800003C00003C00003C06003C0F003C0F003C0F00380C00780600700 700E003C1E000FF80007E000121F7E9D17>I<007C0001FE000783000E03800C07801C07803803 00380000780000700000700000F3F800F7FE00FC0E00F80700F80380F80380F003C0F003C0F003 C0F003C0F003C07003C07003C07003803803803807001C07000E1E0007FC0001F000121F7E9D17 >I<6000007FFFC07FFFC07FFF80600180C00300C00600C00600000C0000180000180000300000 600000600000E00000C00001C00001C00001C00003800003800003800003800007800007800007 8000078000078000078000078000030000121F7D9D17>I<03F0000FFC001E1E00380700300700 7003807003807003807003807807003E07003F0E001FDC000FF80003F80007FC001E7E00383F00 700F80700780E003C0E001C0E001C0E001C0E001C0F001807003803807001E0E000FFC0003F000 121F7E9D17>I<03F00007F8001E1C00380E00380700700700700380F00380F00380F003C0F003 C0F003C0F003C0F003C07007C07007C0380FC01C0FC01FFBC007F3C00003800003800007800007 00300700780E00780E00701C003078001FF0000FC000121F7E9D17>I<70F8F8F8700000000000 000000000070F8F8F87005147C930D>I<70F8F8F8700000000000000000000070F8F8F8781818 18183030606040051D7C930D>I<7FFFFFE0FFFFFFF00000000000000000000000000000000000 000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018 F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000 C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0 0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 DI<001FC04000 7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0 780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8 1C00007FF000001FC0001A217D9F21>IIII<000FC020007FF86001F81CE003E006E0078003 E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C 0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F 24>III76 DII<001F800000FFF00001E0780007C03E000F80 1F000F000F001E0007803C0003C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F8 0001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E0 3C0003C03E0007C01E0007800F000F000F801F0007C03E0001F0F80000FFF000001F80001C217D 9F23>II82 D<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F000007800 007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0C001 C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE0780F 01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C1F7E 9E21>II II<7FF83FF87FF83FF807C00F8003C0060001E00E0001F00C0000F0180000783800007C 3000003C7000003E6000001EC000000FC000000F8000000780000007C0000007E000000DE00000 1DF0000018F8000038780000307C0000603C0000E01E0000C01F0001800F0003800780030007C0 0F8007C0FFE01FFEFFE01FFE1F1F7F9E22>I91 D<0804180C3018703860306030C060C060C060F87CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE0003FF000783800781C00300E00000E00000E00000E0003FE001FFE 003E0E00780E00F80E00F00E30F00E30F00E30F01E30783F703FEFE00F878014147E9317>97 D<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3F 000EFF800FC3C00F00E00E00F00E00700E00780E00780E00780E00780E00780E00780E00780E00 700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19>I<03F80FFC1E1E3C1E380C780070 00F000F000F000F000F000F0007000780038033C031F0E0FFC03F010147E9314>I<000380003F 80003F8000038000038000038000038000038000038000038000038000038003E3800FFB801E0F 803C0780780380780380700380F00380F00380F00380F00380F00380F003807003807003807807 803807801E1F800FFBF803E3F815207E9F19>I<03F0000FFC001E1E003C0F0038070078070070 0380F00380FFFF80FFFF80F00000F00000F000007000007800003801801C03800F070007FE0001 F80011147F9314>I<003C00FE01CF038F03060700070007000700070007000700FFF0FFF00700 0700070007000700070007000700070007000700070007000700070007007FF07FF01020809F0E >I<0001E003E3F00FFF701C1C703C1E00380E00780F00780F00780F00780F00380E003C1E001C 1C003FF80033E0003000003000003800003FFE001FFF803FFFC07003E07000F0E00070E00070E0 0070F000F07801E03E07C00FFF0003FC00141F7F9417>I<0E0000FE0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E3E000EFF000FC3800F01C00F01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC FFE7FC16207F9F19>I<1C001E003E001E001C000000000000000000000000000E007E007E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A1F809E0C> I<00E001F001F001F000E0000000000000000000000000007007F007F000F00070007000700070 00700070007000700070007000700070007000700070007000700070007000706070F0E0F1C07F 803E000C28829E0E>I<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18000E30000E70000EF0000FF8000F38 000E1C000E1E000E0E000E07000E07800E03800E03C0FFCFF8FFCFF815207F9F18>I<0E00FE00 FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE0FFE00B20809F0C>I<0E3F03F000FEFF8FF800FF C1DC1C000F80F80E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E 00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00FFC3800F01C00F01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FCFFE7FC 16147F9319>I<01F80007FE001E07803C03C03801C07000E07000E0F000F0F000F0F000F0F000 F0F000F0F000F07000E07801E03801C03C03C01E078007FE0001F80014147F9317>I<0E3F00FE FF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E00780E00780E00780E00780E00700E 00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E00000E00000E00000E00000E0000FF E000FFE000151D7F9319>I<03E1800FF9801E1F803C0780780780780380700380F00380F00380 F00380F00380F00380F003807003807803807807803C0F801E1F800FF38003E380000380000380 000380000380000380000380000380003FF8003FF8151D7E9318>I<0E78FEFCFF9E0F1E0F0C0F 000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00F147F9312>I<1F903F F07070E030E030E030F00078007F803FE00FF000F0C078C038C038E038E038F070DFE08F800D14 7E9312>I<06000600060006000E000E001E003E00FFF8FFF80E000E000E000E000E000E000E00 0E000E000E000E180E180E180E180E18073007E003C00D1C7F9B12>I<0E01C0FE1FC0FE1FC00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00E 03C0070FC007FDFC01F1FC16147F9319>III<7FC7FC7FC7FC0703E007038003830001C70000EE0000EC 00007800003800003C00007C0000EE0001C7000187000303800701C00F01E0FF87FEFF87FE1714 809318>II<3FFF3FFF380E301C703C6038607060F001E0 01C0038007830F030E031C073C063806701EFFFEFFFE10147F9314>II E /FB 44 122 df45 DI<00 7E0001FF8007FFE00FFFF01FC3F81F00F83E007C3E007C3C003C7C003E7C003E78001E78001EF8 001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8 001FF8001F78001E7C003E7C003E7C003E7C003E3E007C3F00FC1F00F81FC3F80FFFF007FFE001 FF80007E0018297EA71D>48 D<00180000380000F80007F800FFF800FFF800FFF800F8F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8 0000F8007FFFF07FFFF07FFFF07FFFF014287CA71D>I<00FE0003FF800FFFC01FFFE01F07F03C 01F87800FC78007C70007CF0003EF0003E60003E20003E00003E00003E00003E00007C00007C00 00780000F80001F00001E00003E00007C0000F80001F00003E00007C0000F80001F00003E00003 C0000780000F00001E00003C00007FFFFE7FFFFE7FFFFE7FFFFE17287EA71D>I<007E0003FF80 07FFE00FFFF01F81F83E00F87C007C78007C30007C10007C00007C00007C0000780000F80000F8 0001F00007E000FFC000FF8000FFC000FFF00001F80000FC00007C00003E00003E00001F00001F 00001F00001F00001F80001FC0003EE0003EF0007E7C00FC3F01F81FFFF00FFFE003FFC000FE00 18297EA71D>I<0007E0000FE0000BE0001BE0001BE0003BE0003BE00073E00073E000F3E000F3 E001E3E001E3E003E3E003C3E007C3E00F83E00F83E01F03E01F03E03E03E03E03E07C03E07C03 E0F803E0FFFFFFFFFFFFFFFFFFFFFFFF0003E00003E00003E00003E00003E00003E00003E00003 E00003E00003E018277EA61D>I<3FFFF83FFFF83FFFF83FFFF83E00003E00003E00003E00003E 00003E00003E00003E00003E00003E00003E3E003EFF803FFFE03FFFF03FC1F03F80F83F007C3E 007C00007C00003E00003E00003E00003E00003E00003E00003E20003E60007C70007CF800F87C 01F87F07F03FFFE00FFFC007FF0001FC0017287EA61D>I<000FE0003FF800FFF801FFF803F818 07E0000F80000F80001F00003E00003E00003C00007C00007C02007C3FC0F8FFE0F9FFF0FBFFF8 FF80FCFE007CFE003EFC003EFC003FFC001FF8001FF8001FF8001FF8001F78001F7C001F7C001F 7C001E3C003E3E003E1E007C1F00FC0FC1F807FFF003FFE001FF80007E0018297EA71D>II<007E0003FFC007FFE00FFFF01F81F83F00FC3E007C7E007E7C003E7C003E7C003E7C 003E7C003E3E007C3E007C1F00F80FC3F007FFE003FFC003FFC00FFFF01F81F83F00FC3E007C7C 003E7C003EF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E7E007E3F00FC1F81F80F FFF007FFE003FFC0007E0018297EA71D>I<007E0001FF8007FFC00FFFE01F83F01F00F83E0078 7C007C7C003C78003EF8003EF8003EF8001EF8001FF8001FF8001FF8001FF8003FFC003F7C003F 7C007F3E007F3F01FF1FFFDF0FFF9F07FF1F03FC1E00403E00003E00003E00007C00007C0000F8 0000F80001F01803E01E0FC03FFF801FFF000FFE0003F80018297EA71D>I66 D<0003FE00000FFFC0003FFFF000FFFFF001FE03F003F8006007E000 2007C000000F8000001F8000001F0000003E0000003E0000007C0000007C0000007C0000007C00 0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 0000007C0000007C0000007C0000007C0000003E0000003E0000001F0000001F8000000F800000 07C0000007E0000803F8003801FE01F800FFFFF8003FFFF0000FFFC00003FE001D2C7CAA25>I< FFFFE00000FFFFFC0000FFFFFF0000FFFFFF8000F8003FC000F8000FE000F80003F000F80001F8 00F80000F800F800007C00F800007E00F800003E00F800003E00F800001F00F800001F00F80000 1F00F800000F80F800000F80F800000F80F800000F80F800000F80F800000F80F800000F80F800 000F80F800000F80F800000F80F800000F00F800001F00F800001F00F800001F00F800003E00F8 00007E00F800007C00F80000FC00F80001F800F80003F000F8000FE000F8003FC000FFFFFF8000 FFFFFF0000FFFFFC0000FFFFE00000212A7BA92A>I73 D77 DI80 D<007FC00001FFF80007FFFE000FFFFF 001FC07F003F000F007E0006007C0000007C000000F8000000F8000000F8000000F8000000F800 0000FC0000007E0000007F0000003F8000001FF800000FFF000007FFE00003FFF80000FFFC0000 0FFE000000FF0000003F0000001F8000000F8000000FC0000007C0000007C0000007C0000007C0 000007C0000007C000000F8060000F80F0001F00FC003F00FF80FE007FFFFC001FFFF80007FFE0 0000FF80001A2C7DAA21>83 DI<01FC000F FF003FFF803FFFC03C07E03003E00001F00001F00001F00001F00001F00003F001FFF007FFF01F FFF03FC1F07E01F0FC01F0F801F0F801F0F801F0FC03F07E0FF07FFFF03FFFF01FF9F00FE1F014 1B7E9A1C>97 DI<007F8001FFE007FFF80FFFF81FC0783F 00103E00007C00007C00007C0000F80000F80000F80000F80000F80000F80000F800007C00007C 00007C00003E00003F00181FC0780FFFF807FFF801FFE0007F80151B7E9A1A>I<00007C00007C 00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C 00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C7C007C7C007CFC007CF8007CF8007CF8007C F8007CF8007CF8007CF8007CF8007C7C007C7C007C7E00FC3F00FC1F83FC1FFFFC0FFF7C03FE7C 01F87C162A7EA91E>I<00FC0003FF0007FF800FFFC01F87E03F01E03E00F07C00F07C0070F800 78FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800007800007C00007C00003E00003F00181FC0 F80FFFF807FFF801FFE0007F00151B7E9A1A>I<001FC0007FC000FFC001FFC003F00003E00007 C00007C00007C00007C00007C00007C00007C00007C00007C000FFFE00FFFE00FFFE00FFFE0007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C000122A7FA912>I<00F80F 0003FE7F800FFFFF801FFFFF801F07C0003E03E0003E03E0007C01F0007C01F0007C01F0007C01 F0007C01F0003E03E0003E03E0001F07C0001FFFC0003FFF80003BFE000038F800007800000078 0000007C0000003FFF80003FFFF0001FFFF8001FFFFC003FFFFE007C00FE007C003F00F8001F00 F8001F00F8001F00F8001F00FC003F007E007E003F81FC001FFFF8000FFFF00007FFE00000FF00 0019287E9A1D>III107 DIII<007E0001FF8007FFE00FFFF01F81F83F00FC3E00 7C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E7C00 3E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E00181B7E9A1D>II<00FC7C 03FF7C07FFFC0FFFFC1FC1FC3F00FC3E00FC7E007C7C007CFC007CF8007CF8007CF8007CF8007C F8007CF8007CF8007CFC007C7C007C7C007C7E007C3F00FC1FC3FC1FFFFC0FFF7C03FE7C01F87C 00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C16277E 9A1E>II<07F8001FFF003FFF807FFF807C 0780F80100F80000F80000F80000FC00007F00007FF0003FFC001FFE0007FF0000FF80001FC000 0FC00007C00007C04007C0E007C0FC1F80FFFF807FFF001FFE0007F800121B7E9A16>I<07C000 07C00007C00007C00007C00007C00007C000FFFF80FFFF80FFFF80FFFF8007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000 07C00007C08007E38003FFC003FFC001FF0000FC0012227FA115>IIII121 D E /FC 10 118 df<000000007FFC00000E0000000FFFFFC0001E0000007FFFFFF8003E 000003FFFFFFFE007E00000FFFFFFFFF80FE00003FFFF8007FC1FE0000FFFF80000FF3FE0003FF FC000003FFFE0007FFF0000001FFFE000FFFC00000007FFE001FFF800000003FFE003FFF000000 001FFE007FFC000000000FFE00FFFC000000000FFE01FFF80000000007FE03FFF00000000003FE 03FFE00000000003FE07FFE00000000001FE0FFFC00000000001FE0FFFC00000000000FE1FFF80 0000000000FE1FFF8000000000007E3FFF8000000000007E3FFF0000000000007E3FFF00000000 00003E7FFF0000000000003E7FFF0000000000003E7FFF0000000000003E7FFE00000000000000 7FFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00 000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000 000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000 FFFE000000000000007FFE000000000000007FFE000000000000007FFF000000000000007FFF00 00000000003E7FFF0000000000003E3FFF0000000000003E3FFF0000000000003E3FFF80000000 00003E1FFF8000000000007E1FFF8000000000007C0FFFC000000000007C0FFFC00000000000FC 07FFE00000000000FC03FFE00000000000F803FFF00000000001F801FFF80000000003F000FFFC 0000000003F0007FFE0000000007E0003FFF000000000FC0001FFF800000001F80000FFFC00000 003F000007FFF0000000FE000003FFFC000001FC000000FFFF80000FF80000003FFFF8007FE000 00000FFFFFFFFFC000000003FFFFFFFF00000000007FFFFFFC00000000000FFFFFE00000000000 007FFE00000047497AC754>67 D80 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFFF8000003FF003FFC000007FF80 0FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007FF8001FFC00007FF8001FFC000 03FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FFE00000000000FFE00000000000 FFE00000000000FFE00000000000FFE000000000FFFFE00000007FFFFFE0000007FFFFFFE00000 3FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE0000FFF0000FFE0001FFE0000FF E0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF80000FFE000FFF00000FFE000FFF0 0000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000FFF80003FFE0007FFC0007FFE0 007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC3FFFF003FFFFF01FFFF0007FFF C00FFFF00007FE0003FFF0342E7DAD38>97 D<00001FFF80000001FFFFF8000007FFFFFE00001F FFFFFF00007FF801FF8000FFE003FFC003FFC003FFC007FF8003FFC007FF0003FFC00FFE0003FF C01FFE0003FFC01FFC0001FF803FFC0000FF003FFC00007E007FFC000000007FF8000000007FF8 00000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000 00FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF8000000007FFC 000000007FFC000000007FFC000000003FFC000000003FFE000000001FFE000001F01FFF000001 F00FFF000003F007FF800007E007FFC00007C003FFE0000FC000FFF8003F80007FFF00FF00001F FFFFFC000007FFFFF0000001FFFFC00000001FFC00002C2E7CAD34>99 D<00FC0001FF0003FF80 07FF800FFFC00FFFC00FFFE00FFFE00FFFE00FFFC00FFFC007FF8003FF8001FF0000FC00000000 000000000000000000000000000000000000000000000000000000000000000000007FC0FFFFC0 FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 01FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC820>105 D<00FF8003FF000003FF000000FFFF80 1FFFE0001FFFE00000FFFF807FFFF8007FFFF80000FFFF81FFFFFC01FFFFFC0000FFFF83F01FFE 03F01FFE0000FFFF87C007FF07C007FF000003FF8F0007FF8F0007FF800001FF9E0003FF9E0003 FF800001FFBC0003FFBC0003FF800001FFF80001FFF80001FFC00001FFF80001FFF80001FFC000 01FFF00001FFF00001FFC00001FFE00001FFE00001FFC00001FFE00001FFE00001FFC00001FFE0 0001FFE00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFF FF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80592E7BAD62>109 D<00FF8007FE0000FFFF803FFFC000FFFF80FFFFF000FFFF81FFFFFC00FFFF87F01FFC00FFFF8F C00FFE0003FF9F0007FF0001FF9E0007FF0001FFBC0007FF0001FFF80003FF8001FFF80003FF80 01FFF00003FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC000 03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003 FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF 81FFFFFFFFFFFF81FFFFFF382E7BAD41>I<00000FFE0000000001FFFFF000000007FFFFFC0000 001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF00003FF00001FF80007FE0000 0FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC000007FF803FF8000003FF803F F8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0FFF8000003FFE0FFF8000003 FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8 000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07FF8000003FFC07FF8000003FF C07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007FF001FFC000007FF000FFE00 000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FFE000FFE000007FFC07FFC000 001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF000000332E7DAD3A>I<0001F000 000001F000000001F000000001F000000001F000000001F000000003F000000003F000000003F0 00000007F000000007F000000007F00000000FF00000000FF00000001FF00000003FF00000003F F00000007FF0000001FFF0000003FFF000000FFFFFFFC0FFFFFFFFC0FFFFFFFFC0FFFFFFFFC0FF FFFFFFC0007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000 007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF000 00007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF0 0000007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007F F001F0007FF001F0007FF003F0003FF803E0003FF807E0001FFC07C0001FFE0F80000FFFFF8000 03FFFE000000FFFC0000001FE00024427EC12D>116 D<007FC00000FF80FFFFC001FFFF80FFFF C001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF8003FFC00007FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0 0003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80 01FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC000 03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00007FF8001FFC00007FF8001FFC00007 FF8001FFC0000FFF8001FFC0001FFF8000FFC0003FFF8000FFE0007FFF80007FE000FBFFC0007F F803F3FFFF003FFFFFE3FFFF000FFFFF83FFFF0003FFFE03FFFF00007FF003FFFF382E7BAD41> I E /FD 8 117 df<00001E000000003E00000000FE00000003FE0000003FFE0000FFFFFE0000 FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00007FFFFFFF 807FFFFFFF807FFFFFFF807FFFFFFF80213879B730>49 D<0000001FFE0000E0000003FFFFE001 E000001FFFFFF803E000007FFFFFFE07E00001FFFC00FF0FE00007FFC0001FDFE0000FFF000007 FFE0003FFC000001FFE0007FF0000000FFE000FFE00000007FE001FFC00000003FE003FF800000 001FE007FF800000001FE007FF000000000FE00FFE0000000007E00FFE0000000007E01FFC0000 000007E01FFC0000000003E03FFC0000000003E03FF80000000003E07FF80000000001E07FF800 00000001E07FF80000000001E07FF0000000000000FFF0000000000000FFF0000000000000FFF0 000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FF F0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000 FFF00000000000007FF00000000000007FF80000000000007FF80000000001E07FF80000000001 E03FF80000000001E03FFC0000000001E01FFC0000000001E01FFC0000000003E00FFE00000000 03C00FFE0000000007C007FF0000000007C007FF800000000F8003FF800000000F8001FFC00000 001F0000FFE00000003E00007FF00000007C00003FFC000000F800000FFF000003F0000007FFC0 000FE0000001FFFC007FC00000007FFFFFFF000000001FFFFFFC0000000003FFFFE00000000000 1FFE0000003B3D7BBB46>67 D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF8 01FF80001FF800FFC0001FF8007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF0 0003C0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000FFFFF000000F FFFFF000007FF83FF00003FF803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0 007FE0003FF000FFE0003FF000FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0 007FF000FFE000FFF0007FE001DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FF E0003FE001FFE02B267DA52F>97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF 8007FC003F800FF8003FC01FF8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE000 0FF87FE0000FF8FFE0000FF8FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0 000000FFE0000000FFE0000000FFE00000007FE00000007FE00000007FF00000003FF00000783F F00000781FF80000F80FF80000F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00 000FFFFC000001FFC00025267DA52C>101 D<00FF0000000000FFFF0000000000FFFF00000000 00FFFF0000000000FFFF000000000007FF000000000003FF000000000003FF000000000003FF00 0000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000000 03FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000 00000003FF000000000003FF000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003 FF0F81FF800003FF3C00FF800003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007F E00003FFC0007FE00003FF80007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF 00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE0 0003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00 007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000 03FF00007FE00003FF00007FE00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1F FFFF80FFFFFC1FFFFF80313C7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFF FE03FF00FFFFF000FFC007FFE0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001F FC03FF00000FFC03FF00000FFE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF 000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007 FF03FF000007FF03FF00000FFE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF 00001FF803FF80003FF003FFC0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC 0003FF0FFFF00003FF01FF000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 0003FF0000000003FF00000000FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC0000003037 7EA536>112 D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FE E07FE003FEE07FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000 03FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF0000 0003FF00000003FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE 000023267EA528>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000 001F0000001F0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00 001FFFFFF0FFFFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF 000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003 FF000003FF000003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C 03FF003C03FF003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377E B626>116 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 75 386 a FD(Chapter)31 b(1)75 620 y FC(P)m(oin)m(t)39 b(to)g(P)m(oin)m(t) g(Comm)m(u)o(nication)75 886 y FB(1.1)59 b(Intro)r(duction)75 1038 y FA(Sending)13 b(and)f(receiving)h(of)e(messages)f(b)o(y)i(pro)q (cesses)g(is)g(the)f(basic)h(MPI)g(comm)o(unication)g(mec)o(hanism.)75 1095 y(All)20 b(other)f(comm)o(unication)h(functions)g(can)f(b)q(e)h (de\014ned)h(and)e(implemen)o(ted)i(on)e(top)g(of)g(this)g(basic)75 1151 y(comm)o(unication)e(la)o(y)o(er,)f(using)h(few)f(auxiliary)i(data)e (managemen)o(t)f(routines,)i(and)g(using)g(the)f(group)75 1207 y(and)f(comm)o(unicator)g(manipulation)i(functions.)166 1290 y(The)i(basic)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unication)g(op)q (erations)g(are)f Fz(send)g FA(and)h Fz(receiv)o(e)p FA(.)29 b(A)18 b Fz(send)75 1346 y FA(op)q(eration)13 b(creates)f(and)h(sends)g(a)g (message.)18 b(The)13 b(op)q(eration)g(sp)q(eci\014es)i(a)d Fz(send)i(bu\013er)f FA(in)h(the)e(sender)75 1403 y(memory)f(from)g(whic)o(h) h(the)g(message)f(data)g(is)h(tak)o(en.)18 b(In)12 b(addition,)h(the)f(send)g (op)q(eration)g(asso)q(ciates)f(an)75 1459 y Fz(en)o(v)o(elop)q(e)16 b FA(with)g(the)g(message.)22 b(This)16 b(en)o(v)o(elop)q(e)h(sp)q(eci\014es) h(the)e(message)g(destination)h(and)f(con)o(tains)75 1516 y(distinguishing)h (information)d(that)f(can)i(b)q(e)g(used)f(b)o(y)g(the)g Fz(receiv)o(e)g FA(op)q(eration)g(to)g(select)h(a)e(particular)75 1572 y(message.)166 1654 y(A)22 b Fz(receiv)o(e)f FA(op)q(eration)h(consumes)g(a)f(message.)39 b(The)22 b(message)f(to)g(b)q(e)i(receiv)o(ed)f(is)h(selected)75 1711 y(according)13 b(to)f(the)g(v)m(alue)i(on)e(its)g(en)o(v)o(elop)q(e,)i (and)f(the)f(message)g(data)g(is)h(put)f(in)o(to)g(the)h Fz(receiv)o(e)g (bu\013er)p FA(.)166 1793 y(The)f(next)g(sections)h(describ)q(e)g(the)g (basic)f(\(blo)q(c)o(king\))h(send)g(and)f(receiv)o(e)h(op)q(erations.)19 b(W)l(e)12 b(discuss)75 1850 y(send,)21 b(receiv)o(e,)g(basic)f(comm)o (unication)g(seman)o(tics,)h(t)o(yp)q(e)e(matc)o(hing)h(requiremen)o(ts,)h(t) o(yp)q(e)e(con)o(v)o(er-)75 1906 y(sion)14 b(in)h(heterogeneous)f(en)o (vironmen)o(ts,)f(and)h(more)g(general)g(comm)o(unication)g(mo)q(des.)20 b(Non)o(blo)q(c)o(king)75 1962 y(comm)o(unication)c(is)h(addressed)f(next,)f (follo)o(w)o(ed)h(b)o(y)g(c)o(hannel-lik)o(e)i(constructs)e(and)f (send-receiv)o(e)j(op-)75 2019 y(erations.)i(W)l(e)15 b(then)g(consider)h (general)f(datat)o(yp)q(es)f(that)g(allo)o(w)h(one)g(to)g(transfer)f (heterogeneous)h(and)75 2075 y(noncon)o(tiguous)g(data.)75 2366 y FB(1.2)59 b(Basic)20 b(send)g(op)r(eration)75 2518 y FA(The)15 b(syn)o(tax)g(of)g(the)g(simplest)h(send)g(op)q(eration)f(is)h(giv) o(en)g(b)q(elo)o(w.)p 75 2661 720 2 v 127 2688 a Fy(1)144 2704 y Fx(V)m(ersion)e(of)f(Jan)g(4,)f(1994)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 2 2 bop 75 -100 a FA(2)631 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(SEND\(buf,)g(count,)g (datat)o(yp)q(e,)g(dest,)g(tag,)f(comm)m(\))117 130 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 213 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)g(in)g(send)h(bu\013er)h(\(nonnegativ)o(e)e(in)o(te-)905 269 y(ger\))117 352 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 434 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 517 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 599 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))75 727 y Ft(int)23 b(MPI)p 245 727 15 2 v 17 w(Send\(void*)g(buf,)g(int)g (count,)h(MPI)p 979 727 V 16 w(Datatype)f(datatype,)g(int)g(dest,)393 784 y(int)h(tag,)f(MPI)p 683 784 V 17 w(Comm)g(comm\))75 874 y(MPI)p 150 874 V 17 w(SEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)h (COMM,)f(IERROR\))170 930 y()g(BUF\(*\))170 987 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(IERROR)75 1160 y Fs(1.2.1)49 b(Message)16 b(data)75 1253 y FA(The)e(send)g(bu\013er)g(sp)q(eci\014ed)i(b)o (y)e(the)f Fv(MPI)p 803 1253 14 2 v 16 w(SEND)h FA(op)q(eration)g(consists)g (of)f Fv(count)i FA(successiv)o(e)g(en)o(tries)f(of)75 1309 y(the)f(t)o(yp)q(e)h(indicated)h(b)o(y)e Fv(datat)o(yp)q(e)p FA(,)i(starting)e(with)h(the)f(en)o(try)g(at)g(address)h Fv(buf)p FA(.)20 b(Note)13 b(that)g(w)o(e)g(sp)q(ecify)75 1366 y(the)19 b(message)f(length)i(in)g(terms)e(of)g(n)o(um)o(b)q(er)h(of)g Fr(elements)p FA(,)f(not)g(n)o(um)o(b)q(er)i(of)e Fr(bytes)p FA(.)30 b(The)19 b(former)f(is)75 1422 y(mac)o(hine)e(indep)q(enden)o(t)i (and)d(closer)h(to)e(the)h(application)i(lev)o(el.)166 1482 y(The)g(data)f(part)g(of)g(the)g(message)h(consists)f(of)h(a)f(sequence)i(of) e Fv(count)i FA(v)m(alues,)f(eac)o(h)g(of)f(the)h(t)o(yp)q(e)75 1539 y(indicated)j(b)o(y)e Fv(datat)o(yp)q(e)p FA(.)31 b Fv(count)20 b FA(ma)o(y)d(b)q(e)i(zero,)g(in)g(whic)o(h)g(case)g(the)f(data)g(part)f(of)h (the)h(message)e(is)75 1595 y(empt)o(y)l(.)i(The)13 b(basic)h(datat)o(yp)q (es)e(that)h(can)g(b)q(e)h(sp)q(eci\014ed)h(for)d(message)h(data)f(v)m(alues) i(corresp)q(ond)g(to)e(the)75 1652 y(basic)j(datat)o(yp)q(es)f(of)g(the)h (host)f(language.)19 b(The)c(p)q(ossible)h(v)m(alues)g(of)e(this)h(argumen)o (t)f(for)f(F)l(ortran)h(and)75 1708 y(the)h(corresp)q(onding)h(F)l(ortran)e (t)o(yp)q(es)i(are)f(listed)h(b)q(elo)o(w)p 472 1892 1006 2 v 471 1949 2 57 v 497 1932 a(MPI)f(datat)o(yp)q(e)p 1046 1949 V 294 w(F)l(ortran)f(datat)o(yp)q(e)p 1477 1949 V 472 1951 1006 2 v 471 2007 2 57 v 497 1990 a Fv(MPI)p 582 1990 14 2 v 16 w(INTEGER)p 1046 2007 2 57 v 287 w Ft(INTEGER)p 1477 2007 V 471 2064 V 497 2047 a Fv(MPI)p 582 2047 14 2 v 16 w(REAL)p 1046 2064 2 57 v 365 w Ft(REAL)p 1477 2064 V 471 2120 V 497 2103 a Fv(MPI)p 582 2103 14 2 v 16 w(DOUBLE)p 777 2103 V 17 w(PRECISION)p 1046 2120 2 57 v 50 w Ft(DOUBLE)23 b(PRECISION)p 1477 2120 V 471 2176 V 497 2159 a Fv(MPI)p 582 2159 14 2 v 16 w(COMPLEX)p 1046 2176 2 57 v 263 w Ft(COMPLEX)p 1477 2176 V 471 2233 V 497 2216 a Fv(MPI)p 582 2216 14 2 v 16 w(LOGICAL)p 1046 2233 2 57 v 291 w Ft(LOGICAL)p 1477 2233 V 471 2289 V 497 2272 a Fv(MPI)p 582 2272 14 2 v 16 w(CHARA)o(CTER)p 1046 2289 2 57 v 211 w Ft(CHARACTER\(1\))p 1477 2289 V 471 2346 V 497 2329 a Fv(MPI)p 582 2329 14 2 v 16 w(BYTE)p 1046 2346 2 57 v 1477 2346 V 472 2347 1006 2 v 75 2549 a FA(The)13 b(p)q(ossible)i(v)m (alues)g(for)d(this)i(argumen)o(t)f(for)f(C)h(and)h(the)f(corresp)q(onding)h (C)f(t)o(yp)q(es)g(are)g(listed)i(b)q(elo)o(w.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 3 3 bop 75 -100 a Fw(1.2.)34 b(BASIC)16 b(SEND)f(OPERA)l(TION)1097 b FA(3)p 466 5 1019 2 v 465 62 2 57 v 491 45 a(MPI)15 b(datat)o(yp)q(e)p 1004 62 V 259 w(C)g(datat)o(yp)q(e)p 1483 62 V 466 63 1019 2 v 465 120 2 57 v 491 103 a Fv(MPI)p 576 103 14 2 v 16 w(CHAR)p 1004 120 2 57 v 321 w Ft(signed)23 b(char)p 1483 120 V 465 176 V 491 159 a Fv(MPI)p 576 159 14 2 v 16 w(SHORT)p 1004 176 2 57 v 291 w Ft(signed)g(short)p 1483 176 V 465 233 V 491 216 a Fv(MPI)p 576 216 14 2 v 16 w(INT)p 1004 233 2 57 v 365 w Ft(signed)g(int)p 1483 233 V 465 289 V 491 272 a Fv(MPI)p 576 272 14 2 v 16 w(LONG)p 1004 289 2 57 v 321 w Ft(signed)g(long)g(int)p 1483 289 V 465 346 V 491 329 a Fv(MPI)p 576 329 14 2 v 16 w(UNSIGNED)p 815 329 V 17 w(CHAR)p 1004 346 2 57 v 81 w Ft(unsigned)f(char)p 1483 346 V 465 402 V 491 385 a Fv(MPI)p 576 385 14 2 v 16 w(UNSIGNED)p 815 385 V 17 w(SHORT)p 1004 402 2 57 v 51 w Ft(unsigned)g(short)i(int)p 1483 402 V 465 459 V 491 442 a Fv(MPI)p 576 442 14 2 v 16 w(UNSIGNED)p 1004 459 2 57 v 218 w Ft(unsigned)e(int)p 1483 459 V 465 515 V 491 498 a Fv(MPI)p 576 498 14 2 v 16 w(UNSIGNED)p 815 498 V 17 w(LONG)p 1004 515 2 57 v 81 w Ft(unsigned)g(long)i(int)p 1483 515 V 465 571 V 491 555 a Fv(MPI)p 576 555 14 2 v 16 w(FLO)o(A)l(T)p 1004 571 2 57 v 301 w Ft(float)p 1483 571 V 465 628 V 491 611 a Fv(MPI)p 576 611 14 2 v 16 w(DOUBLE)p 1004 628 2 57 v 262 w Ft(double)p 1483 628 V 465 684 V 491 667 a Fv(MPI)p 576 667 14 2 v 16 w(LONG)p 712 667 V 16 w(DOUBLE)p 1004 684 2 57 v 126 w Ft(long)f(double)p 1483 684 V 465 741 V 491 724 a Fv(MPI)p 576 724 14 2 v 16 w(BYTE)p 1004 741 2 57 v 1483 741 V 466 742 1019 2 v 166 912 a FA(The)16 b(datat)o(yp)q(e)f Fv(MPI)p 536 912 14 2 v 15 w(BYTE)i FA(do)q(es)e(not)g(corresp)q(ond)h(to)f(a)g(F)l (ortran)g(or)g(C)g(datat)o(yp)q(e.)20 b(A)c(v)m(alue)g(of)75 968 y(t)o(yp)q(e)d Fv(MPI)p 260 968 V 16 w(BYTE)h FA(consists)f(of)g(a)g(b)o (yte)g(\(8)g(binary)h(digits\).)19 b(A)13 b(b)o(yte)h(is)f(unin)o(terpreted)i (and)e(is)h(di\013eren)o(t)75 1025 y(from)i(a)g(c)o(haracter.)23 b(Di\013eren)o(t)16 b(mac)o(hines)h(ma)o(y)f(ha)o(v)o(e)g(di\013eren)o(t)h (represen)o(tations)f(for)g(c)o(haracters,)f(or)75 1081 y(ma)o(y)i(use)i (more)e(than)h(one)g(b)o(yte)g(to)f(represen)o(t)h(c)o(haracters.)28 b(On)18 b(the)g(other)g(hand,)h(a)e(b)o(yte)h(has)g(the)75 1138 y(same)d(binary)h(v)m(alue)g(on)f(all)h(mac)o(hines.)166 1194 y(MPI)11 b(supp)q(orts)g(the)f(basic)i(datat)o(yp)q(es)e(of)g(F)l (ortran)g(77)g(and)h(ANSI)h(C.)e(Additional)j(MPI)e(datat)o(yp)q(es)75 1251 y(should)32 b(b)q(e)g(pro)o(vided)f(if)h(the)f(host)f(language)h(has)g (additional)i(data)d(t)o(yp)q(es:)51 b(F)l(or)30 b(example,)75 1307 y Fv(MPI)p 160 1307 V 16 w(LONG)p 296 1307 V 17 w(LONG)p 433 1307 V 17 w(INT)p FA(,)13 b(for)h(64)g(bit)h(in)o(tegers)g(in)g(C;)f Fv(MPI)p 1113 1307 V 16 w(DOUBLE)p 1308 1307 V 16 w(COMPLEX)h FA(for)f(double)i(preci-)75 1364 y(sion)h(complex)g(in)g(F)l(ortran;)e Fv(MPI)p 667 1364 V 16 w(REAL2)p FA(,)h Fv(MPI)p 928 1364 V 16 w(REAL4)g FA(and)h Fv(MPI)p 1266 1364 V 15 w(REAL8)g FA(for)e(2,4)g(or)h (8)g(b)o(yte)g(reals,)75 1420 y(in)g(F)l(ortran;)e Fv(MPI)p 391 1420 V 15 w(INTEGER1)i(MPI)p 716 1420 V 16 w(INTEGER2)f FA(and)g Fv(MPI)p 1129 1420 V 16 w(INTEGER4)h FA(for)e(1,)h(2)g(or)f(4)h(b)o (yte)g(in)o(tegers)75 1476 y(in)h(F)l(ortran;)e(etc.)189 1581 y Fr(R)n(ationale.)77 b FA(One)23 b(goal)e(of)g(the)h(design)g(is)g(to)f (allo)o(w)h(for)e(MPI)i(to)f(b)q(e)h(implemen)o(ted)h(as)e(a)189 1637 y(library)l(,)d(with)f(no)g(need)h(for)e(additional)j(prepro)q(cessing)f (or)e(compilation.)27 b(Th)o(us,)17 b(one)g(cannot)189 1693 y(assume)c(that)g(a)g(comm)o(unication)h(call)g(has)g(information)f(on)h(the) f(datat)o(yp)q(e)g(of)g(v)m(ariables)i(in)f(the)189 1750 y(comm)o(unication) 20 b(bu\013er;)i(this)f(information)f(m)o(ust)f(b)q(e)i(supplied)h(b)o(y)e (an)g(explicit)i(argumen)o(t.)189 1806 y(The)e(need)h(for)f(suc)o(h)g(datat)o (yp)q(e)g(information)g(will)i(b)q(ecome)f(clear)f(in)h(section)g(1.6.)34 b(\()p Fr(End)20 b(of)189 1863 y(r)n(ationale.)p FA(\))75 1984 y Fs(1.2.2)49 b(Message)16 b(envelop)q(e)75 2070 y FA(In)g(addition)g(to)e (the)h(data)g(part,)f(messages)g(con)o(tain)h(information)h(that)e(can)h(b)q (e)h(used)f(to)g(distinguish)75 2127 y(messages)h(and)g(selectiv)o(ely)i (receiv)o(e)f(them.)23 b(This)17 b(information)f(is)h(con)o(tained)g(in)g(a)f (\014xed)h(n)o(um)o(b)q(er)f(of)75 2183 y(\014xed-format)f(\014elds,)h(whic)o (h)g(w)o(e)f(collectiv)o(ely)i(call)g(the)e Fz(message)i(en)o(v)o(elop)q(e)p FA(.)i(These)d(\014elds)g(are)75 2275 y Fz(source)75 2368 y(destination)75 2462 y(tag)75 2555 y(con)o(text)166 2647 y FA(The)c(message)g(source)g(is)h (implicitly)i(determined)f(b)o(y)e(the)g(iden)o(tit)o(y)h(of)f(the)g(message) g(sender.)20 b(The)75 2704 y(other)15 b(\014elds)h(are)f(sp)q(eci\014ed)i(b)o (y)f(argumen)o(ts)e(in)i(the)f(send)h(op)q(eration.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 4 4 bop 75 -100 a FA(4)631 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)166 49 y FA(The)h(in)o(teger-v)m(alued)h(message)e(tag)f (is)i(sp)q(eci\014ed)i(b)o(y)d(the)g Fv(tag)h FA(argumen)o(t.)22 b(This)17 b(in)o(teger)g(can)f(b)q(e)75 106 y(used)c(b)o(y)g(the)f(program)g (to)g(distinguish)i(di\013eren)o(t)f(t)o(yp)q(es)g(of)f(messages.)18 b(The)12 b(range)f(of)g(v)m(alid)i(tag)e(v)m(alues)75 162 y(is)18 b(implemen)o(tation)h(dep)q(enden)o(t)g(and)f(can)f(b)q(e)h(queried)h(using)f (the)g Fv(MPI)p 1362 162 14 2 v 16 w(GET)p 1466 162 V 17 w(V)l(ALID)p 1610 162 V 16 w(T)l(A)o(G)p 1712 162 V 17 w(RANGE)75 219 y FA(en)o(vironmen)o(tal)e(inquiry)h(function,)e(as)g(describ)q(ed)i(in)f (Chapter)f Fz(??)p FA(.)166 386 y Fq(Discussion:)34 b Fu(Alternativ)o(e)14 b(design)g(\(on)g(the)g(\\Europ)q(ean)h(wish)e(list"\):)166 478 y(The)22 b(range)g(of)g(v)n(alid)e(tag)h(v)n(alues)h(is)g(0)p Fp(:::U)5 b(B)r Fu(,)22 b(where)h(the)f(v)n(alue)f(of)h Fp(U)5 b(B)24 b Fu(can)e(b)q(e)g(found)g(using)f(the)75 534 y Fo(MPI)p 152 534 13 2 v 14 w(GET)p 247 534 V 15 w(V)m(ALID)p 379 534 V 14 w(T)m(A)o(G)p 473 534 V 14 w(RANGE)13 b Fu(en)o(vironmen)o(tal)g (inquiry)h(function,)f(as)i(describ)q(ed)h(in)e(Chapter)h Fq(??)p Fu(.)k Fq(\(Need)75 591 y(to)14 b(c)o(hange)f(the)g(de\014nition)d(of)k(this) e(inquiry)g(function.\))j Fu(The)e(v)n(alue)f(of)g Fp(U)5 b(B)15 b Fu(shall)c(b)q(e)i(no)g(less)g(than)f(127.)75 647 y(W)m(e)h(exp)q(ect)j(go) q(o)q(d)d(qualit)o(y)g(MPI)h(implemen)o(tations)d(to)j(pro)o(vide)f(a)h(m)o (uc)o(h)f(larger)g(tagt)h(range.)166 821 y FA(The)c(con)o(text)g(of)g(the)h (message)e(sen)o(t)h(is)h(sp)q(eci\014ed)i(b)o(y)d(the)g Fv(comm)5 b FA(argumen)o(t.)17 b(The)11 b(message)f(carries)75 878 y(the)21 b(con)o(text)g(asso)q(ciated)g(with)h(this)f(comm)o(unicator)g(\(see)g (Chapter)g Fz(??)p FA(\).)37 b(Eac)o(h)21 b(comm)o(unication)75 934 y(con)o(text)11 b(pro)o(vides)i(a)e(separate)h(\\comm)o(unication)g(univ) o(erse":)19 b(messages)12 b(are)f(alw)o(a)o(ys)h(receiv)o(ed)h(within)75 991 y(the)k(con)o(text)f(they)h(w)o(ere)g(sen)o(t,)g(and)g(messages)f(sen)o (t)h(in)g(di\013eren)o(t)g(con)o(texts)g(do)g(not)f(in)o(terfere.)25 b(This)75 1047 y(pro)o(vides)16 b(a)f(p)q(o)o(w)o(erful)g(encapsulation)i (mec)o(hanism)e(for)g(libraries)i(and)e(mo)q(dules.)166 1139 y(The)f(message)f(destination)h(is)h(sp)q(eci\014ed)g(b)o(y)f(the)f Fv(dest)j FA(argumen)o(t)c(as)i(a)f(rank)g(within)i(the)f(pro)q(cess)75 1195 y(group)i(asso)q(ciated)h(with)g(that)f(same)h(comm)o(unicator)f (\(remote)g(pro)q(cess)h(group,)f(for)g(in)o(tercomm)o(uni-)75 1252 y(cators\).)32 b(The)19 b(range)h(of)f(v)m(alid)i(v)m(alues)g(is)f Fv(0,)g(...)32 b(,)20 b(n-1)p FA(,)h(where)e Fv(n)h FA(is)h(the)e(n)o(um)o(b) q(er)h(of)f(pro)q(cesses)h(in)75 1308 y(this)e(group.)25 b(Th)o(us,)17 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unications)g(do)f(not)g(use)g(absolute) h(addresses,)f(but)g(only)75 1365 y(relativ)o(e)f(ranks)f(within)h(a)f (group.)k(This)d(pro)o(vides)g(imp)q(ortan)o(t)f(mo)q(dularit)o(y)l(.)166 1456 y(The)j(message)f(en)o(v)o(elop)q(e)i(w)o(ould)f(normally)g(b)q(e)h (enco)q(ded)g(b)o(y)e(a)g(\014xed-length)j(message)d(header.)75 1513 y(Ho)o(w)o(ev)o(er,)c(the)i(actual)f(mec)o(hanism)h(used)g(to)f(asso)q (ciate)g(an)g(en)o(v)o(elop)q(e)i(with)e(a)g(message)g(is)h(implemen-)75 1569 y(tation)d(dep)q(enden)o(t:)20 b(some)12 b(of)g(the)g(information)g (\(e.g.,)g(source)g(or)g(destination\))g(ma)o(y)g(b)q(e)h(implicit,)i(and)75 1626 y(need)f(not)g(b)q(e)g(explicitly)i(carried)e(b)o(y)g(messages;)f(pro)q (cesses)h(ma)o(y)f(b)q(e)h(iden)o(ti\014ed)i(b)o(y)d(relativ)o(e)h(ranks,)f (or)75 1682 y(absolute)j(ids;)f(etc.)189 1964 y Fr(A)n(dvic)n(e)j(to)i (users.)60 b FA(Users)18 b(that)g(are)g(comfortable)g(with)h(the)g(notion)g (of)f(a)g(\015at)g(name)h(space)189 2020 y(for)14 b(pro)q(cesses,)h(and)h(a)f (unique)h(comm)o(unication)g(con)o(text,)f(as)f(o\013ered)h(b)o(y)g(most)g (existing)h(com-)189 2077 y(m)o(unication)f(libraries,)h(need)f(only)h(use)e (the)h(prede\014ned)h(v)m(ariable)g Fo(MPI)p 1437 2077 V 15 w(COMM)p 1582 2077 V 14 w(W)o(ORLD)f FA(as)f(the)189 2133 y Fv(comm)d FA(argumen)o(t.)25 b(This)17 b(will)i(allo)o(w)f(to)e(comm)o (unicate)i(with)f(all)h(the)f(pro)q(cesses)h(a)o(v)m(ailable)h(at)189 2190 y(initialization)f(time.)i(\()p Fr(End)c(of)g(advic)n(e)g(to)h(users.)p FA(\))75 2534 y FB(1.3)59 b(Basic)20 b(receive)f(op)r(eration)75 2704 y FA(The)c(syn)o(tax)g(of)g(the)g(simplest)h(receiv)o(e)g(op)q(eration)g (is)f(giv)o(en)h(b)q(elo)o(w.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 5 5 bop 75 -100 a Fw(1.3.)34 b(BASIC)16 b(RECEIVE)g(OPERA)l(TION)1012 b FA(5)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(RECV)16 b(\(buf,)f(count,)h(datat) o(yp)q(e,)h(source,)e(tag,)g(comm)m(,)d(status\))117 126 y Fu(OUT)108 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(receiv)o(e)h (bu\013er)g(\(c)o(hoice\))117 201 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g(\(in) o(teger\))117 275 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e(\(handle\))117 349 y(IN)155 b Fv(source)468 b Fu(rank)14 b(of)f(source)j(\(in)o(teger\))117 423 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 497 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 571 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 696 y Ft(int)23 b(MPI)p 245 696 15 2 v 17 w(Recv\(void*)g(buf,)g(int)g (count,)h(MPI)p 979 696 V 16 w(Datatype)f(datatype,)g(int)g(source,)393 752 y(int)h(tag,)f(MPI)p 683 752 V 17 w(Comm)g(comm,)g(MPI)p 1034 752 V 17 w(Status)g(*status\))75 838 y(MPI)p 150 838 V 17 w(RECV\(BUF,)f(COUNT,)h(DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(STATUS,)g (IERROR\))170 895 y()g(BUF\(*\))170 951 y(INTEGER)g(COUNT,)g(DATATYPE,) g(SOURCE,)g(TAG,)g(COMM,)g(STATUS\(MPI)p 1463 951 V 16 w(STATUS)p 1623 951 V 17 w(SIZE\),)170 1008 y(IERROR)166 1094 y FA(The)15 b(receiv)o(e)h(bu\013er)f(consists)h(of)e(the)h(storage)f(con)o(taining)i Fv(count)g FA(consecutiv)o(e)g(elemen)o(ts)g(of)f(the)75 1151 y(t)o(yp)q(e)e(sp)q(eci\014ed)i(b)o(y)e Fv(datat)o(yp)q(e)p FA(,)h(starting)e(at)h(address)f Fv(buf)p FA(.)20 b(The)13 b(length)h(of)e(the)h(receiv)o(ed)h(message)f(m)o(ust)75 1207 y(b)q(e)i(less)g(than)f(or)g(equal)h(to)e(the)i(length)g(of)e(the)i(receiv)o (e)g(bu\013er.)20 b(I.e.,)13 b(an)i(error)e(o)q(ccurs)i(if)g(all)g(incoming) 75 1264 y(data)j(do)q(es)h(not)g(\014t,)g(without)g(truncation,)g(in)o(to)g (the)g(receiv)o(e)h(bu\013er.)30 b(The)19 b Fv(MPI)p 1531 1264 14 2 v 16 w(PROBE)h FA(function)75 1320 y(describ)q(ed)d(in)f(section)g(1.9)e (can)i(b)q(e)f(used)h(to)f(receiv)o(e)h(messages)f(of)f(unkno)o(wn)i(length.) 166 1453 y Fq(Discussion:)48 b Fu(It)11 b(w)o(as)g(suggested)h(that)f(MPI)g (should)g(sp)q(ecify)g(the)h(b)q(eha)o(vior)e(of)h(receiv)o(e)h(when)f (incoming)66 b Fn(\()75 1502 y Fu(message)16 b(is)g(to)q(o)g(long.)24 b(\(An)17 b(error)g(indication)e(is)h(returned)i(b)o(y)e(the)h(receiv)o(e)g (call,)f(and)g(the)h(error)g(handler)f(is)75 1552 y(called;)g(one)f(migh)o(t) f(still)h(w)o(an)o(t)g(the)h(con)o(ten)o(t)g(of)f(the)h(receiv)o(e)h (bu\013er)g(and)e(of)g(the)h(status)h(v)n(ariable)d(to)i(b)q(e)g(w)o(ell-)75 1602 y(de\014ned.\))166 1652 y(The)e(w)o(eak)o(est)h(requiremen)o(ts)f(are)g (that:)145 1735 y Fm(\017)23 b Fu(No)13 b(area)h(outside)h(the)f(receiv)o(e)h (bu\013er)g(is)f(o)o(v)o(erwritten.)145 1808 y Fm(\017)23 b Fo(MPI)p 266 1808 13 2 v 14 w(GET)p 361 1808 V 15 w(TYPE)13 b Fu(and)g Fo(MPI)p 652 1808 V 15 w(GET)p 748 1808 V 14 w(SOURCE)f Fu(return)k(the)e(righ)o(t)f(answ)o(er.)145 1880 y Fm(\017)23 b Fu(The)12 b(v)n(alues)h(returned)h(b)o(y)e Fo(MPI)p 695 1880 V 14 w(GET)p 790 1880 V 15 w(COUNT)g Fu(and)g Fo(MPI)p 1115 1880 V 14 w(GET)p 1210 1880 V 15 w(ELEMENT)h Fu(indicate)f(ho)o(w)g(man)o(y)f (items)189 1930 y(\(resp.)19 b(elemen)o(ts\))14 b(w)o(ere)h(actually)e (receiv)o(ed)i(in)e(the)i(receiv)o(e)g(bu\013er.)166 2013 y(A)j(stronger)i (requiremen)o(t)e(is)h(that)f(the)h(receiv)o(e)h(op)q(eration)e(\014ts)h(as)g (m)o(uc)o(h)e(data)h(as)h(can)g(b)q(e)g(\014t)f(in)g(the)75 2063 y(receiv)o(e)d(bu\013er.)k(\(One)c(can)f(statisfy)f(the)i(previous)f (requiremen)o(ts)g(b)o(y)f(receiving)h(no)g(data)f(and)h(returning)g(zero)75 2113 y(coun)o(ts,)g(in)f(case)i(of)f(o)o(v)o(er\015o)o(w.\))166 2169 y(Do)20 b(w)o(e)h(w)o(an)o(t)f(to)h(sp)q(ecify)g(suc)o(h)g(b)q(eha)o (vior?)38 b(\(So)21 b(far,)g(w)o(e)g(ha)o(v)o(e)g(a)o(v)o(oided)e(de\014ning) i(the)g(b)q(eha)o(vior)g(of)75 2226 y(erroneous)15 b(programs.\))166 2365 y FA(The)k(selection)h(of)f(a)f(message)h(b)o(y)f(a)h(receiv)o(e)h(op)q (eration)f(is)g(done)g(uniquely)i(according)f(to)e(the)75 2421 y(v)m(alue)i(of)f(the)g(message)g(en)o(v)o(elop)q(e.)32 b(The)19 b(receiv)o(e)h(op)q(eration)g(sp)q(eci\014es)h(an)d Fz(en)o(v)o(elop)q(e)k (pattern)p FA(;)f(a)75 2478 y(message)13 b(can)g(b)q(e)g(receiv)o(ed)h(b)o(y) f(that)g(receiv)o(e)h(op)q(eration)f(only)g(if)h(its)f(en)o(v)o(elop)q(e)h (matc)o(hes)f(that)f(pattern.)75 2534 y(A)19 b(pattern)g(sp)q(eci\014es)i(v)m (alues)g(for)e(the)g Fv(source)p FA(,)i Fv(tag)e FA(and)h Fv(context)g FA(\014elds)h(of)e(the)g(message)g(en)o(v)o(elop)q(e.)75 2591 y(The)d(receiv)o(er)h(ma)o(y)e(sp)q(ecify)j(a)e(wildcard)h Fo(MPI)p 889 2591 V 14 w(ANY)p 988 2591 V 15 w(SOURCE)d FA(v)m(alue)k(for)d Fv(source)p FA(,)h(and/or)g(a)g(wildcard)75 2647 y Fo(MPI)p 152 2647 V 14 w(ANY)p 251 2647 V 15 w(T)m(A)o(G)c FA(v)m(alue)j(for)d Fv(tag)p FA(,)i(indicating)h(that)d(an)o(y)h(source)h(and/or)e(tag)h(are)g (acceptable.)20 b(It)13 b(cannot)75 2704 y(sp)q(ecify)h(a)f(wildcard)h(v)m (alue)g(for)f Fv(context)p FA(.)20 b(Th)o(us,)13 b(a)g(message)f(can)i(b)q(e) f(receiv)o(ed)h(b)o(y)f(a)g(receiv)o(e)h(op)q(eration)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 6 6 bop 75 -100 a FA(6)631 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y FA(only)23 b(if)g(it)g(is)g(addressed)h(to)e(the)g (receiving)j(pro)q(cess,)f(has)f(a)f(matc)o(hing)h(con)o(text,)g(has)g(matc)o (hing)75 106 y(source)14 b(unless)g(source=)p Fo(MPI)p 581 106 13 2 v 15 w(ANY)p 681 106 V 15 w(SOURCE)d FA(in)k(the)e(pattern,)g(and)h (has)g(a)f(matc)o(hing)g(tag)g(unless)i(tag=)75 162 y Fo(MPI)p 152 162 V 14 w(ANY)p 251 162 V 15 w(T)m(A)o(G)f FA(in)i(the)f(pattern.)166 235 y(The)d(message)g(tag)f(is)i(sp)q(eci\014ed)h(b)o(y)e(the)g Fv(tag)h FA(argumen)o(t)e(of)h(the)g(receiv)o(e)h(op)q(eration.)19 b(The)12 b(message)75 292 y Fv(context)20 b FA(is)f(the)f(con)o(text)g(attac) o(hed)g(with)h(the)f(comm)o(unicator)g(sp)q(eci\014ed)j(b)o(y)d(the)h (argumen)o(t)e Fv(comm)m FA(.)75 348 y(The)g(message)g(source,)h(if)f (di\013eren)o(t)h(from)32 b Fo(MPI)p 929 348 V 14 w(ANY)p 1028 348 V 15 w(SOURCE)p FA(,)15 b(is)j(sp)q(eci\014ed)h(as)e(a)g(rank)f(within)j (the)75 405 y(pro)q(cess)g(group)f(asso)q(ciated)h(with)g(that)e(same)h(comm) o(unicator)h(\(remote)e(pro)q(cess)i(group,)g(for)f(in)o(ter-)75 461 y(comm)o(unicators\).)27 b(Th)o(us,)18 b(the)g(range)g(of)f(v)m(alid)j(v) m(alues)f(for)e(the)h Fv(source)g FA(argumen)o(t)g(is)g Fl(f)g Fv(0,)f(...)28 b(,)18 b(n-1)75 518 y Fl(g[f)d Fv(MPI)p 251 518 14 2 v 16 w(ANY)p 359 518 V 17 w(SOURCE)i Fl(g)p FA(,)d(where)h Fv(n)h FA(is)g(the)f(n)o(um)o(b)q(er)h(of)e(pro)q(cesses)i(in)g(this)g (group.)166 591 y(Note)i(the)i(asymmetry)d(b)q(et)o(w)o(een)j(send)f(and)g (receiv)o(e)h(op)q(erations:)28 b(A)19 b(receiv)o(e)h(op)q(eration)f(ma)o(y) 75 647 y(accept)13 b(messages)f(from)g(an)h(arbitrary)f(sender;)i(on)f(the)g (other)f(hand,)i(a)e(send)h(op)q(eration)g(m)o(ust)g(sp)q(ecify)75 704 y(a)i(unique)j(receiv)o(er.)k(This)16 b(matc)o(hes)f(a)h(\\push")f(comm)o (unication)i(mec)o(hanism,)f(where)g(data)f(transfer)75 760 y(is)i(e\013ected)h(b)o(y)f(the)g(sender)g(\(rather)f(than)h(a)g(\\pull")h (mec)o(hanism,)g(where)f(data)f(transfer)g(is)i(e\013ected)75 817 y(b)o(y)d(the)g(receiv)o(er\).)166 890 y(Source)f(=)h(destination)f(is)h (allo)o(w)o(ed:)k(a)14 b(pro)q(cess)g(can)g(send)h(a)f(message)f(to)g (itself.)21 b(\(Ho)o(w)o(ev)o(er,)12 b(it)i(is)75 946 y(unsafe)h(to)g(do)g (so)g(with)h(the)f(basic)h(send)g(and)f(receiv)o(e)h(op)q(erations)g(describ) q(ed)h(ab)q(o)o(v)o(e,)e(since)h(this)g(ma)o(y)75 1003 y(lead)g(to)f(deadlo)q (c)o(k;)g(see)h(Section)g(1.4\))75 1220 y Fs(1.3.1)49 b(Return)16 b(status)75 1339 y FA(The)e(source)g(or)f(the)h(tag)f(of)g(a)h(receiv)o(ed)h (message)e(ma)o(y)g(not)h(b)q(e)g(kno)o(wn)g(if)g(wildcard)h(v)m(alues)g(w)o (ere)e(used)75 1395 y(in)h(the)f(receiv)o(e)h(op)q(eration.)19 b(Also,)13 b(the)g(actual)g(length)h(of)f(the)g(message)f(receiv)o(ed)i(ma)o (y)e(not)h(b)q(e)h(kno)o(wn.)75 1452 y(Th)o(us,)j(this)h(information)f(needs) h(to)e(b)q(e)i(returned)g(b)o(y)f(the)g(receiv)o(e)h(op)q(eration.)26 b(The)17 b(information)g(is)75 1508 y(returned)12 b(b)o(y)g(the)f Fv(status)j FA(argumen)o(t)d(of)g(the)h Fv(MPI)p 923 1508 V 16 w(RECV)g FA(function.)20 b(This)12 b(is)g(an)g(argumen)o(t)f(of)g(a)g(sp)q (ecial)75 1564 y(MPI-de\014ned)19 b(t)o(yp)q(e.)27 b(Status)17 b(v)m(ariables)i(need)f(to)f(b)q(e)h(explicitly)i(allo)q(cated)f(b)o(y)e(the) h(user)g({)f(they)g(are)75 1621 y(not)e(system)g(ob)s(jects.)166 1694 y(The)f(status)f(v)m(ariable)i(can)f(b)q(e)g(\\deco)q(ded")h(to)e (retriev)o(e)h(the)g Fv(count)p FA(,)g Fv(source)h FA(and)f Fv(tag)f FA(\014elds,)i(using)75 1751 y(the)g(query)g(functions)h(listed)g(b) q(elo)o(w.)k(The)15 b(information)g(returned)h(b)o(y)f(these)g(query)g (functions)g(is)h(the)75 1807 y(information)k(stored)e(in)j(the)e Fv(status)j FA(v)m(ariable)e(b)o(y)g(the)f(last)g(receiv)o(e)i(\(or)d(prob)q (e)i({)f(see)h(Section)g(1.9\))75 1864 y(call)c(that)f(w)o(as)f(passed)h Fv(status)i FA(as)e(argumen)o(t.)k(It)c(is)h(erroneous)f(to)f(call)j(these)e (query)g(functions)h(if)g(the)75 1920 y Fv(status)h FA(v)m(ariable)g(w)o(as)d (nev)o(er)h(set)g(b)o(y)h(a)e(receiv)o(e)j(\(or)d(prob)q(e\).)75 2040 y Fv(MPI)p 160 2040 V 16 w(GET)p 264 2040 V 17 w(SOURCE\(status,)k (source\))117 2134 y Fu(IN)155 b Fv(status)476 b Fu(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f(\(Status\))117 2243 y(OUT)108 b Fv(source)468 b Fu(source)15 b(rank)f(\(in)o(teger\))75 2384 y Ft(int)23 b(MPI)p 245 2384 15 2 v 17 w(Get)p 334 2384 V 17 w(source\(MPI)p 591 2384 V 16 w(Status)g(status,)g(int)g(*source\))75 2487 y(MPI)p 150 2487 V 17 w(GET)p 239 2487 V 17 w(SOURCE\(STATUS,)e(SOURCE,) i(IERROR\))170 2544 y(INTEGER)g(STATUS\(MPI)p 604 2544 V 16 w(STATUS)p 764 2544 V 17 w(SIZE\),)g(SOURCE,)f(IERROR)166 2647 y FA(Returns)13 b(the)g(rank)g(of)f(the)h(message)f(source)h(in)h(the)f (group)f(asso)q(ciated)h(with)g(the)g(comm)o(unicator)75 2704 y(that)h(w)o(as)h(used)h(to)e(receiv)o(e)i(\(remote)f(group,)f(for)h(in)o (tercomm)o(unicators\).)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 7 7 bop 75 -100 a Fw(1.3.)34 b(BASIC)16 b(RECEIVE)g(OPERA)l(TION)1012 b FA(7)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(GET)p 264 49 V 17 w(T)l(A)o(G\(status,)17 b(tag\))117 130 y Fu(IN)155 b Fv(status)476 b Fu(return)15 b(status)g(of)e(receiv)o(e)i(or)f(prob)q(e)h(op)q(eration)f (\(Status\))117 213 y(OUT)108 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))75 341 y Ft(int)23 b(MPI)p 245 341 15 2 v 17 w(Get)p 334 341 V 17 w(tag\(MPI)p 519 341 V 16 w(Status)g(status,)g (int)h(*tag\))75 431 y(MPI)p 150 431 V 17 w(GET)p 239 431 V 17 w(TAG\(STATUS,)e(TAG,)h(IERROR\))170 487 y(INTEGER)g(STATUS\(MPI)p 604 487 V 16 w(STATUS)p 764 487 V 17 w(SIZE\),)g(TAG,)g(IERROR)166 578 y FA(Returns)16 b(the)f(tag)f(of)h(receiv)o(ed)h(message.)75 685 y Fv(MPI)p 160 685 14 2 v 16 w(GET)p 264 685 V 17 w(COUNT\(status,)h (datat)o(yp)q(e,)f(count\))117 766 y Fu(IN)155 b Fv(status)476 b Fu(return)15 b(status)g(of)e(receiv)o(e)i(op)q(eration)f(\(Status\))117 848 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e(\(handle\))117 931 y(OUT)108 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h (elemen)o(ts)f(\(in)o(teger\))75 1059 y Ft(int)23 b(MPI)p 245 1059 15 2 v 17 w(Get)p 334 1059 V 17 w(count\(MPI)p 567 1059 V 16 w(Status)g(status,)g(MPI)p 1013 1059 V 17 w(Datatype)f(datatype,)h(int)h (*count\))75 1149 y(MPI)p 150 1149 V 17 w(GET)p 239 1149 V 17 w(COUNT\(STATUS,)e(DATATYPE,)g(COUNT,)h(IERROR\))170 1206 y(INTEGER)g(STATUS\(MPI)p 604 1206 V 16 w(STATUS)p 764 1206 V 17 w(SIZE\),)g(DATATYPE,)f(COUNT,)h(IERROR)166 1296 y FA(Returns)15 b(the)f(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)h(receiv)o(ed.)21 b(\(Here,)14 b(again,)h(w)o(e)f(coun)o(t)g Fr(elements)p FA(,)f(not)h Fr(bytes)p FA(.\))75 1352 y(The)f Fv(datat)o(yp)q(e)h FA(argumen)o(t)e (should)i(matc)o(h)e(the)h(argumen)o(t)f(pro)o(vided)i(b)o(y)f(the)g(receiv)o (e)g(call)h(that)e(set)h(the)75 1409 y Fv(status)19 b FA(v)m(ariable.)26 b(\(W)l(e)17 b(shall)h(later)f(see,)g(in)h(Section)g(1.13.5,)d(that)h Fv(MPI)p 1347 1409 14 2 v 16 w(GET)p 1451 1409 V 16 w(COUNT)i FA(ma)o(y)e(return,)75 1465 y(in)g(certain)g(situations,)f(the)g(v)m(alue)i Fo(MPI)p 773 1465 13 2 v 14 w(UNDEFINED)p FA(.\))166 1525 y(Note)e(that)g(it) h(is)g(not)f(mandatory)g(to)g(query)g(the)h(return)g(status)e(after)h(a)g (receiv)o(e.)22 b(The)16 b(receiv)o(er)75 1582 y(will)h(use)e(these)h(calls,) g(or)e(some)h(of)g(them,)g(only)h(when)f(the)h(information)f(they)g(return)g (is)h(needed.)189 1706 y Fr(R)n(ationale.)43 b FA(The)16 b(use)g(of)f(a)h (separate)f(status)g(argumen)o(t)g(prev)o(en)o(ts)h(errors)e(that)i(are)f (often)g(at-)189 1763 y(tac)o(hed)f(with)h Fu(INOUT)g FA(argumen)o(t)f (\(e.g.,)f(passing)i(the)28 b Fo(MPI)p 1233 1763 V 14 w(ANY)p 1332 1763 V 15 w(T)m(A)o(G)13 b FA(constan)o(t)h(as)g(the)h(actual)189 1819 y(argumen)o(t)k(for)g Fv(tag)p FA(\).)32 b(The)20 b(use)g(of)g(an)f (explicit)j(user)e(ob)s(ject)f(b)q(ecomes)h(imp)q(ortan)o(t)g(for)f(non-)189 1876 y(blo)q(c)o(king)d(comm)o(unication)g(as)f(it)g(allo)o(ws)g(the)h (receiv)o(e)g(op)q(eration)f(to)g(b)q(e)g(stateless)g(and,)g(hence,)189 1932 y(reen)o(tran)o(t.)j(This)d(prev)o(en)o(ts)f(confusions)h(in)g(the)f (case)g(where)h(m)o(ultiple)h(receiv)o(es)f(can)f(b)q(e)h(p)q(osted)189 1989 y(b)o(y)g(a)g(pro)q(cess.)189 2071 y(The)j Fv(datat)o(yp)q(e)i FA(argumen)o(t)d(is)i(passed)g(to)e(the)i Fv(MPI)p 1096 2071 14 2 v 16 w(GET)p 1200 2071 V 16 w(COUNT)g FA(so)f(as)g(to)g(impro)o(v)o(e)g (p)q(erfor-)189 2128 y(mance)k(\(a)f(message)h(migh)o(t)g(b)q(e)g(receiv)o (ed)i(without)e(coun)o(ting)g(the)g(n)o(um)o(b)q(er)g(of)g(elemen)o(ts)h(it) 189 2184 y(con)o(tains\),)14 b(and)i(to)e(allo)o(w)i(its)f(use)h(after)e(a)h (call)h(to)f Fv(MPI)p 1170 2184 V 16 w(PROBE)p FA(.)h(\()p Fr(End)f(of)i(r)n(ationale.)p FA(\))166 2391 y Fq(Implemen)o(tati)o(on)d (note:)44 b Fu(One)16 b(exp)q(ects)i(deco)q(de)f(functions)f(to)g(b)q(e)g (in-lined)f(in)g(man)o(y)f(implemen)o(ta-)75 2448 y(tions.)166 2591 y FA(All)19 b(send)g(and)g(receiv)o(e)g(op)q(erations)f(use)h(the)f Fv(buf)p FA(,)h Fv(count)p FA(,)h Fv(datat)o(yp)q(e)p FA(,)g Fv(source)p FA(,)f Fv(dest)p FA(,)h Fv(tag)p FA(,)f Fv(comm)75 2647 y FA(and)e Fv(status)i FA(argumen)o(ts)d(in)i(the)f(same)g(w)o(a)o(y)f (as)g(the)h(basic)h Fv(MPI)p 1201 2647 V 16 w(SEND)f FA(and)g Fv(MPI)p 1523 2647 V 16 w(RECV)h FA(op)q(erations)75 2704 y(describ)q(ed)f (in)f(this)g(section.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 8 8 bop 75 -100 a FA(8)631 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h (COMMUNICA)l(TION)75 49 y Fv(Alternative)g(Prop)q(osal)75 143 y FA(The)26 b(source)13 b(or)g(the)g(tag)f(of)h(a)f(receiv)o(ed)j(message)d (ma)o(y)h(not)f(b)q(e)i(kno)o(wn)f(if)g(wildcard)h(v)m(alues)h(w)o(ere)d (used)-1902 b Fk(\))75 200 y FA(in)14 b(the)f(receiv)o(e)h(op)q(eration.)19 b(Also,)13 b(the)g(actual)g(length)h(of)f(the)g(message)f(receiv)o(ed)i(ma)o (y)e(not)h(b)q(e)h(kno)o(wn.)75 256 y(Th)o(us,)j(this)h(information)f(needs)h (to)e(b)q(e)i(returned)g(b)o(y)f(the)g(receiv)o(e)h(op)q(eration.)26 b(The)17 b(information)g(is)75 313 y(returned)12 b(b)o(y)g(the)f Fv(status)j FA(argumen)o(t)d(of)g(the)h Fv(MPI)p 923 313 14 2 v 16 w(RECV)g FA(function.)20 b(This)12 b(is)g(an)g(argumen)o(t)f(of)g(a)g (sp)q(ecial)75 369 y(MPI-de\014ned)19 b(t)o(yp)q(e.)27 b(Status)17 b(v)m(ariables)i(need)f(to)f(b)q(e)h(explicitly)i(allo)q(cated)f(b)o(y)e(the) h(user)g({)f(they)g(are)75 425 y(not)e(system)g(ob)s(jects.)166 486 y(The)f(status)f(v)m(ariable)i(can)f(b)q(e)g(\\deco)q(ded")h(to)e (retriev)o(e)h(the)g Fv(count)p FA(,)g Fv(source)h FA(and)f Fv(tag)f FA(\014elds,)i(using)75 543 y(the)g(query)g(functions)h(listed)g(b)q (elo)o(w.)k(The)15 b(information)g(returned)h(b)o(y)f(these)g(query)g (functions)g(is)h(the)75 599 y(information)k(stored)e(in)j(the)e Fv(status)j FA(v)m(ariable)e(b)o(y)g(the)f(last)g(receiv)o(e)i(\(or)d(prob)q (e)i({)f(see)h(Section)g(1.9\))75 655 y(call)c(that)f(w)o(as)f(passed)h Fv(status)i FA(as)e(argumen)o(t.)k(It)c(is)h(erroneous)f(to)f(call)j(these)e (query)g(functions)h(if)g(the)75 712 y Fv(status)h FA(v)m(ariable)g(w)o(as)d (nev)o(er)h(set)g(b)o(y)h(a)e(receiv)o(e)j(\(or)d(prob)q(e\).)166 773 y(The)i Fv(status)h FA(argumen)o(t)e(in)h(the)f Fv(MPI)p 808 773 V 16 w(RECV)h FA(function)h(is)f(a)f(structure)g(\(of)f(prede\014ned) j(datat)o(yp)q(e)75 829 y Fo(MPI)p 152 829 13 2 v 14 w(Status)p FA(\))d(in)h(C,)f(an)g(arra)o(y)f(of)g(in)o(tegers)i(in)g(F)l(ortran.)j(In)d (C,)e(the)h Fv(source)h FA(and)f Fv(tag)g FA(\014eld)i(are)d(accessed)75 886 y(as)h(comp)q(onen)o(ts)h(of)f(this)h(structure:)k Fv(status.source)e FA(and)e Fv(status.tag)p FA(,)h(resp)q(ectiv)o(ely)l(.)21 b(In)16 b(F)l(ortran,)d(they)75 942 y(are)j(accessed)h(as)g(en)o(tries)g(in)g(an)f (arra)o(y:)22 b Fv(ST)l(A)l(TUS\(MPI)p 1064 942 14 2 v 17 w(SOURCE\))17 b FA(AND)g Fv(ST)l(A)l(TUS\(MPI)p 1675 942 V 17 w(T)l(A)o(G\))p FA(,)f(re-)75 998 y(sp)q(ectiv)o(ely;)f Fo(MPI)p 373 998 13 2 v 15 w(SOURCE)c FA(and)j Fo(MPI)p 723 998 V 14 w(ST)m(A)m(TUS)e FA(are)h(prede\014ned)i(constan)o(ts)e(set)g(to)g(the)g(correct)g(indices)75 1055 y(withing)19 b Fv(ST)l(A)l(TUS)p FA(.)g(The)g Fv(status)h FA(v)m(ariable)g(ma)o(y)d(con)o(tain)i(additional)h(\014elds)f(with)g (additional)h(infor-)75 1111 y(mation)15 b(that)g(is)g(not)g(directly)i (accessible)g(to)d(the)h(user.)166 1172 y(Information)i(on)g(the)f(length)i (of)e(the)h(incoming)i(message)d(is)h(not)g(directly)h(a)o(v)m(ailable)h(as)d (a)h(\014led)75 1228 y(of)d Fv(status)p FA(.)22 b(Rather,)15 b(an)g(inquiry)h(function)g(need)f(to)g(b)q(e)g(applied)i(to)d Fv(status)p FA(,)j(in)f(order)e(to)g(retriev)o(e)i(this)75 1285 y(information.)75 1393 y Fv(MPI)p 160 1393 14 2 v 16 w(GET)p 264 1393 V 17 w(COUNT\(status,)h(datat)o(yp)q(e,)f(count\))117 1474 y Fu(IN)155 b Fv(status)476 b Fu(return)15 b(status)g(of)e(receiv)o(e)i (op)q(eration)f(\(Status\))117 1558 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e (\(handle\))117 1641 y(OUT)108 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h(elemen)o(ts)f(\(in)o(teger\))75 1770 y Ft(int)23 b(MPI)p 245 1770 15 2 v 17 w(Get)p 334 1770 V 17 w(count\(MPI)p 567 1770 V 16 w(Status)g(status,)g(MPI)p 1013 1770 V 17 w(Datatype)f(datatype,)h(int)h(*count\))75 1861 y(MPI)p 150 1861 V 17 w(GET)p 239 1861 V 17 w(COUNT\(STATUS,)e(DATATYPE,)g(COUNT,)h (IERROR\))170 1917 y(INTEGER)g(STATUS\(MPI)p 604 1917 V 16 w(STATUS)p 764 1917 V 17 w(SIZE\),)g(DATATYPE,)f(COUNT,)h(IERROR)166 2008 y FA(Returns)15 b(the)f(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)h(receiv)o (ed.)21 b(\(Here,)14 b(again,)h(w)o(e)f(coun)o(t)g Fr(elements)p FA(,)f(not)h Fr(bytes)p FA(.\))75 2064 y(The)f Fv(datat)o(yp)q(e)h FA(argumen)o(t)e(should)i(matc)o(h)e(the)h(argumen)o(t)f(pro)o(vided)i(b)o(y) f(the)g(receiv)o(e)g(call)h(that)e(set)h(the)75 2121 y Fv(status)19 b FA(v)m(ariable.)26 b(\(W)l(e)17 b(shall)h(later)f(see,)g(in)h(Section)g (1.13.5,)d(that)h Fv(MPI)p 1347 2121 14 2 v 16 w(GET)p 1451 2121 V 16 w(COUNT)i FA(ma)o(y)e(return,)75 2177 y(in)g(certain)g(situations,) f(the)g(v)m(alue)i Fo(MPI)p 773 2177 13 2 v 14 w(UNDEFINED)p FA(.\))166 2238 y(Note)e(that)g(it)h(is)g(not)f(mandatory)g(to)g(query)g(the) h(return)g(status)e(after)h(a)g(receiv)o(e.)22 b(The)16 b(receiv)o(er)75 2294 y(will)h(use)e(these)h(calls,)g(or)e(some)h(of)g(them,)g(only)h(when)f (the)h(information)f(they)g(return)g(is)h(needed.)189 2421 y Fr(R)n(ationale.)43 b FA(The)16 b(use)g(of)f(a)h(separate)f(status)g (argumen)o(t)g(prev)o(en)o(ts)h(errors)e(that)i(are)f(often)g(at-)189 2478 y(tac)o(hed)f(with)h Fu(INOUT)g FA(argumen)o(t)f(\(e.g.,)f(passing)i (the)28 b Fo(MPI)p 1233 2478 V 14 w(ANY)p 1332 2478 V 15 w(T)m(A)o(G)13 b FA(constan)o(t)h(as)g(the)h(actual)189 2534 y(argumen)o(t)k(for)g Fv(tag)p FA(\).)32 b(The)20 b(use)g(of)g(an)f(explicit)j(user)e(ob)s(ject)f (b)q(ecomes)h(imp)q(ortan)o(t)g(for)f(non-)189 2591 y(blo)q(c)o(king)d(comm)o (unication)g(as)f(it)g(allo)o(ws)g(the)h(receiv)o(e)g(op)q(eration)f(to)g(b)q (e)g(stateless)g(and,)g(hence,)189 2647 y(reen)o(tran)o(t.)j(This)d(prev)o (en)o(ts)f(confusions)h(in)g(the)f(case)g(where)h(m)o(ultiple)h(receiv)o(es)f (can)f(b)q(e)h(p)q(osted)189 2704 y(b)o(y)g(a)g(pro)q(cess.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 9 9 bop 75 -100 a Fw(1.4.)34 b(SEMANTICS)16 b(OF)f(POINT-TO-POINT)i(COMMUNICA)l (TION)472 b FA(9)189 49 y(The)14 b Fv(count)i FA(v)m(alue)g(is)f(retriev)o (ed)g(b)o(y)g(a)f(function)h(call)h(with)f(a)f Fv(datat)o(yp)q(e)j FA(argumen)o(t,)c(rather)h(than)189 106 y(b)o(y)e(a)h(direct)h(access)f(to)f (a)h Fv(status)i FA(\014eld,)f(so)e(as)h(to)f(a)o(v)o(oid)h(the)g(need)h(for) e(computing)h Fv(count)i FA(at)d(eac)o(h)189 162 y(receiv)o(e,)j(and)h(to)e (allo)o(w)i(its)f(use)h(after)e(a)h(call)h(to)f Fv(MPI)p 1123 162 14 2 v 16 w(PROBE)p FA(.)h(\()p Fr(End)f(of)i(r)n(ationale.)p FA(\))166 276 y(All)i(send)g(and)g(receiv)o(e)g(op)q(erations)f(use)h(the)f Fv(buf)p FA(,)h Fv(count)p FA(,)h Fv(datat)o(yp)q(e)p FA(,)g Fv(source)p FA(,)f Fv(dest)p FA(,)h Fv(tag)p FA(,)f Fv(comm)75 332 y FA(and)e Fv(status)i FA(argumen)o(ts)d(in)i(the)f(same)g(w)o(a)o(y)f (as)g(the)h(basic)h Fv(MPI)p 1201 332 V 16 w(SEND)f FA(and)g Fv(MPI)p 1523 332 V 16 w(RECV)h FA(op)q(erations)75 388 y(describ)q(ed)f(in)f (this)g(section.)166 446 y Fr(A)n(dditional)g(pr)n(op)n(osal)166 504 y FA(The)k Fv(status)i FA(v)m(ariable)g(also)e(con)o(tain)h(a)e Fv(size)i FA(\014eld)g(\()p Fv(status.size)h FA(in)f(C,)f Fv(ST)l(A)l (TUS\(MPI)p 1693 504 V 16 w(SIZE\))g FA(in)75 561 y(F)l(ortran\))12 b(that)i(enco)q(des)g(the)g(n)o(um)o(b)q(er)g(of)g(b)o(ytes)g(of)f(data)g(in) i(the)f(incoming)h(message.)k(Note)13 b(that)h(this)75 617 y(n)o(um)o(b)q(er)g(of)g(b)o(ytes)f(ma)o(y)g(b)q(e)i(di\013eren)o(t)f(than)g (the)g(n)o(um)o(b)q(er)g(of)f(b)o(ytes)h(actually)h(stored)e(in)i(the)f (receiv)o(er's)75 674 y(memory)l(.)166 808 y Fq(Discussion:)166 865 y Fu(The)h(alternativ)o(e)g(design)g(ma)o(y)e(a)o(v)o(oid)g(the)j(need)g (for)e(a)h(function)f(call)g(to)h(get)g Fo(tag)g Fu(and)f Fo(source)i Fu(\(standard)75 922 y(F)m(ortran)j(do)q(es)i(not)e(ha)o(v)o(e)g(macro)g (expansion\).)35 b(The)20 b(coun)o(terargumen)o(t)f(is)h(that)f(man)o(y)f (\(most?\))34 b(F)m(ortran)75 978 y(compilers)12 b(do)h(supp)q(ort)g (inlining,)e(although,)h(p)q(erhaps)i(di\013eren)o(tly)f(on)g(eac)o(h)h (system,)e(and)h(that)g(it)g(is)f(inelegan)o(t)75 1035 y(to)i(ha)o(v)o(e)f(a) h(di\013eren)o(t)h(access)h(metho)q(d)d(for)g(di\013eren)o(t)i(status)g (\014elds.)75 1269 y FB(1.4)59 b(Semantics)18 b(of)i(p)r(oint-to-p)r(oint)e (communication)75 1373 y FA(A)g(v)m(alid)h(MPI)e(implemen)o(tation)i(guaran)o (tees)e(certain)h(general)g(prop)q(erties)g(of)f(p)q(oin)o(t-to-p)q(oin)o(t)h (com-)75 1430 y(m)o(unication,)e(whic)o(h)g(are)f(describ)q(ed)i(in)f(this)f (section.)75 1558 y Fv(Order)46 b FA(Messages)14 b(are)h Fr(non-overtaking)p FA(,)g(within)h(eac)o(h)f(con)o(text:)20 b(if)c(t)o(w)o(o)d(messages)i(are)g (successiv)o(ely)75 1614 y(sen)o(t)10 b(from)g(the)g(same)g(source,)h(to)e (the)i(same)f(destination,)h(with)g(the)f(same)g(con)o(text,)h(then)f(the)g (messages)75 1671 y(are)17 b(receiv)o(ed)i(in)f(the)f(order)g(they)h(w)o(ere) f(sen)o(t.)26 b(I.e.,)17 b(if)h(the)f(receiv)o(er)i(p)q(osts)e(t)o(w)o(o)f (successiv)o(e)i(receiv)o(es)75 1727 y(that)g(b)q(oth)g(matc)o(h)g(either)i (message,)e(then)h(the)f(\014rst)h(receiv)o(e)g(will)h(receiv)o(e)g(the)e (\014rst)g(message,)h(and)75 1784 y(the)e(second)h(receiv)o(e)h(will)g (receiv)o(e)f(the)f(second)h(message.)26 b(This)18 b(requiremen)o(t)g (facilitates)g(matc)o(hing)75 1840 y(of)e(sends)h(to)f(receiv)o(es.)25 b(It)16 b(guaran)o(tees)g(that)g(message-passing)g(co)q(de)h(is)g (deterministic,)i(if)e(pro)q(cesses)75 1896 y(are)h(single-threaded)j(and)e (wildcard)h Fo(MPI)p 831 1896 13 2 v 14 w(ANY)p 930 1896 V 15 w(SOURCE)d FA(is)i(not)f(used)h(in)h(receiv)o(es.)31 b(\(Some)19 b(of)f(the)75 1953 y(calls)k(describ)q(ed)h(later,)g(suc)o(h)e(as)g Fv(MPI)p 771 1953 14 2 v 16 w(CANCEL)g FA(or)g Fv(MPI)p 1124 1953 V 16 w(W)l(AIT)l(ANY)p FA(,)h(are)f(additional)h(sources)f(of)75 2009 y(nondeterminism.\),)166 2067 y(If)c(a)g(pro)q(cess)h(has)f(a)g(single)h (thread)f(of)g(execution,)i(then)e(an)o(y)g(t)o(w)o(o)f(comm)o(unications)i (executed)75 2124 y(b)o(y)e(this)h(pro)q(cess)g(are)f(ordered.)23 b(On)17 b(the)f(other)g(hand,)h(if)g(the)f(pro)q(cess)h(is)g(m)o (ulti-threaded,)g(then)g(the)75 2180 y(seman)o(tics)e(of)f(thread)h (execution)h(ma)o(y)e(not)g(de\014ne)i(a)e(relativ)o(e)i(order)e(b)q(et)o(w)o (een)h(t)o(w)o(o)e(send)j(op)q(erations)75 2237 y(executed)g(b)o(y)f(t)o(w)o (o)f(distinct)j(threads:)i(the)d(op)q(erations)f(are)g(logically)i(concurren) o(t,)e(ev)o(en)h(though)f(one)75 2293 y(ph)o(ysically)22 b(precedes)e(the)g (other.)33 b(In)20 b(suc)o(h)g(a)f(case,)i(the)e(t)o(w)o(o)g(messages)g(sen)o (t)g(can)h(b)q(e)g(receiv)o(ed)h(in)75 2350 y(an)o(y)f(order.)37 b(Similarly)l(,)24 b(if)d(t)o(w)o(o)e(receiv)o(e)j(op)q(erations)f(that)f (are)g(logically)j(concurren)o(t)e(receiv)o(e)g(t)o(w)o(o)75 2406 y(successiv)o(ely)i(sen)o(t)d(messages,)h(then)g(the)g(t)o(w)o(o)f (messages)g(can)h(matc)o(h)f(the)h(t)o(w)o(o)e(receiv)o(es)j(in)f(either)75 2462 y(order.)31 b(Note)19 b(that)f(the)h(receiv)o(er)h(is)g(not)e(required)i (to)f(receiv)o(e)h(messages)e(in)i(the)f(order)g(they)g(w)o(ere)75 2519 y(sen)o(t,)c(if)g(they)h(ha)o(v)o(e)e(distinct)j(en)o(v)o(elop)q(es.)75 2647 y Fv(Progress)45 b FA(If)13 b(a)g(pair)g(of)f(matc)o(hing)h(send)h(and)f (receiv)o(es)g(ha)o(v)o(e)g(b)q(een)h(initiated)g(on)f(t)o(w)o(o)e(pro)q (cesses,)j(then)75 2704 y(at)k(least)h(one)h(of)e(these)h(t)o(w)o(o)f(op)q (erations)h(will)i(complete,)f(indep)q(enden)o(tly)i(of)c(other)h(actions)g (in)h(the)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 10 10 bop 75 -100 a FA(10)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y FA(system:)i(the)13 b(send)h(op)q(eration)f(will)h(complete,)g(unless)g(the)f(receiv)o(e)h(is)f (satis\014ed)h(b)o(y)e(another)h(message,)75 106 y(and)h(completes;)g(the)g (receiv)o(e)h(op)q(eration)f(will)h(complete,)g(unless)g(the)f(message)f(sen) o(t)g(is)i(consumed)f(b)o(y)75 162 y(another)h(matc)o(hing)g(receiv)o(e)h (that)f(w)o(as)f(p)q(osted)h(at)g(the)g(same)g(destination)h(pro)q(cess.)75 281 y Fv(F)o(airness)46 b FA(MPI)15 b(mak)o(es)g(no)g(guaran)o(tee)g(of)g Fr(fairness)f FA(in)i(the)g(handling)h(of)e(comm)o(unication.)21 b(Supp)q(ose)75 337 y(that)c(a)f(send)i(w)o(as)f(p)q(osted.)26 b(Then)18 b(it)f(is)h(p)q(ossible)h(that)e(the)g(destination)h(pro)q(cess)g (rep)q(eatedly)g(p)q(osts)75 394 y(a)d(receiv)o(e)h(that)f(matc)o(hes)f(this) i(send,)g(y)o(et)e(the)i(message)f(is)g(nev)o(er)h(receiv)o(ed,)g(b)q(ecause) g(it)g(is)f(eac)o(h)h(time)75 450 y(o)o(v)o(ertak)o(en)e(b)o(y)i(another)f (message,)g(sen)o(t)h(from)f(another)g(source.)21 b(Similarly)l(,)d(supp)q (ose)f(that)e(a)g(receiv)o(e)75 507 y(w)o(as)f(p)q(osted)i(b)o(y)f(a)g(m)o (ulti-threaded)h(pro)q(cess.)k(Then)c(it)f(is)h(p)q(ossible)h(that)d (messages)h(that)f(matc)o(h)h(this)75 563 y(receiv)o(e)20 b(are)g(rep)q (eatedly)g(receiv)o(ed,)i(y)o(et)d(the)g(receiv)o(e)i(is)f(nev)o(er)f (satis\014ed,)i(b)q(ecause)f(it)g(is)g(o)o(v)o(ertak)o(en)75 620 y(b)o(y)15 b(other)h(receiv)o(es)g(p)q(osted)g(at)e(this)i(no)q(de)h(\(b) o(y)e(other)g(executing)h(threads\).)21 b(It)15 b(is)h(the)g(programmer's)75 676 y(resp)q(onsibilit)o(y)i(to)c(prev)o(en)o(t)h(starv)m(ation)g(in)h(suc)o (h)g(situations.)75 795 y Fv(Resource)f(limitations)43 b FA(The)29 b(receiv)o(e)15 b(op)q(eration)f Fv(MPI)p 1043 795 14 2 v 16 w(RECV)g FA(clearly)i(is)e(non-lo)q(cal:)21 b(it)14 b(cannot)g(com-)-1903 b Fk(\))75 851 y FA(plete)16 b(unless)h(a)e(matc)o(hing)g(send)h(is)g(p)q (osted)f(b)o(y)g(another)g(pro)q(cess.)21 b(The)15 b(send)h(op)q(eration)g Fv(MPI)p 1745 851 V 16 w(SEND)75 908 y FA(also)c(is)h(non-lo)q(cal:)20 b(The)13 b(send)f(op)q(eration)h(ma)o(y)f(not)f(b)q(e)i(able)g(to)f(complete) h(b)q(efore)g(a)f(matc)o(hing)g(receiv)o(e)75 964 y(is)17 b(p)q(osted.)23 b(Messages)15 b(sen)o(t)h(b)o(y)h(an)f Fv(MPI)p 795 964 V 15 w(SEND)h FA(op)q(eration)g(can)f(b)q(e)h(bu\013ered)f(b)o(y)h(the)f(system,)g (so)f(as)75 1021 y(to)g(allo)o(w)h(the)g(blo)q(c)o(king)h(send)f(op)q (eration)g(to)f(complete)h(ahead)g(of)f(the)h(matc)o(hing)f(receiv)o(e.)22 b(Ho)o(w)o(ev)o(er,)75 1077 y(if)15 b(no)g(suc)o(h)g(bu\013er)f(space)h(is)g (a)o(v)m(ailable,)h(then)f(the)g(send)g(op)q(eration)g(blo)q(c)o(ks)g(un)o (til)h(a)e(matc)o(hing)h(receiv)o(e)75 1133 y(is)h(executed,)f(or)g (su\016cien)o(t)h(bu\013er)f(space)h(b)q(ecomes)g(a)o(v)m(ailable.)166 1190 y(In)g(some)g(situations,)g(lac)o(k)g(of)g(bu\013er)g(space)g(ma)o(y)f (lead)i(to)e(deadlo)q(c)o(k)i(situations.)23 b(This)16 b(is)h(illus-)75 1246 y(trated)d(b)o(y)i(the)f(examples)h(b)q(elo)o(w)g(\(in)o(v)o(olving)g(t) o(w)o(o)e(pro)q(cesses)h(with)h(ranks)f(0)f(and)i(1\))166 1303 y(The)21 b(follo)o(wing)i(program)d(is)i(safe,)g(and)f(can)h(succeed)g(ev)o (en)g(if)g(no)f(bu\013er)g(space)h(for)f(data)f(is)75 1359 y(a)o(v)m(ailable.)75 1455 y Ft(CALL)j(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\)) 75 1512 y(IF)h(\(rank.EQ.0\))123 1568 y(THEN)170 1625 y(CALL)g (MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(ierr\))170 1681 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g (status,)g(ierr\))123 1738 y(ELSE)95 b(!)23 b(rank.EQ.1)170 1794 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))170 1851 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g (0,)h(tag,)f(comm,)g(ierr\))75 1907 y(END)g(IF)166 2003 y FA(The)15 b(follo)o(wing)h(program)e(is)i(erroneous,)f(and)g(will)i(alw)o(a)o(ys)d (fail.)75 2099 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2156 y(IF)h(\(rank.EQ.0\))123 2212 y(THEN)170 2269 y(CALL)g (MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g (ierr\))170 2325 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(1,)h (tag,)f(comm,)g(ierr\))123 2382 y(ELSE)95 b(!)23 b(rank.EQ.1)170 2438 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))170 2495 y(CALL)h(MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g (0,)h(tag,)f(comm,)g(ierr\))75 2551 y(END)g(IF)166 2647 y FA(The)15 b(receiv)o(e)g(op)q(eration)g(of)f(the)h(\014rst)f(pro)q(cess)h(m)o(ust)f (complete)i(b)q(efore)f(its)f(send,)h(and)g(can)g(com-)75 2704 y(plete)j(only)g(if)g(the)f(matc)o(hing)g(send)h(of)f(the)g(second)h(pro)q (cessor)f(is)h(executed;)g(the)g(receiv)o(e)g(op)q(eration)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 11 11 bop 75 -100 a Fw(1.4.)34 b(SEMANTICS)16 b(OF)f(POINT-TO-POINT)i(COMMUNICA)l (TION)450 b FA(11)75 49 y(of)14 b(the)h(second)g(pro)q(cess)g(m)o(ust)f (complete)h(b)q(efore)g(its)f(send)i(and)e(can)h(complete)g(only)g(if)g(the)g (matc)o(hing)75 106 y(send)h(of)f(the)g(\014rst)g(pro)q(cess)g(is)h (executed.)21 b(This)15 b(program)f(will)j(deadlo)q(c)o(k.)166 162 y(The)c(follo)o(wing)h(program)f(is)g(unsafe,)h(and)f(will)i(deadlo)q(c)o (k)g(if)e(there)h(is)g(no)f(su\016cien)o(t)h(bu\013er)f(space.)75 251 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 308 y(IF)h(\(rank.EQ.0\))123 364 y(THEN)170 421 y(CALL)g(MPI_SEND\(sendbuf,)d (count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(ierr\))170 477 y(CALL)h (MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(1,)h(tag,)f(comm,)g(status,)g (ierr\))123 533 y(ELSE)95 b(!)23 b(rank.EQ.1)170 590 y(CALL)h (MPI_SEND\(sendbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g(ierr\))170 646 y(CALL)h(MPI_RECV\(recvbuf,)d(count,)i(MPI_REAL,)g(0,)h(tag,)f(comm,)g (status,)g(ierr\))75 703 y(END)g(IF)166 792 y FA(The)f(message)g(sen)o(t)f(b) o(y)h(eac)o(h)g(pro)q(cess)g(has)g(to)g(b)q(e)g(copied)h(out)f(b)q(efore)g (the)g(send)h(op)q(eration)75 848 y(returns)15 b(and)h(the)f(receiv)o(e)h(op) q(eration)g(starts.)i(F)l(or)d(the)g(program)g(to)f(complete,)i(it)f(is)h (necessary)g(that)75 905 y(at)h(least)g(one)h(of)f(the)g(t)o(w)o(o)f (messages)h(sen)o(t)g(is)h(bu\013ered.)27 b(Th)o(us,)17 b(this)h(program)e (can)i(succeed)h(only)e(if)75 961 y(the)f(comm)o(unication)h(system)f(can)g (bu\013er)g(at)g(least)g Fv(count)i FA(w)o(ords)d(of)h(data.)22 b(Otherwise,)17 b(a)f(deadlo)q(c)o(k)75 1018 y(o)q(ccurs.)166 1074 y(It)24 b(is)h(exp)q(ected)g(that)e(MPI)h(implemen)o(tations)h(will)h (follo)o(w)e(the)h(practice)f(of)g(most)f(curren)o(t)75 1131 y(message-passing)18 b(systems)f(b)o(y)g(bu\013ering)h(outgoing)g(messages)f (whenev)o(er)h(feasible.)28 b(Ho)o(w)o(ev)o(er,)17 b(the)75 1187 y(amoun)o(t)12 b(of)h(bu\013er)h(space)f(a)o(v)m(ailable)i(and)e(the)h (bu\013er)f(allo)q(cation)h(p)q(olicy)h(ma)o(y)e(di\013er)g(from)g(implemen-) 75 1243 y(tation)19 b(to)f(implemen)o(tation.)33 b(In)20 b(general,)g(the)g (programmer)d(can)j(mak)o(e)e(no)h(assumptions)h(on)f(the)75 1300 y(amoun)o(t)g(of)g(a)o(v)m(ailable)i(bu\013er)e(space,)i(and)e(ho)o(w)g (this)h(space)g(is)g(allo)q(cated.)33 b(The)20 b(only)g(totally)f(safe)75 1356 y(assumption)d(is)h(that)e(an)h(arbitrary)g(subset)g(of)g(the)g(send)g (op)q(erations)h(are)e(going)h(to)g(return)g(b)q(efore)g(a)75 1413 y(matc)o(hing)f(receiv)o(e)h(is)g(p)q(osted,)f(and)h(the)f(rest)g(will)h (blo)q(c)o(k)g(un)o(til)h(a)e(matc)o(hing)g(receiv)o(e)h(is)g(p)q(osted.)166 1469 y(Noblo)q(c)o(king)21 b(message)e(passing)i(op)q(erations,)g(as)e (describ)q(ed)j(in)f(Section)g(1.8,)e(can)h(b)q(e)h(used)g(to)75 1526 y(a)o(v)o(oid)f(the)g(need)h(for)f(bu\013ering)h(outgoing)f(messages.)34 b(This)21 b(prev)o(en)o(ts)f(deadlo)q(c)o(ks)h(due)g(to)f(lac)o(k)g(of)75 1582 y(bu\013er)14 b(space,)f(and)h(impro)o(v)o(es)g(p)q(erformance,)f(b)o(y) h(allo)o(wing)g(o)o(v)o(erlap)g(of)f(computation)g(and)h(comm)o(uni-)75 1639 y(cation,)h(and)h(a)o(v)o(oiding)f(the)h(o)o(v)o(erheads)f(of)f(allo)q (cating)j(bu\013ers)e(and)h(cop)o(ying)f(messages)g(in)o(to)g(bu\013ers.)75 1695 y(F)l(urthermore,)c(an)g(MPI)g(implemen)o(tation)i(will)g(pro)o(vide)f (information)f(and)g(con)o(trol)g(on)h(the)f(amoun)o(t)f(of)75 1752 y(a)o(v)m(ailable)k(bu\013er)f(space)g(and)g(on)f(the)h(bu\013ering)g(p) q(olicy)h(via)f(the)g(calls)g Fv(MPI)p 1378 1752 14 2 v 16 w(GET)p 1482 1752 V 17 w(BUFFER)p 1668 1752 V 17 w(P)l(ARAMS)p FA(,)75 1808 y Fv(MPI)p 160 1808 V 16 w(GET)p 264 1808 V 17 w(BUFFER)p 450 1808 V 17 w(P)l(ARAMS)i FA(and)f Fv(MPI)p 830 1808 V 16 w(USER)p 958 1808 V 17 w(SPECIFIES)p 1189 1808 V 16 w(BUFFER)h FA(describ)q(ed)h(in)e(Chapter)g Fz(??)p FA(.)189 1897 y Fr(R)n(ationale.)189 1970 y FA(MPI)e(could)i(mandate)f(bu\013ering)g (of)g(outgoing)f(messages)h(b)o(y)g(sp)q(ecifying)h(that)f(a)f(blo)q(c)o (king)i(send)189 2026 y(op)q(eration)20 b(is)g(lo)q(cal:)30 b(its)20 b(completion)g(dep)q(ends)i(only)e(on)f(the)h(state)f(of)g(the)h(lo) q(cal)h(executing)189 2083 y(pro)q(cess)16 b(and,)f(in)i(particular,)f(do)q (es)g(not)g(dep)q(end)h(on)f(the)g(o)q(ccurrence)h(of)e(a)h(matc)o(hing)f (receiv)o(e.)189 2139 y(The)h(implication)j(of)c(suc)o(h)i(de\014nition)h(is) f(that)e(a)h(blo)q(c)o(king)i(send)f(op)q(eration)f(will)i(fail)f(if)g (insuf-)189 2195 y(\014cien)o(t)j(bu\013er)g(space)f(is)h(a)o(v)m(ailable)i (when)e(the)f(send)h(executes)g(and)g(no)f(matc)o(hing)h(receiv)o(e)g(is)189 2252 y(p)q(osted.)30 b(This)19 b(is)g(the)f(implicit)j(seman)o(tics)e(of)f (blo)q(c)o(king)i(send)f(op)q(erations)f(on)h(systems)f(suc)o(h)189 2308 y(as)h(PVM)h([)p Fz(?)p FA(])34 b(and)20 b(P)l(ARMA)o(CS)g([)p Fz(?)p FA(])34 b(where,)21 b(in)g(fact,)f(the)g(\014rst)g(condition)h(is)g (su\016cien)o(t)g(to)189 2365 y(cause)16 b(failure.)22 b(W)l(e)16 b(ha)o(v)o(e)g(decided)h(on)f(a)g(more)f(lenien)o(t)i(de\014nition:)23 b(lac)o(k)16 b(of)f(bu\013er)h(space)g(will)189 2421 y(not)d(cause)i(a)f (send)g(to)g(fail;)h(instead)f(the)g(send)h(op)q(eration)g(blo)q(c)o(ks)f(un) o(til)i(a)d(matc)o(hing)h(receiv)o(e)h(or)189 2478 y(su\016cien)o(t)e (bu\013er)f(space)h(b)q(ecame)g(a)o(v)m(ailable.)20 b Fz(This)15 b(allo)o(ws)f(more)g(programs)f(to)h(complete)189 2534 y(successfully)p FA(.)26 b(On)18 b(the)f(do)o(wn)g(side,)i(erroneous)e(programs)f(will)j(fail) f(in)h(a)e(harder)g(to)g(detect)189 2591 y(deadlo)q(c)o(k)e(mo)q(de,)f (rather)g(than)g(b)o(y)g(a)g(simple)h(failure)h(of)d(a)h(send)h(op)q (eration.)20 b(Our)14 b(design)h(p)q(oin)o(t)189 2647 y(matc)o(hes)f(the)i (implicit)h(practice)f(of)f(comm)o(unication)g(libraries)i(suc)o(h)f(as)e (Express)i([)p Fz(?)p FA(],)d(NX)i([)p Fz(?)p FA(],)189 2704 y(CMMD)f([)p Fz(?)p FA(])19 b(and)c(EUI)h([)p Fz(?)p FA(].)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 12 12 bop 75 -100 a FA(12)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)189 49 y FA(Man)o(y)g(users)i(w)o(ould)f(lik) o(e)i(to)e(ha)o(v)o(e)g(a)g(b)q(etter)g(guaran)o(tee)g(on)g(the)g (bu\013ering)h(of)f(messages)g(sen)o(t)189 106 y(b)o(y)f(blo)q(c)o(king)j(op) q(erations;)e(i.e.,)g(a)f(guaran)o(tee)g(that)g(some)h(programs)e(will)k (succeed)f(on)f(all)g(im-)189 162 y(plemen)o(tations)g(of)f(MPI)g(ev)o(en)h (if)g(they)g(are)f(written)g(so)h(as)f(require)h(message)f(bu\013ering.)24 b(This)189 219 y(issue)16 b(arises)f(whether)g(w)o(e)g(mandate)g(bu\013ering) h(of)f(outgoing)g(messages)f(or)h(not.)189 292 y(The)e(function)g Fv(MPI)p 540 292 14 2 v 16 w(GET)p 644 292 V 17 w(BUFFER)p 830 292 V 17 w(P)l(ARAMS)h FA(describ)q(ed)h(in)e(Chapter)g Fz(??)19 b FA(partially)13 b(pro)o(vides)189 348 y(a)d(measure)h(of)f(a)o(v)m (ailalb)q(e)j(bu\013er)e(space;)h(ho)o(w)o(ev)o(er,)f(some)f(of)h(the)g (bu\013ering)g(parameters)f(ma)o(y)g(not)189 405 y(b)q(e)g(signi\014can)o(t)i (on)e(some)f(MPI)i(implemen)o(tations.)19 b(The)10 b(function)h Fv(MPI)p 1427 405 V 16 w(SUGGEST)p 1642 405 V 18 w(BUFFER)p 1829 405 V 17 w(P)l(ARAMS)189 461 y FA(pro)o(vides)k(user)h(con)o(trol)f(of)g (bu\013ering.)20 b(Ho)o(w)o(ev)o(er,)14 b(MPI)i(implemen)o(tations)g(are)f (not)g(mandated)189 517 y(to)f(act)h(on)g(the)g(user)h(suggestion.)189 591 y(W)l(e)i(ha)o(v)o(e)h(not)f(found)h(a)f(w)o(a)o(y)g(of)g(pro)o(viding)i (b)q(etter)f(measure)f(or)g(more)h(con)o(trol)f(of)g(bu\013ering)189 647 y(without)i(o)o(v)o(ersp)q(ecifying)i(MPI)e(implemen)o(tation.)37 b(In)21 b(general,)g(bu\013er)g(space)g(for)e(outgoing)189 703 y(messages)f(ma)o(y)h(b)q(e)h(allo)q(cated)g(at)f(the)g(sender)h(side,)h (the)e(receiv)o(er)h(side,)h(or)e(b)q(oth.)32 b(A)19 b(bu\013er)189 760 y(ma)o(y)13 b(b)q(e)i(dedicated)g(to)f(a)g(pair)g(of)g(comm)o(unicating)h (pro)q(cesses,)f(or)g(shared)g(b)o(y)g(all)h(outgoing)f(\(or)189 816 y(all)g(incoming\))h(comm)o(unication)g(to)e(a)h(pro)q(cess,)g(or)f (shared)h(b)o(y)g(all)h(outgoing)e(\(or)g(all)i(incoming\))189 873 y(comm)o(unication)f(to)g(a)g(pro)q(cessor.)19 b(Bu\013ers)14 b(ma)o(y)f(b)q(e)i(allo)q(cated)g(in)g(sp)q(ecial)g(device)h(memory)l(,)d(or) 189 929 y(in)k(real)h(memory)e(\(user)h(or)f(k)o(ernel\),)i(or)e(in)i (virtual)f(memory)g(\(user)f(or)h(k)o(ernel\).)25 b(A)17 b(de\014nition)189 986 y(that)10 b(precludes)i(some)f(of)f(these)i(c)o(hoices)f(\(as)f(w)o(ell)i (as)f(more)f(designs)i(w)o(e)f(ha)o(v)o(e)f(not)h(y)o(et)f(imagined\))189 1042 y(will)16 b(signi\014can)o(tly)h(hamp)q(er)f(p)q(erformance)f(on)g(some) g(systems.)20 b(\()p Fr(End)15 b(of)i(r)n(ationale.)p FA(\))75 1183 y FB(1.5)59 b(Data)20 b(T)-5 b(yp)r(e)19 b(Matching)75 1285 y FA(One)d(can)f(think)h(of)f(message)g(transmission)g(as)g(consisting)h (of)f(three)g(phases:)131 1368 y(1.)22 b(Data)14 b(is)h(pulled)j(out)d(of)f (the)i(send)f(bu\013er)h(and)f(a)g(message)g(is)g(assem)o(bled)131 1458 y(2.)22 b(A)15 b(message)g(is)g(transferred)g(from)f(sender)i(to)f (receiv)o(er)131 1548 y(3.)22 b(Data)14 b(is)h(pulled)j(from)c(the)h (incoming)i(message)e(and)g(disassem)o(bled)i(in)o(to)e(the)g(receiv)o(e)h (bu\013er)166 1631 y(T)o(yp)q(e)k(matc)o(hing)f(has)h(to)e(b)q(e)j(observ)o (ed)e(at)g(eac)o(h)g(of)h(these)f(three)h(phases:)29 b(The)19 b(t)o(yp)q(e)h(of)f(eac)o(h)75 1687 y(v)m(ariable)i(in)f(the)g(sender)g (bu\013er)g(has)g(to)f(matc)o(h)g(the)g(t)o(yp)q(e)h(sp)q(eci\014ed)i(for)d (that)g(en)o(try)g(b)o(y)h(the)f(send)75 1744 y(op)q(eration;)d(the)f(t)o(yp) q(e)h(sp)q(eci\014ed)i(b)o(y)e(the)f(send)i(op)q(eration)e(has)h(to)f(matc)o (h)g(the)h(t)o(yp)q(e)f(sp)q(eci\014ed)j(b)o(y)e(the)75 1800 y(receiv)o(e)g(op)q(eration;)e(and)h(the)g(t)o(yp)q(e)f(of)h(eac)o(h)f(v)m (ariable)i(in)g(the)e(receiv)o(e)i(bu\013er)f(has)f(to)g(matc)o(h)g(the)h(t)o (yp)q(e)75 1857 y(sp)q(eci\014ed)f(for)e(that)g(en)o(try)g(b)o(y)g(the)g (receiv)o(e)h(op)q(eration.)20 b(A)12 b(program)f(that)h(fails)h(to)e(observ) o(e)h(these)h(three)75 1913 y(rules)j(is)g(erroneous.)166 1970 y(T)l(o)f(de\014ne)i(t)o(yp)q(e)f(matc)o(hing)g(more)f(precisely)l(,)j(w)o(e) d(need)i(to)e(deal)h(with)g(t)o(w)o(o)f(issues:)22 b(matc)o(hing)15 b(of)75 2026 y(t)o(yp)q(es)e(of)g(the)h(host)f(language)g(with)h(t)o(yp)q(es) g(sp)q(eci\014ed)h(in)f(comm)o(unication)h(op)q(erations;)e(and)h(matc)o (hing)75 2083 y(of)h(t)o(yp)q(es)g(at)g(sender)g(and)h(receiv)o(er.)166 2139 y(A)h(t)o(yp)q(e)f(sp)q(eci\014ed)j(for)d(an)h(en)o(try)f(b)o(y)h(a)f (send)h(op)q(eration)g(matc)o(hes)f(the)h(t)o(yp)q(e)g(sp)q(eci\014ed)i(for)d (that)75 2195 y(en)o(try)11 b(b)o(y)h(a)f(receiv)o(e)h(op)q(eration)g(if)g(b) q(oth)f(op)q(erations)h(use)g(iden)o(tical)h(names:)36 b Fv(MPI)p 1500 2195 V 16 w(INTEGER)12 b FA(matc)o(hes)75 2252 y Fv(MPI)p 160 2252 V 16 w(INTEGER)p FA(,)j Fv(MPI)p 475 2252 V 16 w(REAL)h FA(matc)o(hes)30 b Fv(MPI)p 891 2252 V 15 w(REAL)p FA(,)16 b(and)f(so)g(on.)166 2308 y(The)k(t)o(yp)q(e)g(of)f(a)g(v)m(ariable)i(in)g(a) e(host)h(program)e(matc)o(hes)h(the)h(t)o(yp)q(e)g(sp)q(eci\014ed)i(in)e(the) g(comm)o(u-)75 2365 y(nication)g(op)q(eration)g(if)f(the)h(datat)o(yp)q(e)f (name)g(used)h(b)o(y)f(that)f(op)q(eration)i(corresp)q(onds)f(to)g(the)g (basic)75 2421 y(t)o(yp)q(e)i(of)g(the)g(host)g(program)f(v)m(ariable:)31 b(an)20 b(en)o(try)f(with)i(t)o(yp)q(e)f(name)40 b Fv(MPI)p 1449 2421 V 16 w(INTEGER)21 b FA(matc)o(hes)e(a)75 2478 y(F)l(ortran)c(v)m (ariable)j(of)f(t)o(yp)q(e)f Ft(INTEGER)p FA(,)g(an)g(en)o(try)g(with)h(t)o (yp)q(e)g(name)g Fv(MPI)p 1368 2478 V 16 w(REAL)g FA(matc)o(hes)f(a)g(F)l (ortran)75 2534 y(v)m(ariable)j(of)e(t)o(yp)q(e)g Ft(REAL)p FA(,)f(and)i(so)f(on.)26 b(There)18 b(is)f(one)h(exception)g(to)f(this)h (last)f(rule:)25 b(An)18 b(en)o(try)f(with)75 2591 y(t)o(yp)q(e)h(name)37 b Fv(MPI)p 408 2591 V 15 w(BYTE)19 b FA(can)f(b)q(e)h(used)g(to)e(matc)o(h)h (an)o(y)g(b)o(yte)g(of)g(storage)e(\(on)i(a)g(b)o(yte-addressable)75 2647 y(mac)o(hine\),)d(irresp)q(ectiv)o(e)i(of)d(the)h(datat)o(yp)q(e)g(of)f (the)h(v)m(ariable)i(that)d(con)o(tains)h(this)g(b)o(yte.)20 b(The)15 b(v)m(alue)h(of)75 2704 y(the)f(message)g(en)o(try)g(will)i(b)q(e)f (the)f(binary)h(v)m(alue)g(of)f(the)g(corresp)q(onding)h(b)o(yte)f(in)h (memory)l(.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 13 13 bop 75 -100 a Fw(1.5.)34 b(D)o(A)l(T)l(A)15 b(TYPE)g(MA)l(TCHING)1105 b FA(13)166 49 y(W)l(e)15 b(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)143 144 y Fl(\017)23 b FA(Comm)o(unication)11 b(of)f(t)o(yp)q(ed)i(v)m(alues)g (\(e.g.,)e(with)h(datat)o(yp)q(e)g(di\013eren)o(t)g(from)21 b Fv(MPI)p 1586 144 14 2 v 16 w(BYTE)p FA(\),)11 b(where)189 200 y(the)k(datat)o(yp)q(es)f(of)g(the)h(corresp)q(onding)h(en)o(tries)f(in)h (the)f(sender)g(program,)e(in)j(the)f(send)g(call,)h(in)189 257 y(the)f(receiv)o(e)h(call)g(and)g(in)g(the)f(receiv)o(er)h(program)e (should)i(all)g(matc)o(h.)143 352 y Fl(\017)23 b FA(Comm)o(unication)16 b(of)g(un)o(t)o(yp)q(ed)h(v)m(alues)g(\(e.g.,)e(of)g(datat)o(yp)q(e)h Fv(MPI)p 1330 352 V 16 w(BYTE)p FA(\),)g(where)g(b)q(oth)g(sender)189 408 y(and)g(receiv)o(er)h(use)f(the)g(datat)o(yp)q(e)g Fv(MPI)p 881 408 V 15 w(BYTE)p FA(.)h(In)f(this)h(case,)f(there)g(are)g(no)f (requiremen)o(ts)i(on)189 464 y(the)d(t)o(yp)q(es)h(of)f(the)g(corresp)q (onding)h(en)o(tries)g(in)h(the)e(sender)h(and)g(the)f(receiv)o(er)h (programs,)e(nor)h(is)189 521 y(it)h(required)h(that)f(they)g(b)q(e)h(the)f (same.)166 616 y(The)g(follo)o(wing)h(examples)g(illustrate)h(these)e(t)o(w)o (o)f(cases.)166 672 y(First)h(program:)75 767 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 824 y(IF\(rank.EQ.0\))123 880 y(THEN)170 936 y(CALL)h(MPI_SEND\(a\(1\),)e(10,)h(MPI_REAL,)g(1,)g(tag,)h (comm,)f(ierr\))123 993 y(ELSE)170 1049 y(CALL)h(MPI_RECV\(a\(1\),)e(15,)h (MPI_REAL,)g(0,)g(tag,)h(comm,)f(status,)g(ierr\))75 1106 y(END)g(IF)166 1201 y FA(This)c(co)q(de)f(is)h(correct)e(if)i(b)q(oth)f(sender)h(and)f (receiv)o(er)h(programs)d(ha)o(v)o(e)i(allo)q(cated)h(consecutiv)o(e)75 1257 y(storage)14 b(for)h(ten)g(real)g(n)o(um)o(b)q(ers,)g(starting)g(from)f Fv(a\(1\))p FA(.)166 1314 y(Second)i(program:)75 1409 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1465 y(IF\(rank.EQ.0\))123 1521 y(THEN)170 1578 y(CALL)h(MPI_SEND\(a\(1\),)e(10,)h(MPI_REAL,)g(1,)g (tag,)h(comm,)f(ierr\))123 1634 y(ELSE)170 1691 y(CALL)h(MPI_RECV\(a\(1\),)e (40,)h(MPI_BYTE,)g(0,)g(tag,)h(comm,)f(status,)g(ierr\))75 1747 y(END)g(IF)166 1842 y FA(This)d(co)q(de)g(is)f(erroneous,)h(since)g (sender)g(and)g(receiv)o(er)g(do)f(not)f(pro)o(vide)i(matc)o(hing)f(datat)o (yp)q(e)75 1898 y(argumen)o(ts.)166 1955 y(Third)d(program:)75 2050 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2106 y(IF\(rank.EQ.0\))123 2163 y(THEN)170 2219 y(CALL)h(MPI_SEND\(a\(1\),)e (40,)h(MPI_BYTE,)g(1,)g(tag,)h(comm,)f(ierr\))123 2276 y(ELSE)170 2332 y(CALL)h(MPI_RECV\(a\(1\),)e(60,)h(MPI_BYTE,)g(0,)g(tag,)h(comm,)f (status,)g(ierr\))75 2389 y(END)g(IF)166 2483 y FA(This)16 b(co)q(de)f(is)h(correct,)f(irresp)q(ectiv)o(e)h(of)f(the)g(t)o(yp)q(e)h(of)e Fv(a\(1\))g FA(and)i(follo)o(wing)g(v)m(ariables)g(in)g(store.)189 2591 y Fr(A)n(dvic)n(e)11 b(to)h(users.)56 b FA(If)11 b(a)g(bu\013er)f(of)h (t)o(yp)q(e)f Fo(MPI)p 974 2591 13 2 v 15 w(BYTE)g FA(is)h(passed)g(as)f(an)h (argumen)o(t)f(to)g Fv(MPI)p 1732 2591 14 2 v 16 w(SEND)p FA(,)69 b Fk(\()189 2647 y FA(then)14 b(MPI)g(will)h(send)g(the)f(data)f(stored)g(at) g(con)o(tiguous)h(lo)q(cations,)h(starting)e(from)g(the)h(address)189 2704 y(indicated)22 b(b)o(y)e(the)h Fv(buf)g FA(argumen)o(t.)35 b(This)21 b(ma)o(y)f(ha)o(v)o(e)g(unexp)q(ected)i(results)f(when)g(the)g (data)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 14 14 bop 75 -100 a FA(14)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)189 49 y FA(la)o(y)o(out)c(is)i(not)e(as)h(a) f(casual)i(user)f(w)o(ould)g(exp)q(ect)h(it)f(to)g(b)q(e:)19 b(F)l(or)12 b(example,)i(some)f(compilers)h(pass)189 106 y(a)e(F)l(ortran)f Fo(CHARA)o(CTER)g FA(argumen)o(t)h(as)g(a)g(structure)g(that)g(con)o(tains)g (the)h(c)o(haracter)f(length)h(and)189 162 y(a)k(p)q(oin)o(ter)i(to)e(the)h (actual)g(string.)29 b(In)18 b(suc)o(h)h(en)o(vironmen)o(t,)f(sending)h(or)f (receiving)h(a)f(F)l(ortran)189 219 y Fo(CHARA)o(CTER)f FA(v)m(ariable)j (using)g(the)f Fo(MPI)p 907 219 13 2 v 15 w(BYTE)f FA(will)j(not)e(ha)o(v)o (e)f(the)i(an)o(ticipated)g(result.)32 b(F)l(or)189 275 y(this)15 b(reason,)f(the)h(user)g(is)g(advised)h(to)e(use)h(t)o(yp)q(ed)g(comm)o (unication)h(whenev)o(er)f(p)q(ossible.)22 b(\()p Fr(End)189 332 y(of)16 b(advic)n(e)g(to)h(users.)p FA(\))75 454 y Fs(1.5.1)49 b(T)l(yp)q(e)17 b Fo(MPI)p 417 454 V 14 w(CHARA)o(CTER)75 541 y FA(The)23 b(t)o(yp)q(e)11 b Fo(MPI)p 351 541 V 14 w(CHARA)o(CTER)f FA(matc)o(hes)h(one)g(c)o(haracter)g(of)f(a)h(F)l(ortran)f(v)m(ariable)j(of)e (t)o(yp)q(e)g Fo(CHARA)o(CTER)p FA(.)-1905 b Fk(\))75 597 y FA(F)l(ortran)12 b(v)m(ariables)j(of)e(t)o(yp)q(e)h Fo(CHARA)o(CTER)e FA(or)h(substrings)h(are)f(transferred)g(as)g(if)h(they)g(w)o(ere)f(arra)o (ys)f(of)75 654 y(c)o(haracters.)18 b(The)c Fv(buf)f FA(argumen)o(t)g(should) h(b)q(e)f(a)g(reference)h(to)e(a)h(string)g(or)f(substring)i(that)e(starts)g (with)75 710 y(the)j(\014rst)g(c)o(haracter)g(of)f(the)i(comm)o(unication)g (bu\013er.)j(This)d(is)g(illustrated)g(in)g(the)g(example)g(b)q(elo)o(w.)75 817 y Ft(CHARACTER*10)22 b(a)75 874 y(CHARACTER*10)g(b)75 987 y(CALL)h(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1043 y(IF\(rank.EQ.0\))123 1100 y(THEN)170 1156 y(CALL)h(MPI_SEND\(a,)e(5,)i(MPI_CHARACTER,)e(1,)h(tag,) h(comm,)f(ierr\))123 1213 y(ELSE)170 1269 y(CALL)h(MPI_RECV\(b\(6:10\),)d(5,) j(MPI_CHARACTER,)e(0,)h(tag,)h(comm,)f(status,)g(ierr\))75 1326 y(END)g(IF)166 1433 y FA(The)14 b(last)h(\014v)o(e)f(c)o(haracters)g(of) f(string)i Fo(b)f FA(at)g(pro)q(cess)g(1)g(are)g(replaced)i(b)o(y)e(the)g (\014rst)g(\014v)o(e)h(c)o(haracters)75 1489 y(of)g(string)g Fo(a)g FA(at)g(pro)q(cess)g(0.)166 1629 y Fq(Implemen)o(tati)o(on)10 b(note:)33 b Fu(Some)10 b(compilers)h(pass)h(F)m(ortran)f Fo(CHARA)o(CTER)f Fu(argumen)o(ts)h(as)h(a)f(structure)75 1685 y(with)16 b(a)f(\014eld)h (length)g(and)g(a)g(p)q(oin)o(ter)g(to)g(the)g(actual)g(string.)24 b(In)16 b(suc)o(h)h(en)o(vironmen)o(t,)e(the)i(MPI)f(call)f(need)i(to)75 1741 y(dereference)g(the)d(p)q(oin)o(ter)g(in)g(order)g(to)g(reac)o(h)h(the)f (string.)75 1945 y Fv(Alternative)i(De\014nition)75 2032 y FA(The)h(t)o(yp)q(e)g Fo(MPI)p 351 2032 V 14 w(CHARA)o(CTER)e FA(matc)o(hes)h(a)h(F)l(ortran)f(v)m(ariable)i(of)e(t)o(yp)q(e)h Fo(CHARA)o(CTER)e FA(or)h(a)h(substring)75 2088 y(of)e(arbitrary)f(length.)75 2195 y Ft(CHARACTER*10)22 b(a)75 2252 y(CHARACTER*10)g(b)75 2365 y(CALL)h(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 2421 y(IF\(rank.EQ.0\))123 2478 y(THEN)170 2534 y(CALL)h(MPI_SEND\(a\(1:5\),)d(1,) j(MPI_CHARACTER,)e(1,)i(tag,)f(comm,)g(ierr\))123 2591 y(ELSE)170 2647 y(CALL)h(MPI_RECV\(b\(6:10\),)d(1,)j(MPI_CHARACTER,)e(0,)h(tag,)h(comm,) f(status,)g(ierr\))75 2704 y(END)g(IF)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 15 15 bop 75 -100 a Fw(1.6.)34 b(D)o(A)l(T)l(A)15 b(CONVERSION)1197 b FA(15)166 49 y(The)14 b(last)h(\014v)o(e)f(c)o(haracters)g(of)f(string)i Fo(b)f FA(at)g(pro)q(cess)g(1)g(are)g(replaced)i(b)o(y)e(the)g(\014rst)g (\014v)o(e)h(c)o(haracters)75 106 y(of)g(string)g Fo(a)g FA(at)g(pro)q(cess)g (0.)166 163 y(Mismatc)o(hes)f(b)q(et)o(w)o(een)g(the)g(length)g(of)f(an)h (incoming)h(c)o(haracter)e(and)h(the)g(length)h(of)e(a)h(c)o(haracter)75 219 y(in)23 b(the)g(receiv)o(e)g(bu\013er)f(are)g(handled)i(as)e(in)h(F)l (ortran)f(c)o(haracter)f(assignmen)o(ts:)34 b(if)23 b(the)g(incoming)75 275 y(c)o(haracter)16 b(is)h(longer)f(than)h(the)f(target)f(c)o(haracter)h (then)h(it)f(is)h(truncated;)g(if)g(it)f(is)h(shorter)f(then)h(it)f(is)75 332 y(padded)g(with)g(blanks.)166 465 y Fq(Discussion:)33 b Fu(The)14 b(later)g(de\014nition)f(is)g(more)g(elegan)o(t)g(than)h(the)g (former)e(one.)19 b(Unfortunately)m(,)12 b(it)h(do)q(es)75 515 y(not)h(sit)g(w)o(ell)f(with)g(the)i(use)g(of)e(deriv)o(ed)h(datat)o(yp)q (es.)166 565 y(Consider)c(a)f(system)g(where)i(the)f(length)g(of)f(a)g(c)o (haracter)i(argumen)o(t)d(is)h(passed)i(as)f(an)f(additional)f(argumen)o(t.) 75 615 y(Assume)16 b(that)h(the)g(program)e(pic)o(ks)h(the)h(address)h(of)e (a)g(string)g(with)g(a)g(call)g(to)g Fo(MPI)p 1448 615 13 2 v 14 w(ADRRESS)p Fu(,)f(next)i(builds)75 664 y(the)c(deriv)o(ed)g(datat)o(yp) q(e)g(for)f(a)g(bu\013er)h(that)g(includes)g(the)g(string)f(with)g(a)h(call)e (to)i Fo(MPI)p 1435 664 V 14 w(STRUCT)p Fu(.)e(The)i(length)g(of)75 714 y(the)i(string)g(w)o(as)g(passed)h(as)f(an)g(extra)g(argumen)o(t)f(to)g (the)i(function)e Fo(MPI)p 1250 714 V 15 w(ADDRESS)p Fu(,)f(whic)o(h)i (ignored)g(it.)21 b(The)75 764 y(information)8 b(that)j(a)g(particular)g(v)n (ariable)f(is)h(of)g(c)o(haracter)h(t)o(yp)q(e)g(is)f(passed)h(as)f(an)g (argumen)o(t)f(to)h Fo(MPI)p 1690 764 V 14 w(STRUCT)p Fu(,)75 814 y(whic)o(h)g(has)g(no)g(w)o(a)o(y)f(to)g(reco)o(v)o(er)j(the)e (information)d(on)j(the)g(c)o(haracter)i(length)d(\(unless)i(one)f(w)o(an)o (ts)g Fo(MPI)p 1701 814 V 15 w(STRUCT)75 864 y Fu(to)k(access)i(a)e(sym)o(b)q (ol)f(table\).)22 b(Th)o(us,)15 b(w)o(e)h(w)o(ould)e(need)i Fo(MPI)p 1049 864 V 15 w(ADDRESS)e Fu(to)h(alw)o(a)o(ys)g(c)o(hec)o(k)h (whether)g(an)f(extra)75 914 y(argumen)o(t)e(\(c)o(haracter)i(length\))f(w)o (as)g(passed)h(and,)e(if)g(so,)h(sa)o(v)o(e)g(the)g(v)n(alue)f(of)h(this)g (extra)g(argumen)o(t.)166 964 y Fo(MPI)p 243 964 V 14 w(SEND)g Fu(also)f(needs)i(sp)q(ecial)f(co)q(de)g(to)g(handle)f(the)i(case)f(where)h Fo(t)o(yp)q(e)g(=)f(MPI)p 1476 964 V 14 w(CHARA)o(CTER)p Fu(.)d(In)j(this)75 1014 y(case,)e(the)f(v)n(alue)f(of)g(the)h(extra)g(length)f(argumen)o(t)f (needs)j(b)q(e)f(used)h(to)e(\014nd)h(the)g(length)f(of)g(the)h(outgoing)f (message.)166 1070 y(The)18 b(basic)g(problem)e(is)i(that)f(w)o(e)h(assumed)f (that)h(a)g(deriv)o(ed)g(datat)o(yp)q(e)g(is)f(a)g(static)h(pattern)h(\(a)e (list)h(of)75 1127 y(displacemen)o(ts)11 b(and)h(lengths\))g(that,)g (together)g(with)g(a)f(base)h(address,)h(iden)o(ti\014es)f(the)h(set)f(of)f (memory)e(lo)q(cations)75 1183 y(that)i(form)f(the)i(comm)o(unicati)o(on)d (bu\013er.)18 b(With)10 b(the)i(second)h(de\014nition)d(for)h Fo(MPI)p 1372 1183 V 15 w(CHARA)o(CTER)p Fu(,)e(w)o(e)i(no)o(w)g(ha)o(v)o(e) 75 1240 y(a)17 b(dynamic)f(deriv)o(ed)i(datat)o(yp)q(e)g(that)f(assumes)g(v)n (alue)g(\(length\))h(only)e(when)i(b)q(ound)g(to)f(a)g(sp)q(eci\014c)i(lo)q (cation.)75 1296 y(Mo)o(ving)14 b(to)o(w)o(ard)h(suc)o(h)g(dynamic,)f (parameterized)h(datat)o(yp)q(es)h(is)f(tempting:)k(the)c(need)h(for)f(them)f (is)h(lik)o(ely)f(to)75 1353 y(arise)h(in)f(other)i(places)f(in)g(F)m(ortran) f(90)g(or)h(C++,)g(whenev)o(er)h(language)e(t)o(yping)g(is)h(dynamic,)e(and)i (ob)r(jects)h(are)75 1409 y(self-t)o(yp)q(ed.)28 b(\(An)17 b(example:)23 b(an)17 b(arra)o(y)g(section)h(in)e(F)m(ortran)h(90,)g(with)g (a)f(compiler)g(that)h(uses)h(dop)q(e)g(v)o(ectors)75 1466 y(to)d(main)o(tain)d(the)k(dynamic)d(information)f(on)j(data)g(la)o(y)o(out.) 20 b(The)15 b(righ)o(t)g(buzzw)o(ord)h(is)e(\\p)q(olymorphic".\))20 b(The)75 1522 y(question)15 b(is)f(whether)i(w)o(e)e(w)o(an)o(t)g(to)h (handle)f(this)h(at)f(the)h(curren)o(t)h(MPI)f(la)o(y)o(er,)e(or)i(as)f(a)g (metastructure)i(on)e(top)75 1578 y(of)f(curren)o(t)j(MPI.)75 1807 y FB(1.6)59 b(Data)20 b(conversion)75 1909 y FA(One)c(of)f(the)g(goals)g (of)g(MPI)g(is)h(to)e(supp)q(ort)i(parallel)g(computations)f(across)g (heterogeneous)g(en)o(viron-)75 1965 y(men)o(ts.)20 b(Comm)o(unication)15 b(in)h(a)f(heterogeneous)g(en)o(vironmen)o(t)h(ma)o(y)e(require)i(data)f(con) o(v)o(ersions.)166 2022 y(W)l(e)g(use)h(the)f(follo)o(wing)h(terminology:)75 2118 y Fz(t)o(yp)q(e)h(con)o(v)o(ersion)22 b FA(c)o(hanges)16 b(the)h(datat)o(yp)q(e)f(of)g(a)g(v)m(alue,)h(e.g.)23 b(b)o(y)17 b(rounding)g(a)f Fv(REAL)h FA(to)f(an)g Fv(INTE-)189 2174 y(GER)p FA(.)75 2269 y Fz(represen)o(tation)h(con)o(v)o(ersion)22 b FA(c)o(hanges)16 b(the)h(binary)g(represen)o(tation)f(of)g(a)g(v)m(alue,)h (e.g.)23 b(from)16 b(Hex)189 2326 y(\015oating)f(p)q(oin)o(t)g(to)g(IEEE)g (\015oating)h(p)q(oin)o(t.)166 2421 y(The)d(t)o(yp)q(e)g(matc)o(hing)g(rules) g(imply)h(that)e(MPI)h(comm)o(unication)h(nev)o(er)f(en)o(tails)g(t)o(yp)q(e) g(con)o(v)o(ersion.)75 2478 y(On)20 b(the)f(other)f(hand,)i(MPI)f(requires)h (that)e(a)h(represen)o(tation)g(con)o(v)o(ersion)g(b)q(e)h(p)q(erformed)f (when)g(a)75 2534 y(t)o(yp)q(ed)i(v)m(alue)h(is)f(transferred)f(across)g(en)o (vironmen)o(ts)h(that)f(use)h(di\013eren)o(t)f(represen)o(tations)h(for)f (the)75 2591 y(datat)o(yp)q(e)g(of)g(this)h(v)m(alue.)38 b(MPI)20 b(do)q(es)h(not)f(sp)q(ecify)i(rules)g(for)d(represen)o(tation)i(con)o(v)o (ersion.)36 b(Suc)o(h)75 2647 y(con)o(v)o(ersion)21 b(is)g(exp)q(ected)i(to)d (preserv)o(e)h(in)o(teger,)h(logical)g(or)e(c)o(haracter)h(v)m(alues,)i(and)e (to)f(con)o(v)o(ert)g(a)75 2704 y(\015oating)15 b(p)q(oin)o(t)h(v)m(alue)g (to)f(the)g(nearest)g(v)m(alue)h(that)f(can)g(represen)o(ted)h(on)f(the)g (target)f(system.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 16 16 bop 75 -100 a FA(16)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(Ov)o(er\015o)o(w)10 b(and)h(under\015o)o(w)g(exceptions)h(ma)o(y)e(o)q(ccur)i(during)f (\015oating)g(p)q(oin)o(t)g(con)o(v)o(ersions;)h(con)o(v)o(er-)75 106 y(sion)i(of)e(in)o(tegers)i(or)e(c)o(haracters)h(ma)o(y)f(also)h(lead)h (to)f(exceptions)h(when)g(a)e(v)m(alue)j(that)d(can)h(represen)o(ted)75 162 y(in)18 b(one)g(system)f(cannot)g(b)q(e)h(represen)o(ted)g(in)h(the)e (other)g(system.)26 b(An)18 b(exception)h(o)q(ccurring)f(during)75 219 y(represen)o(tation)f(con)o(v)o(ersion)f(results)h(in)h(a)e(failure)i(of) e(the)g(comm)o(unication;)i(an)e(error)g(o)q(ccurs)h(either)75 275 y(in)f(the)f(send)h(op)q(eration,)f(or)g(the)g(receiv)o(e)h(op)q (eration,)f(or)g(b)q(oth.)166 337 y(If)22 b(a)f(v)m(alue)i(sen)o(t)e(in)h(a)f (message)g(is)h(un)o(t)o(yp)q(ed)h(\(i.e.,)f(of)f(t)o(yp)q(e)h Fv(MPI)p 1370 337 14 2 v 15 w(BYTE)p FA(\),)g(then)g(the)f(binary)75 393 y(represen)o(tation)15 b(of)f(the)h(b)o(yte)g(stored)f(at)g(the)h(receiv) o(er)h(is)f(iden)o(tical)i(to)d(the)h(binary)h(represen)o(tation)e(of)75 450 y(the)f(b)o(yte)g(loaded)h(at)e(the)h(sender.)20 b(This)14 b(holds)f(true,)g(whether)g(sender)h(and)f(receiv)o(er)h(run)f(in)h(the)f (same)75 506 y(or)i(in)h(distinct)g(en)o(vironmen)o(ts;)f(no)g(represen)o (tation)h(con)o(v)o(ersion)f(is)h(required.)166 568 y(Note)d(that)g(no)g(con) o(v)o(ersion)h(ev)o(er)f(o)q(ccurs)h(when)g(an)f(MPI)g(program)g(executes)h (in)g(a)f(homogeneous)75 624 y(system,)i(where)h(all)g(pro)q(cesses)g(run)g (in)g(the)g(same)f(en)o(vironmen)o(t.)22 b(Also)16 b(note)f(the)h(di\013eren) o(t)f(b)q(eha)o(vior)75 681 y(of)e Fv(MPI)p 210 681 V 15 w(BYTE)h FA(and)f(of)g Fv(MPI)p 575 681 V 15 w(CHAR)p FA(.)h(A)f(bu\013er)g (descriptor)g(en)o(try)g(with)g(datat)o(yp)q(e)g(of)f Fv(MPI)p 1660 681 V 16 w(CHAR)i FA(can)75 737 y(only)f(matc)o(h)f(a)g(C)g(v)m(ariable) i(of)e(t)o(yp)q(e)h Ft(CHAR)p FA(;)e(and)i(represen)o(tation)g(con)o(v)o (ersion)f(ma)o(y)g(o)q(ccur)h(when)g(v)m(alues)75 793 y(of)g(t)o(yp)q(e)h Fv(MPI)p 311 793 V 15 w(CHAR)h FA(are)e(transferred.,)g(e.g.,)f(from)h(an)g (EBCDIC)h(enco)q(ding)h(to)e(an)g(ASCI)q(I)i(enco)q(ding.)166 855 y(Consider)h(the)f(previous)h(three)f(examples)h(on)f(page)g(13.)166 917 y(The)c(\014rst)f(program)f(is)i(correct,)g(assuming)g(b)q(oth)f(sender)i (and)e(receiv)o(er)i(declared)f(ten)g(consecutiv)o(e)75 973 y(real)20 b(v)m(ariables)h(in)g(storage)e(starting)g(at)g Fv(a\(1\))p FA(.)33 b(If)20 b(the)g(sender)g(and)g(receiv)o(er)h(execute)f(in)h (di\013eren)o(t)75 1030 y(en)o(vironmen)o(ts,)12 b(then)f(the)g(ten)f(real)i (v)m(alues)f(that)g(are)f(fetc)o(hed)h(from)f(the)h(send)h(bu\013er)e(will)j (b)q(e)e(con)o(v)o(erted)75 1086 y(to)g(the)h(represen)o(tation)f(for)g (reals)h(on)f(the)h(receiv)o(er)g(site)g(b)q(efore)g(they)g(are)f(stored)g (in)h(the)g(receiv)o(e)g(bu\013er.)75 1142 y(While)19 b(the)e(n)o(um)o(b)q (er)h(of)f(real)g(elemen)o(ts)h(fetc)o(hed)g(from)f(the)g(send)h(bu\013er)f (equal)i(the)e(n)o(um)o(b)q(er)h(of)f(real)75 1199 y(elemen)o(ts)d(stored)e (in)i(the)f(receiv)o(e)h(bu\013er,)f(the)g(n)o(um)o(b)q(er)g(of)g(b)o(ytes)g (stored)f(need)i(not)f(equal)h(the)f(n)o(um)o(b)q(er)75 1255 y(of)f(b)o(ytes)g(loaded:)19 b(e.g.)g(the)12 b(sender)h(ma)o(y)e(use)i(a)f (four)g(b)o(yte)g(represen)o(tation)h(and)f(the)g(receiv)o(er)i(an)e(eigh)o (t)75 1312 y(b)o(yte)17 b(represen)o(tation)h(for)f(reals.)27 b(If)18 b(the)g(send)g(or)f(receiv)o(e)h(bu\013er)g(do)f(not)h(con)o(tain)f (ten)h(consecutiv)o(e)75 1368 y(real)e(v)m(ariables,)g(then)f(the)g(program)g (is)g(erroneous,)g(and)g(its)h(b)q(eha)o(vior)f(is)h(unde\014ned.)166 1430 y(The)f(second)h(program)e(is)i(erroneous,)f(and)g(its)g(b)q(eha)o(vior) h(is)g(unde\014ned.)166 1492 y(The)e(third)g(program)e(is)i(correct.)19 b(The)13 b(exact)g(same)g(sequence)i(of)e(fort)o(y)f(b)o(ytes)h(that)g(w)o (ere)g(loaded)75 1548 y(from)18 b(the)h(send)h(bu\013er)f(will)i(b)q(e)e (stored)g(in)h(the)f(receiv)o(e)h(bu\013er,)f(ev)o(en)h(if)f(sender)h(and)f (receiv)o(er)h(run)75 1604 y(in)e(a)g(di\013eren)o(t)f(en)o(vironmen)o(t.)28 b(The)18 b(message)f(sen)o(t)g(has)h(exactly)f(the)h(same)f(length)i(\(in)f (b)o(ytes\))f(and)75 1661 y(the)f(same)f(binary)h(represen)o(tation)f(as)g (the)h(message)f(receiv)o(ed.)22 b(If)16 b(the)f(v)m(ariables)i(in)f(the)g (send)g(bu\013er)75 1717 y(are)f(of)h(di\013eren)o(t)g(t)o(yp)q(es)g(from)f (the)h(v)m(ariables)h(in)f(the)g(receiv)o(e)h(bu\013er,)e(or)h(they)g(are)f (of)h(the)f(same)h(t)o(yp)q(e)75 1774 y(but)h(di\013eren)o(t)g(data)g (represen)o(tations)f(are)h(used,)h(then)f(the)g(bits)h(stored)e(in)i(the)f (receiv)o(e)h(bu\013er)f(ma)o(y)75 1830 y(enco)q(de)f(v)m(alues)h(that)d(are) h(di\013eren)o(t)g(from)g(the)g(v)m(alues)h(they)g(enco)q(ded)g(in)g(the)g (send)f(bu\013er.)166 1892 y(Data)f(represen)o(tation)i(con)o(v)o(ersion)f (also)h(applies)h(to)e(the)g(en)o(v)o(elop)q(e)i(of)e(a)g(message:)20 b(source,)15 b(des-)75 1948 y(tination)h(and)f(tag)f(are)h(all)h(in)o(tegers) g(that)e(ma)o(y)h(need)h(to)e(b)q(e)i(con)o(v)o(erted.)166 2086 y Fq(Implemen)o(tati)o(on)11 b(note:)36 b Fu(The)13 b(curren)o(t)i (de\014nition)e(do)q(es)h(not)f(require)h(messages)g(to)f(carry)h(data)e(t)o (yp)q(e)75 2136 y(information.)j(A)e(message)g(can)g(b)q(e)g(comp)q(osed)g (and)g(sen)o(t)g(using)g(only)f(the)i(information)c(pro)o(vided)j(b)o(y)f (the)i(send)75 2186 y(call,)k(and)g(can)h(b)q(e)g(receiv)o(ed)h(and)e(stored) h(using)f(only)g(the)h(information)d(pro)o(vided)i(b)o(y)g(the)h(receiv)o(e)h (call.)30 b(If)75 2235 y(messages)16 b(are)f(sen)o(t)i(b)q(et)o(w)o(een)g (di\013eren)o(t)f(mac)o(hines)f(then)h(one)f(can)h(either)g(use)h(a)e(\\univ) o(ersal")g(data)g(enco)q(ding)75 2285 y(for)g(messages,)f(use)i(kno)o(wledge) f(of)f(the)i(receiv)o(er)g(en)o(vironmen)o(t)e(in)h(order)g(to)g(con)o(v)o (ert)h(data)e(at)h(the)h(sender,)g(or)75 2335 y(use)e(kno)o(wledge)f(of)f (the)i(sender)g(en)o(vironmen)o(t)e(in)h(order)g(to)g(con)o(v)o(ert)h(data)f (at)g(the)g(receiv)o(er.)19 b(In)13 b(either)h(case)g(the)75 2385 y(datat)o(yp)q(e)g(argumen)o(t)f(in)g(the)i(lo)q(cal)e(call)g(can)h(b)q (e)h(used)f(to)g(deriv)o(e)g(the)h(t)o(yp)q(es)g(of)e(the)h(v)n(alues)g (transferred.)166 2446 y(Additional)e(t)o(yp)q(e)j(information)c(can)j(b)q(e) g(carried)h(in)e(messages)h(in)f(order)i(to)f(pro)o(vide)f(b)q(etter)j(error) f(detec-)75 2503 y(tion.)166 2647 y FA(MPI)35 b(do)q(es)18 b(not)f(require)h(supp)q(ort)g(for)f(in)o(ter-language)h(comm)o(unication:)25 b(The)18 b(b)q(eha)o(vior)g(of)f(a)-1903 b Fk(\))75 2704 y FA(program)12 b(is)i(unde\014ned)h(if)e(messages)g(are)g(sen)o(t)g(b)o(y)g(a) g(C)g(pro)q(cess)g(and)h(receiv)o(ed)g(b)o(y)f(a)g(F)l(ortran)f(pro)q(cess,) 1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 17 17 bop 75 -100 a Fw(1.7.)34 b(COMMUNICA)l(TION)16 b(MODES)1048 b FA(17)75 49 y(or)15 b(vice-v)o(ersa.)166 106 y(MPI)f(implemen)o(tors)g(ma)o (y)f(pro)o(vide)h(supp)q(ort)g(for)f(in)o(ter-language)h(comm)o(unication)h (in)f(a)g(v)m(ariet)o(y)75 162 y(of)h(w)o(a)o(ys:)143 253 y Fl(\017)23 b FA(One)16 b(ma)o(y)f(in)o(v)o(ok)o(e)g(C)g(comm)o(unication)h (routines)g(from)f(a)g(F)l(ortran)f(program)h(or)g(F)l(ortran)f(com-)189 309 y(m)o(unication)g(routines)g(from)f(a)h(C)f(program,)g(on)g(systems)h (that)e(supp)q(ort)i(in)o(v)o(ok)m(ation)h(of)e(foreign)189 366 y(language)i(routines.)143 458 y Fl(\017)23 b FA(One)e(ma)o(y)e(allo)o(w) i(the)f(use)h(of)f(\\C)g(MPI)g(t)o(yp)q(es",)h(suc)o(h)f(as)g Fo(MPI)p 1332 458 13 2 v 15 w(INT,)e(MPI)p 1521 458 V 14 w(CHAR)p FA(,)h(etc.)36 b(from)189 515 y(F)l(ortran)14 b(programs,)f(and)j(vice-v)o (ersa.)143 607 y Fl(\017)23 b FA(One)16 b(ma)o(y)e(\014x)h(the)h(corresp)q (ondance)g(b)q(et)o(w)o(een)f(C)g(datat)o(yp)q(es)g(and)g(F)l(ortran)f(datat) o(yp)q(es.)189 709 y Fr(R)n(ationale.)46 b FA(MPI)17 b(do)q(es)f(not)g (handle)i(in)o(ter-language)f(comm)o(unication)g(b)q(ecause)g(there)g(seem) 189 766 y(to)d(b)q(e)i(no)f(agreed)g(standards)g(for)g(in)o(ter-language)g(t) o(yp)q(e)h(con)o(v)o(ersions.)k(\()p Fr(End)15 b(of)i(r)n(ationale.)p FA(\))75 908 y FB(1.7)59 b(Communication)17 b(Mo)r(des)75 1010 y FA(The)c(basic)h(send)f(op)q(eration)g(describ)q(ed)i(in)f(section)f(1.2)f (used)h(the)g Fz(standard)g FA(comm)o(unication)h(mo)q(de.)75 1066 y(In)g(suc)o(h)g(a)f(comm)o(unication)h(mo)q(de,)g(a)f(send)h(op)q (eration)g(can)f(b)q(e)h(started)f(whether)h(or)f(not)g(a)g(matc)o(hing)75 1123 y(receiv)o(e)18 b(w)o(as)e(p)q(osted.)24 b(The)17 b(completion)h(of)e (the)h(send)h(op)q(eration)e(indicates)j(that)d(the)g(message)h(and)75 1179 y(its)f(en)o(v)o(elop)q(e)h(ha)o(v)o(e)e(b)q(een)i(safely)f(stored)f(a)o (w)o(a)o(y)f(and)i(that)f(the)h(sender)g(is)g(free)g(to)f(access)h(and)f(mo)q (dify)75 1236 y(the)j(sender)h(bu\013er.)30 b(Th)o(us,)18 b(the)h(op)q (eration)f(is)h Fz(lo)q(cally-blo)r(c)o(king)p FA(:)29 b(it)19 b(do)q(es)f(not)g(return)g(un)o(til)i(the)75 1292 y(send)c(op)q(eration)f (has)g(completed)h Fz(lo)q(cally)p FA(,)i(on)d(the)g(sender)h(side.)166 1349 y(The)f(completion)i(of)e(a)g(send)h(op)q(eration)f(giv)o(es)h(no)f (indication)i(that)e(the)g(message)g(w)o(as)g(receiv)o(ed)75 1405 y(on)i(the)g(receiv)o(er)h(side.)26 b(A)17 b(blo)q(c)o(king)i(send)e(ma) o(y)g(b)q(e)g(implemen)o(ted)i(so)e(that)f(it)h(returns)g(only)h(after)e(a)75 1462 y(matc)o(hing)j(receiv)o(e)h(has)f(b)q(een)h(executed)g(on)f(the)h (receiv)o(er)f(side.)33 b(This)20 b(a)o(v)o(oids)e(the)i(need)g(to)e (bu\013er)75 1518 y(message)e(data)f(out)h(of)f(sender)i(or)e(receiv)o(er)i (memory)l(.)22 b(In)17 b(this)f(case)g(the)g(send)h(op)q(eration)f(completes) 75 1574 y(only)e(after)e(the)i(matc)o(hing)f(receiv)o(e)h(has)g(started)e (executing.)21 b(On)13 b(the)h(other)f(hand,)g(it)h(is)g(also)f(p)q(ossible) 75 1631 y(for)j(MPI)g(to)f(bu\013er)h(messages,)g(so)g(as)g(to)f(allo)o(w)i (the)f(sender)h(to)e(pro)q(ceed)i(ahead)g(of)e(the)i(receiv)o(er.)23 b(In)75 1687 y(suc)o(h)15 b(a)g(case)g(the)g(send)g(op)q(eration)g(ma)o(y)f (complete)i(successfully)h(b)q(efore)e(the)g(message)f(w)o(as)g(receiv)o(ed.) 166 1744 y(There)h(are)g(t)o(w)o(o)f(additional)j(comm)o(unication)f(mo)q (des:)166 1800 y(A)e(send)h(that)e(uses)h(the)g Fz(ready)g FA(comm)o(unication)g(mo)q(de)h(ma)o(y)e(b)q(e)i(started)e(only)h(if)h(the)f (matc)o(hing)75 1857 y(receiv)o(e)g(is)f(already)g(p)q(osted;)g(otherwise)g (the)g(op)q(eration)g(is)g(erroneous)f(and)h(its)g(outcome)f(is)h (unde\014ned.)75 1913 y(In)19 b(some)f(systems,)h(this)f(allo)o(ws)h(the)f (remo)o(v)m(al)h(of)f(a)g(hand-shak)o(e)h(op)q(eration)g(that)e(is)i (otherwise)g(re-)75 1970 y(quired,)g(and)f(results)g(in)g(impro)o(v)o(ed)g(p) q(erformance.)27 b(The)18 b(completion)h(of)e(the)h(send)g(op)q(eration)g(do) q(es)75 2026 y(not)e(dep)q(end)i(on)e(the)g(status)f(of)h(a)g(matc)o(hing)g (receiv)o(e,)h(and)f(merely)h(indicates)h(that)d(the)h(send)h(bu\013er)75 2083 y(can)i(b)q(e)g(reused.)30 b(A)19 b(send)g(op)q(eration)g(that)f(uses)g (the)h(ready)f(mo)q(de)h(has)g(the)f(same)g(seman)o(tics)h(as)f(a)75 2139 y(standard)h(send)h(op)q(eration;)h(it)f(is)g(merely)g(that)f(it)g(can)h (b)q(e)g(used)g(only)g(in)g(some)f(of)g(the)h(situations)75 2195 y(where)15 b(a)g(regular)h(send)f(can)h(b)q(e)g(used.)166 2252 y(A)i(send)g(that)f(uses)g(the)h Fz(sync)o(hronous)e FA(mo)q(de)i(can)g (b)q(e)g(started)f(whether)g(or)g(not)h(a)f(matc)o(hing)75 2308 y(receiv)o(e)12 b(w)o(as)e(p)q(osted.)19 b(Ho)o(w)o(ev)o(er,)10 b(the)h(send)h(will)g(complete)g(successfully)h(only)f(if)f(a)g(matc)o(hing)g (receiv)o(e)g(is)75 2365 y(p)q(osted,)h(and)g(the)g(receiv)o(e)g(op)q (eration)g(has)f(started)g(to)g(receiv)o(e)i(the)e(message)g(sen)o(t)h(b)o(y) f(the)h(sync)o(hronous)75 2421 y(send.)29 b(\(I.e.,)19 b(the)f(receiv)o(e)h (has)f(b)q(een)h(p)q(osted,)g(and)g(the)f(incoming)h(message)f(has)g(b)q(een) i(matc)o(hed)e(to)75 2478 y(this)g(p)q(osted)g(receiv)o(e.\))28 b(Th)o(us,)18 b(the)g(completion)g(of)g(a)f(sync)o(hronous)h(send)g(not)f (only)i(indicates)g(that)75 2534 y(the)g(send)h(bu\013er)f(can)g(b)q(e)h (reused,)g(but)f(also)g(indicates)i(that)d(the)i(receiv)o(er)f(has)g(reac)o (hed)h(a)f(certain)75 2591 y(p)q(oin)o(t)e(in)h(its)f(execution,)h(namely)g (that)e(it)h(has)g(started)f(executing)i(the)f(matc)o(hing)g(receiv)o(e.)26 b(If)17 b(b)q(oth)75 2647 y(sends)f(and)h(receiv)o(es)f(are)g(blo)q(c)o(king) h(op)q(erations)f(then)h(the)f(use)g(of)f(the)h(sync)o(hronous)g(mo)q(de)g (pro)o(vides)75 2704 y(sync)o(hronous)h(comm)o(unication)g(seman)o(tics:)23 b(a)16 b(comm)o(unication)h(do)q(es)g(not)g(complete)g(at)f(either)h(end)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 18 18 bop 75 -100 a FA(18)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y FA(b)q(efore)c(b)q(oth)g(pro)q (cesses)g(\\attend")f(to)g(the)h(comm)o(unication;)h(the)f(completion)h(of)e (a)h(sync)o(hronous)f(send)75 106 y(is)16 b(a)f Fz(global)i FA(ev)o(en)o(t,)d(and)h(the)h(op)q(eration)f(is)h Fz(globally-blo)q(cking)p FA(.)166 162 y(Tw)o(o)22 b(additional)j(send)e(functions)h(are)f(pro)o(vided) h(for)f(the)g(t)o(w)o(o)f(additional)i(comm)o(unication)75 219 y(mo)q(des.)38 b(The)22 b(comm)o(unication)g(mo)q(de)f(is)h(indicated)h (b)o(y)e(a)g(one)h(letter)f(pre\014x:)32 b Fv(R)22 b FA(for)f(ready)g(and)75 275 y Fv(S)15 b FA(for)g(sync)o(hronous.)166 332 y(Send)h(in)g(ready)f(mo)q (de)75 435 y Fv(MPI)p 160 435 14 2 v 16 w(RSEND)h(\(buf,)f(count,)h(datat)o (yp)q(e,)h(dest,)f(tag,)f(comm)m(\))117 512 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 586 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h (send)h(bu\013er)f(\(in)o(teger\))117 660 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 734 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h (\(in)o(teger\))117 808 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 881 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))75 1006 y Ft(int)23 b(MPI)p 245 1006 15 2 v 17 w(Rsend\(void*)g(buf,)g(int)g(count,)g(MPI)p 1002 1006 V 17 w(Datatype)g(datatype,)g(int)g(dest,)393 1062 y(int)h(tag,)f(MPI)p 683 1062 V 17 w(Comm)g(comm\))75 1149 y(MPI)p 150 1149 V 17 w(RSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)h(IERROR\))170 1205 y()f(BUF\(*\))170 1262 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (TAG,)h(COMM,)f(IERROR)166 1348 y FA(Send)16 b(in)g(sync)o(hronous)f(mo)q(de) 75 1452 y Fv(MPI)p 160 1452 14 2 v 16 w(SSEND)h(\(buf,)f(count,)h(datat)o(yp) q(e,)g(dest,)h(tag,)e(comm)m(\))117 1529 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 1603 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in) h(send)h(bu\013er)f(\(in)o(teger\))117 1677 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 1751 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h (\(in)o(teger\))117 1824 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 1898 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator) 11 b(\(handle\))75 2023 y Ft(int)23 b(MPI)p 245 2023 15 2 v 17 w(Ssend\(void*)g(buf,)g(int)g(count,)g(MPI)p 1002 2023 V 17 w(Datatype)g(datatype,)g(int)g(dest,)393 2079 y(int)h(tag,)f(MPI)p 683 2079 V 17 w(Comm)g(comm\))75 2165 y(MPI)p 150 2165 V 17 w(SSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)h(IERROR\))170 2222 y()f(BUF\(*\))170 2278 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (TAG,)h(COMM,)f(IERROR)166 2365 y FA(There)16 b(is)g(only)g(one)g(receiv)o(e) h(mo)q(de,)e(whic)o(h)i(can)f(matc)o(h)f(an)o(y)g(of)h(the)f(send)i(mo)q (des.)k(The)16 b(receiv)o(e)75 2421 y(op)q(eration)g(describ)q(ed)h(in)g(the) e(last)h(section)g(is)g Fz(blo)q(c)o(king)p FA(:)22 b(it)16 b(returns)f(only)h(after)f(the)h(receiv)o(e)g(bu\013er)75 2478 y(con)o(tains)h(the)h(newly)g(receiv)o(ed)g(message.)26 b(It)17 b(is)h Fz(async)o(hronous)p FA(:)23 b(the)17 b(completion)i(of)d(a)h(receiv)o (e)h(is)75 2534 y(a)e(lo)q(cal)i(op)q(eration,)f(and)f(a)g(receiv)o(e)i(can)f (complete)g(b)q(efore)g(the)f(matc)o(hing)h(send)g(has)f(completed)i(\(of)75 2591 y(course,)d(it)g(can)h(complete)g(only)f(after)g(the)g(matc)o(hing)g (send)h(has)f(started\).)166 2647 y(Comm)o(unication)e(imp)q(oses)g(an)f (order)h(on)f(the)h(ev)o(en)o(ts)f(o)q(ccurring)h(at)f(the)h(comm)o (unicating)g(no)q(des.)75 2704 y(It)i(is)g(alw)o(a)o(ys)e(the)i(case)g(that)e (the)i(completion)h(of)e(a)g(receiv)o(e)i(o)q(ccurs)e(after)g(the)h(start)e (of)h(the)h(matc)o(hing)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 19 19 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(19)75 49 y(send.)29 b(If)19 b(the)f(sync)o(hronous)g(send)h(mo)q(de)f (is)h(used,)g(then)g(it)f(is)h(also)f(the)g(case)g(that)g(the)g(completion)75 106 y(of)i(a)h(send)g(o)q(ccurs)g(after)f(the)g(start)g(of)g(the)h(matc)o (hing)f(receiv)o(e.)38 b(Of)20 b(course,)i(on)f(eac)o(h)f(pro)q(cess,)i(a)75 162 y(comm)o(unication)14 b(completes)h(after)d(it)i(is)g(started.)19 b(No)13 b(other)h(order)f(is)h(imp)q(osed)h(b)o(y)e(MPI.)g(E.g.,)g(if)h(the) 75 219 y(standard)h(send)g(mo)q(de)h(is)f(used,)h(the)f(send)h(op)q(eration)f (ma)o(y)f(complete)i(b)q(efore)g(the)f(matc)o(hing)g(receiv)o(e)75 275 y(w)o(as)f(started.)166 333 y(In)h(a)f(m)o(ulti-threaded)i(implemen)o (tation)g(of)e(MPI,)h(the)f(system)g(ma)o(y)g(desc)o(hedule)j(a)d(thread)h (that)75 389 y(is)21 b(blo)q(c)o(k)o(ed)g(on)g(a)f(send)h(or)e(receiv)o(e)j (op)q(eration,)f(and)g(sc)o(hedule)h(another)e(thread)g(for)g(execution)h(in) 75 446 y(the)d(same)g(address)g(space.)28 b(In)19 b(suc)o(h)f(a)f(case)h(it)h (the)f(user)g(resp)q(onsibilit)o(y)i(not)e(to)f(access)h(or)f(mo)q(dify)75 502 y(a)f(comm)o(unication)h(bu\013er)f(un)o(til)i(the)e(comm)o(unication)h (completes;)g(otherwise)g(the)f(outcome)g(of)g(the)75 559 y(computation)f(is) h(unde\014ned.)189 671 y Fr(R)n(ationale.)61 b FA(W)l(e)19 b(prohibit)h(read)f(accesses)g(to)f(a)g(send)i(bu\013er)e(while)j(it)e(is)g (b)q(een)h(used,)g(ev)o(en)189 727 y(though)e(the)g(send)g(op)q(eration)h(is) f(not)g(supp)q(osed)h(to)e(alter)h(the)g(con)o(ten)o(t)g(of)f(this)i (bu\013er.)28 b(This)189 784 y(ma)o(y)17 b(seem)h(more)f(stringen)o(t)g(than) h(necessary)l(,)g(but)g(the)g(additional)h(restriction)g(causes)f(little)189 840 y(loss)g(of)g(functionalit)o(y)i(and)e(allo)o(ws)h(b)q(etter)f(p)q (erformance)h(on)f(some)g(systems)g({)h(consider)g(the)189 897 y(case)d(where)g(data)g(transfer)f(is)i(done)f(b)o(y)h(a)e(DMA)h(engine)h (whic)o(h)g(is)g(not)f(cac)o(he-coheren)o(t)h(with)189 953 y(the)e(main)h(pro)q(cessor.)j(\()p Fr(End)d(of)g(r)n(ationale.)p FA(\))166 1141 y Fq(Implemen)o(tati)o(on)10 b(note:)34 b Fu(A)13 b(ready)f(send)h(can)f(b)q(e)h(implemen)o(ted)d(as)i(a)g(standard)h(send;)g (in)e(suc)o(h)i(a)f(case)75 1191 y(there)j(will)e(b)q(e)h(no)g(p)q (erformance)g(adv)n(an)o(tage)f(\(or)h(disadv)n(an)o(tage\))f(for)g(the)i (use)f(of)g(ready)g(send.)166 1242 y(A)f(standard)h(send)h(can)e(b)q(e)h (implemen)o(ted)e(as)h(a)g(sync)o(hronous)i(send.)j(In)c(suc)o(h)g(a)f(case,) h(no)f(data)h(bu\013ering)75 1292 y(is)g(needed.)166 1343 y(A)g(p)q(ossible)g (comm)o(unication)c(proto)q(col)k(for)g(the)g(v)n(arious)f(comm)o(unication)e (mo)q(des)i(is)h(outlined)f(b)q(elo)o(w:)166 1394 y Fo(ready)i(send)p Fu(:)k(The)14 b(message)g(is)g(sen)o(t)h(as)e(so)q(on)h(as)g(p)q(ossible.)166 1445 y Fo(synchronous)f(send:)18 b Fu(The)12 b(sender)h(sends)f(a)f (request-to-send)i(message.)k(The)12 b(receiv)o(er)h(stores)f(this)g (request.)75 1495 y(When)j(a)f(matc)o(hing)f(receiv)o(e)j(is)f(p)q(osted,)g (the)h(receiv)o(er)g(sends)g(bac)o(k)f(a)f(p)q(ermission-to-send)h(message,)f (and)g(the)75 1545 y(sender)h(no)o(w)f(sends)h(the)g(message.)166 1596 y Fo(standa)o(rd)k(send:)27 b Fu(First)18 b(proto)q(col)g(ma)o(y)e(b)q (e)i(used)h(for)e(short)i(messages,)f(and)g(second)h(proto)q(col)e(for)h (long)75 1645 y(messages.)166 1696 y(Additional)d(con)o(trol)h(messages)h (migh)o(t)e(b)q(e)i(needed)h(for)e(\015o)o(w)g(con)o(trol)h(and)f(error)i (reco)o(v)o(ery)m(.)27 b(Of)16 b(course,)75 1746 y(there)f(are)f(man)o(y)f (other)h(p)q(ossible)g(proto)q(cols.)75 1979 y FB(1.8)59 b(Nonblo)r(cking)20 b(communication)75 2083 y FA(One)e(can)g(impro)o(v)o(e)f(p)q(erformance)g(on) h(man)o(y)e(systems)h(b)o(y)h(o)o(v)o(erlapping)f(comm)o(unication)i(and)e (com-)75 2139 y(putation.)35 b(This)21 b(is)g(esp)q(ecially)i(true)d(on)g (systems)g(where)g(comm)o(unication)h(can)g(b)q(e)g(executed)g(au-)75 2195 y(tonomously)12 b(b)o(y)h(an)g(in)o(telligen)o(t)h(comm)o(unication)g (con)o(troller.)19 b(Ligh)o(t-w)o(eigh)o(t)13 b(threads)f(are)h(one)g(mec)o (h-)75 2252 y(anism)21 b(for)g(ac)o(hieving)h(suc)o(h)g(o)o(v)o(erlap.)37 b(An)21 b(alternativ)o(e)h(mec)o(hanism)g(that)e(often)h(leads)g(to)g(b)q (etter)75 2308 y(p)q(erformance)e(is)g(to)f(use)h Fz(non)o(blo)q(c)o(king)j (comm)o(unication)p FA(.)32 b(A)19 b(non)o(blo)q(c)o(king)h(send)f(call)h (initiates)75 2365 y(the)15 b(send)h(op)q(eration,)f(but)h(do)q(es)f(not)g (complete)h(it.)k(The)15 b(send)h(call)h(will)f(return)g(b)q(efore)f(the)g (message)75 2421 y(w)o(as)d(copied)h(out)f(of)g(the)g(send)h(bu\013er.)19 b(A)13 b(separate)e(call)j(is)f(needed)g(to)f(complete)h(the)g(comm)o (unication,)75 2478 y(i.e.)20 b(to)13 b(v)o(erify)i(that)e(the)h(data)g(has)g (b)q(een)h(copied)g(out)f(of)g(the)g(send)h(bu\013er.)k(With)c(suitable)g (hardw)o(are,)75 2534 y(the)k(transfer)f(of)h(data)f(out)h(of)g(the)g(sender) g(memory)g(ma)o(y)f(pro)q(ceed)i(concurren)o(tly)g(with)f(computa-)75 2591 y(tions)g(done)g(at)f(the)g(sender)h(after)f(the)h(send)g(w)o(as)f (initiated)i(and)e(b)q(efore)h(it)g(completed.)31 b(Similarly)l(,)75 2647 y(a)19 b(non)o(blo)q(c)o(king)h(receiv)o(e)g(initiates)h(the)e(receiv)o (e)h(op)q(eration,)g(but)f(do)q(es)g(not)g(complete)h(it.)32 b(The)19 b(call)75 2704 y(will)f(return)f(b)q(efore)g(a)f(message)g(is)i (stored)e(in)o(to)g(the)h(receiv)o(e)h(bu\013er.)24 b(A)17 b(separate)f(call)h(is)h(needed)g(to)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 20 20 bop 75 -100 a FA(20)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y FA(complete)h(the)f(receiv)o(e)i(op)q (eration)e(and)g(v)o(erify)h(that)e(the)i(data)e(has)i(b)q(een)g(receiv)o(ed) g(in)o(to)g(the)f(receiv)o(e)75 106 y(bu\013er.)k(With)14 b(suitable)h(hardw) o(are,)f(the)g(transfer)f(of)h(data)g(in)o(to)g(the)g(receiv)o(er)h(memory)f (ma)o(y)f(pro)q(ceed)75 162 y(concurren)o(tly)f(with)g(computations)f(done)g (after)g(the)g(receiv)o(e)h(w)o(as)f(initiated)i(and)e(b)q(efore)h(it)f (completed.)75 219 y(The)17 b(use)h(of)f(non)o(blo)q(c)o(king)h(receiv)o(es)g (ma)o(y)f(also)g(a)o(v)o(oid)g(system)f(bu\013ering)i(and)f(memory)g(to)g (memory)75 275 y(cop)o(ying,)e(as)g(information)g(is)h(pro)o(vided)g(early)g (on)f(the)g(lo)q(cation)h(of)f(the)g(receiv)o(e)h(bu\013er.)166 366 y(Non)o(blo)q(c)o(king)d(sends)f(can)g(use)h(the)f(same)f(three)h(mo)q (des)g(as)g(blo)q(c)o(king)h(sends:)19 b Fv(standa)o(rd)p FA(,)13 b Fv(ready)f FA(and)75 422 y Fv(synchronous)p FA(.)26 b(These)16 b(carry)g(the)h(same)f(meaning.)24 b(The)17 b(initiation)h(and)f(completion)g (of)f(a)g(standard)75 479 y(send)k(do)f(not)g(dep)q(end)i(on)e(the)h(status)e (of)h(a)g(matc)o(hing)g(receiv)o(e.)33 b(A)20 b(ready)f(send)h(can)f(b)q(e)h (initiated)75 535 y(only)f(if)h(a)e(matc)o(hing)h(receiv)o(e)h(has)f(already) g(b)q(een)h(initiated,)h(otherwise)e(the)g(call)h(is)g(erroneous;)g(its)75 591 y(completion)d(do)q(es)g(not)f(dep)q(end)i(on)e(the)g(status)g(of)f(a)h (matc)o(hing)h(receiv)o(e.)23 b(A)17 b(sync)o(hronous)f(send)h(can)75 648 y(b)q(e)j(initiated)h(b)q(efore)f(a)f(matc)o(hing)h(receiv)o(e)g(has)f(b) q(een)i(initiated,)h(but)d(will)i(complete)g(only)f(after)e(a)75 704 y(matc)o(hing)13 b(receiv)o(e)g(has)g(b)q(een)h(initiated,)g(and)f(has)g (started)f(receiving)i(the)f(message)f(generated)h(b)o(y)g(the)75 761 y(send)j(op)q(eration.)166 851 y(In)f(all)h(cases)e(the)h(op)q(eration)g (that)f(initiates)i(the)f(comm)o(unication)g(\(send)g(or)f(receiv)o(e\))i(is) f(a)f(lo)q(cal,)75 908 y(non)o(blo)q(c)o(king)22 b(call.)39 b(If)21 b(the)g(call)h(causes)g(some)e(system)h(resource)g(to)f(b)q(e)i (exhausted,)h(then)e(it)g(will)75 964 y(fail)f(and)f(return)f(an)h(error)f (co)q(de.)31 b(Go)q(o)q(d)19 b(qualit)o(y)g(implemen)o(tations)h(of)f(MPI)g (should)g(ensure)h(that)75 1021 y(this)d(happ)q(ens)h(only)f(in)h (\\pathological")f(cases.)25 b(I.e.,)16 b(an)h(MPI)g(implemen)o(tation)h (should)g(b)q(e)f(able)h(to)75 1077 y(supp)q(ort)d(a)g(large)g(n)o(um)o(b)q (er)h(of)f(p)q(ending)i(non)o(blo)q(c)o(king)f(op)q(erations.)166 1168 y(Non)o(blo)q(c)o(king)g(sends)g(can)f(b)q(e)h(matc)o(hed)f(with)h(blo)q (c)o(king)g(receiv)o(es,)g(and)f(vice-v)o(ersa.)75 1484 y Fs(1.8.1)49 b(Comm)o(unication)14 b(Objects)75 1636 y FA(Non)o(blo)q(c)o(king)k(comm)o (unications)g(use)f(opaque)g(request)g(ob)s(jects)f(to)h(iden)o(tify)h(comm)o (unication)g(op)q(er-)75 1693 y(ations)f(and)g(matc)o(h)f(the)i(op)q(eration) f(that)f(initiates)i(the)f(comm)o(unication)h(with)f(the)g(op)q(eration)h (that)75 1749 y(terminates)f(it.)27 b(These)17 b(are)g(system)g(ob)s(jects)g (that)f(are)h(accessed)h(via)g(a)f(handle.)27 b(A)17 b(request)h(ob)s(ject)75 1806 y(iden)o(ti\014es)j(v)m(arious)f(prop)q(erties)g(of)f(a)g(comm)o (unication)h(op)q(eration,)h(suc)o(h)e(as)g(the)h(\(send)f(or)g(receiv)o(e\)) 75 1862 y(bu\013er)c(that)g(is)h(asso)q(ciated)f(with)h(it,)f(its)h(con)o (text,)e(the)i(tag)e(and)i(destination)g(argumen)o(ts)f(to)f(b)q(e)i(used)75 1919 y(for)k(a)f(send,)j(or)e(the)g(tag)f(and)i(source)f(argumen)o(ts)f(to)h (b)q(e)h(used)f(for)g(a)g(receiv)o(e.)36 b(In)20 b(addition,)j(this)75 1975 y(ob)s(ject)14 b(stores)g(information)g(ab)q(out)h(the)f(status)g(of)g (the)g(p)q(ending)j(comm)o(unication)e(op)q(eration)g(that)e(is)75 2032 y(p)q(erformed)i(with)h(this)g(ob)s(ject.)75 2348 y Fs(1.8.2)49 b(Comm)o(unication)14 b(initiation)75 2500 y FA(W)l(e)21 b(use)g(the)f(same)h (naming)g(con)o(v)o(en)o(tions)f(as)h(for)f(blo)q(c)o(king)h(comm)o (unication:)32 b(a)20 b(pre\014x)h(of)f Fv(R)h FA(\()p Fv(S)p FA(\))75 2557 y(is)f(used)f(for)f Fv(READ)o(Y)i FA(\()p Fv(SYNCHRONOUS)p FA(\))h(mo)q(de.)32 b(In)20 b(addition)g(a)e(pre\014x)i(of)e Fv(I)h FA(\(for)f Fv(IMMEDIA)l(TE)p FA(\))75 2613 y(indicates)f(that)d(the)h (call)i(is)e(non)o(blo)q(c)o(king.)166 2704 y(Initiate)h(a)f(standard)g(mo)q (de)g(non)o(blo)q(c)o(king)i(send.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 21 21 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(21)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(ISEND\(buf,)15 b(count,)h(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m(,)d(request\))117 143 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 253 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 362 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 471 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 580 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 689 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 798 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 940 y Ft(int)23 b(MPI)p 245 940 15 2 v 17 w(Isend\(void*)g (buf,)g(int)g(count,)g(MPI)p 1002 940 V 17 w(Datatype)g(datatype,)g(int)g (dest,)393 996 y(int)h(tag,)f(MPI)p 683 996 V 17 w(Comm)g(comm,)g(MPI)p 1034 996 V 17 w(Comm)p 1147 996 V 17 w(request)g(*request\))75 1100 y(MPI)p 150 1100 V 17 w(ISEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,) g(COMM,)h(REQUEST,)e(IERROR\))170 1156 y()h(BUF\(*\))170 1213 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f (IERROR)166 1316 y FA(Initiate)16 b(a)f(ready)g(mo)q(de)h(non)o(blo)q(c)o (king)g(send.)75 1437 y Fv(MPI)p 160 1437 14 2 v 16 w(IRSEND\(buf,)g(count,)g (datat)o(yp)q(e,)g(dest,)g(tag,)f(comm)m(,)d(request\))117 1531 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g (bu\013er)h(\(c)o(hoice\))117 1640 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o (teger\))117 1749 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e(\(handle\))117 1858 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 1968 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 2077 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 2186 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 2327 y Ft(int)23 b(MPI)p 245 2327 15 2 v 17 w(Irsend\(void*)f(buf,)i(int)f(count,)g(MPI)p 1026 2327 V 17 w(Datatype)g(datatype,)f(int)i(dest,)393 2384 y(int)g(tag,)f(MPI)p 683 2384 V 17 w(Comm)g(comm,)g(MPI)p 1034 2384 V 17 w(Comm)p 1147 2384 V 17 w(request)g(*request\))75 2487 y(MPI)p 150 2487 V 17 w(IRSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)g(REQUEST,)g (IERROR\))170 2544 y()g(BUF\(*\))170 2600 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 2704 y FA(Initiate)16 b(a)f(sync)o(hronous)g(mo)q(de)h(non)o(blo)q(c)o(king)g (send.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 22 22 bop 75 -100 a FA(22)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(ISSEND\(buf,)f(count,)i(datat)o(yp)q(e,)f(dest,)g(tag,)f(comm)m(,)d (request\))117 131 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 216 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h (bu\013er)f(\(in)o(teger\))117 300 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(send)h(bu\013er)g(elemen)o(t)e (\(handle\))117 385 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h (\(in)o(teger\))117 470 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 554 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 639 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))75 768 y Ft(int)23 b(MPI)p 245 768 15 2 v 17 w(Issend\(void*)f(buf,)i(int)f(count,)g(MPI)p 1026 768 V 17 w(Datatype)g(datatype,)f(int)i(dest,)393 825 y(int)g(tag,)f(MPI)p 683 825 V 17 w(Comm)g(comm,)g(MPI)p 1034 825 V 17 w(Comm)p 1147 825 V 17 w(request)g(*request\))75 916 y(MPI)p 150 916 V 17 w(ISSEND\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)g(REQUEST,)g (IERROR\))170 972 y()g(BUF\(*\))170 1029 y(INTEGER)g(COUNT,)g (DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 1120 y FA(Initiate)16 b(a)f(non)o(blo)q(c)o(king)i(receiv)o(e.)75 1228 y Fv(MPI)p 160 1228 14 2 v 16 w(IRECV)e(\(buf,)h(count,)g(datat)o(yp)q (e,)g(source,)g(tag,)e(comm)n(,)d(request\))117 1310 y Fu(OUT)108 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o (hoice\))117 1395 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g(\(in)o(teger\))117 1479 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(receiv)o(e)h(bu\013er)g(elemen)o(t)e(\(handle\))117 1564 y(IN)155 b Fv(source)468 b Fu(rank)14 b(of)f(source)j(\(in)o(teger\))117 1649 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 1733 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 1818 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 1947 y Ft(int)23 b(MPI)p 245 1947 15 2 v 17 w(Irecv\(void*)g(buf,)g(int)g(count,)g(MPI)p 1002 1947 V 17 w(Datatype)g(datatype,)g(int)g(source,)393 2004 y(int)h(tag,)f(MPI)p 683 2004 V 17 w(Comm)g(comm,)g(MPI)p 1034 2004 V 17 w(Comm)p 1147 2004 V 17 w(request)g(*request\))75 2095 y(MPI)p 150 2095 V 17 w(IRECV\(BUF,)f(COUNT,)h(DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(REQUEST,)g (IERROR\))170 2151 y()g(BUF\(*\))170 2208 y(INTEGER)g(COUNT,)g (DATATYPE,)g(SOURCE,)g(TAG,)g(COMM,)g(REQUEST,)g(IERROR)166 2299 y FA(These)c(calls)g(allo)q(cate)h(a)e(comm)o(unication)h(request)f(and) h(asso)q(ciate)g(it)f(with)h(the)g(handle.)31 b(The)75 2355 y(request)11 b(can)f(b)q(e)i(used)f(later)g(to)f(query)g(the)h(status)f(of)g (the)h(comm)o(unication)g(or)f(w)o(ait)g(for)g(its)h(completion.)166 2417 y(A)17 b(non)o(blo)q(c)o(king)i(send)f(call)g(indicates)h(that)e(the)g (system)g(ma)o(y)g(start)f(cop)o(ying)h(data)g(out)g(of)g(the)75 2473 y(send)g(bu\013er.)24 b(The)17 b(sender)g(should)g(not)g(access)f(an)o (y)g(part)g(of)g(the)h(send)g(bu\013er)g(after)e(a)i(non)o(blo)q(c)o(king)75 2529 y(send)f(op)q(eration)f(is)h(called,)g(un)o(til)h(the)e(send)h (completes.)166 2591 y(A)d(non)o(blo)q(c)o(king)h(receiv)o(e)g(call)g (indicates)g(that)e(the)h(system)f(ma)o(y)g(start)g(writing)h(data)f(in)o(to) h(the)g(re-)75 2647 y(ceiv)o(e)f(bu\013er.)19 b(The)11 b(receiv)o(er)h (should)h(not)e(access)g(an)o(y)g(part)g(of)g(the)g(receiv)o(e)i(bu\013er)e (after)g(a)g(non)o(blo)q(c)o(king)75 2704 y(receiv)o(e)16 b(op)q(eration)g (is)f(called,)i(un)o(til)f(the)f(receiv)o(e)h(completes.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 23 23 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(23)75 49 y Fs(1.8.3)49 b(Comm)o(unication)14 b(Completion)75 161 y FA(The)j(functions)g Fv(MPI)p 452 161 14 2 v 16 w(W)l(AIT)f FA(and)g Fv(MPI)p 768 161 V 16 w(TEST)h FA(are)f(used)h(to)f(complete)h(a)f (non)o(blo)q(c)o(king)h(comm)o(unica-)75 217 y(tion.)i(The)13 b(completion)h(of)e(a)g(send)h(op)q(eration)g(indicates)h(that)e(the)g (sender)h(is)g(no)o(w)f(free)h(to)f(up)q(date)h(the)75 274 y(lo)q(cations)k(in)f(the)h(send)f(bu\013er)g(\(the)g(send)g(op)q(eration)g (itself)h(lea)o(v)o(es)f(the)g(con)o(ten)o(t)g(of)f(the)h(send)h(bu\013er)75 330 y(unc)o(hanged\).)25 b(It)16 b(do)q(es)h(not)g(indicate)h(that)e(the)h (message)f(has)g(b)q(een)i(receiv)o(ed;)g(rather,)e(it)h(ma)o(y)f(ha)o(v)o(e) 75 386 y(b)q(een)g(bu\013ered)g(b)o(y)f(the)h(comm)o(unication)g(subsystem.)k (Ho)o(w)o(ev)o(er,)14 b(if)i(a)f Fv(synchronous)i FA(mo)q(de)f(send)g(w)o(as) 75 443 y(used,)f(the)g(completion)h(of)f(the)f(send)i(op)q(eration)f (indicates)h(that)e(a)h(matc)o(hing)g(receiv)o(e)h(w)o(as)e(initiated,)75 499 y(and)h(that)g(the)g(message)g(will)i(ev)o(en)o(tually)f(b)q(e)g(receiv)o (ed)g(b)o(y)f(this)h(matc)o(hing)f(receiv)o(e.)166 569 y(The)20 b(completion)h(of)e(a)g(receiv)o(e)i(op)q(eration)f(indicates)h(that)e(the)h (receiv)o(e)g(bu\013er)g(con)o(tains)g(the)75 625 y(receiv)o(ed)f(message,)f (and)g(that)f(the)h(status)f(ob)s(ject)h(is)g(set;)h(the)f(receiv)o(er)h(is)f (no)o(w)f(free)h(to)g(access)g(the)75 682 y(receiv)o(e)g(bu\013er.)27 b(It)17 b(do)q(es)h(not)f(indicate)i(that)d(the)i(matc)o(hing)f(send)h(op)q (eration)g(has)f(completed)h(\(but)75 738 y(indicates,)e(of)f(course,)g(that) f(the)i(send)f(w)o(as)g(initiated\).)75 855 y Fv(MPI)p 160 855 V 16 w(W)l(AIT\(request,)h(status\))117 946 y Fu(INOUT)62 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))117 1047 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 1185 y Ft(int)23 b(MPI)p 245 1185 15 2 v 17 w(Wait\(MPI)p 454 1185 V 16 w(Comm)p 566 1185 V 17 w(request)g(*request,)f(MPI)p 1084 1185 V 17 w(Status)h(*status\))75 1284 y(MPI)p 150 1284 V 17 w(WAIT\(REQUEST,)f(STATUS,)h(IERROR\))170 1341 y(INTEGER)g(REQUEST,)g (STATUS\(MPI)p 819 1341 V 16 w(STATUS)p 979 1341 V 16 w(SIZE\),)g(IERROR)166 1440 y FA(A)14 b(call)h(to)f Fv(MPI)p 436 1440 14 2 v 15 w(W)l(AIT)h FA(returns)e(when)i(the)f(send)h(op)q(eration)f(iden)o(ti\014ed)i(b)o(y)e Fv(request)i FA(is)f(complete.)75 1497 y(If)21 b(the)g(comm)o(unication)h(ob) s(ject)f(asso)q(ciated)g(with)g(this)h(request)f(w)o(as)f(created)h(b)o(y)g (a)g(non)o(blo)q(c)o(king)75 1553 y(send)13 b(or)g(receiv)o(e)h(call,)g(then) f(the)g(ob)s(ject)f(is)h(deallo)q(cated)i(b)o(y)d(the)h(call)h(to)25 b Fv(MPI)p 1431 1553 V 16 w(W)l(AIT)13 b FA(and)g(the)g(request)75 1610 y(b)q(ecomes)j(n)o(ull.)189 1782 y Fr(R)n(ationale.)39 b FA(MPI)15 b(alw)o(a)o(ys)f(n)o(ulli\014es)j(handles)f(to)f(ob)s(jects)f (that)g(ha)o(v)o(e)g(b)q(een)i(deallo)q(cated,)g(so)e(as)189 1838 y(to)g(a)o(v)o(oid)h(dangling)h(references.)21 b(\()p Fr(End)16 b(of)g(r)n(ationale.)p FA(\))166 2010 y(The)h(call)g(returns)g(in)g Fv(status)h FA(information)f(on)f(the)h(completed)g(op)q(eration.)24 b(The)17 b(status)f(ob)s(ject)75 2067 y(for)f(a)h(receiv)o(e)g(op)q(eration)g (can)g(b)q(e)g(queried)i(using)e(the)g(functions)g(describ)q(ed)i(in)f (section)f(1.3.1.)42 b(The)69 b Fk(\()75 2123 y FA(status)15 b(ob)s(ject)f(for)h(a)g(send)i(op)q(eration)e(ma)o(y)g(b)q(e)h(queried)h(as)e (w)o(ell)h(b)o(y)g(a)f(call)h(to)f Fv(MPI)p 1559 2123 V 16 w(TEST)p 1689 2123 V 17 w(CANCEL)75 2180 y FA(\(Section)c(1.9,)f(b)q(elo)o (w\).)19 b(Th)o(us,)11 b(one)g(alw)o(a)o(ys)f(need)i(to)e(pro)o(vide)h(a)f(v) m(alid)i(status)e(argumen)o(t)g(to)g Fv(MPI)p 1749 2180 V 16 w(W)l(AIT)75 2236 y FA(and)15 b(the)h(other)e(related)i(functions.)166 2306 y(It)21 b(is)11 b(allo)o(w)o(ed)g(to)f(call)i Fv(MPI)p 634 2306 V 16 w(W)l(AIT)e FA(with)h(a)g(n)o(ull)h Fv(request)g FA(argumen)o(t,)e(in)i(whic)o(h)f(case)g(the)g(op)q(eration)69 b Fk(\()75 2362 y FA(returns)17 b(immediately;)k(the)c Fv(status)j FA(argumen)o(t)d(is)h(set)f(to)g(return)g Fv(tag)h(=)g(MPI)p 1464 2362 V 16 w(ANY)p 1572 2362 V 17 w(T)l(A)o(G)p FA(,)g Fv(source)g(=)75 2419 y(MPI)p 160 2419 V 16 w(ANY)p 268 2419 V 17 w(SOURCE)p FA(,)e(and)g Fv(count)g(=)g(0)p FA(.)189 2591 y Fr(R)n(ationale.)52 b FA(This)18 b(mak)o(es)e Fv(MPI)p 772 2591 V 16 w(W)l(AIT)h FA(functionally)i(equiv)m(alen)o(t)g(to)e Fv(MPI)p 1528 2591 V 16 w(W)l(AIT)l(ALL)g FA(with)g(a)189 2647 y(list)e(of)f(length)h(one)f(and)h(adds)f(some)g(elegance.)21 b(Status)14 b(is)h(set)f(so)g(as)g(to)g(prev)o(en)o(t)g(errors)g(due)h(to)189 2704 y(access)g(to)g(stale)g Fv(status)i FA(information.)j(\()p Fr(End)c(of)g(r)n(ationale.)p FA(\))-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 24 24 bop 75 -100 a FA(24)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(TEST\(request,)g(\015ag,)f(status\))117 130 y Fu(INOUT)62 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))117 212 y(OUT)108 b Fv(\015ag)518 b Fu(\(logical\))117 295 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 423 y Ft(int)23 b(MPI)p 245 423 15 2 v 17 w(Test\(MPI)p 454 423 V 16 w(Comm)p 566 423 V 17 w(request)g(*request,)f(int)i(flag,)f(MPI)p 1323 423 V 17 w(Status)g(*status\))75 513 y(MPI)p 150 513 V 17 w(TEST\(REQUEST,)f(FLAG,)h(STATUS,)g(IERROR\))170 569 y(LOGICAL)g(FLAG)170 626 y(INTEGER)g(REQUEST,)g(STATUS\(MPI)p 819 626 V 16 w(STATUS)p 979 626 V 16 w(SIZE\),)g(IERROR)166 716 y FA(A)17 b(call)i(to)e Fv(MPI)p 446 716 14 2 v 15 w(TEST)h FA(returns)f Fv(\015ag)h(=)f(true)h FA(if)g(the)g(op)q(eration)f(iden)o(ti\014ed)j(b)o(y)d Fv(request)i FA(is)f(com-)75 772 y(plete.)37 b(In)21 b(suc)o(h)g(a)f(case,)i(the)e(status) g(ob)s(ject)g(is)h(set)f(to)g(con)o(tain)h(information)g(on)f(the)h (completed)75 829 y(op)q(eration;)15 b(if)h(the)f(comm)o(unication)h(ob)s (ject)e(w)o(as)h(created)g(b)o(y)g(a)g(non)o(blo)q(c)o(king)h(send)g(or)f (receiv)o(e,)h(then)75 885 y(it)i(is)g(deallo)q(cated)i(and)e(the)g(request)f (b)q(ecomes)i(n)o(ull.)29 b(The)18 b(call)h(returns)f Fv(\015ag=false)p FA(,)g(otherwise.)28 b(In)75 942 y(suc)o(h)16 b(a)e(case,)h(the)h(v)m(alue)g (of)f(the)g(status)f(ob)s(ject)h(is)h(unde\014ned.)166 1002 y Fv(MPI)p 251 1002 V 16 w(TEST)f FA(is)h(a)f(lo)q(cal,)h(non)o(blo)q(c)o (king)g(op)q(eration.)166 1062 y(The)11 b(return)g(status)f(ob)s(ject)h(for)f (a)h(receiv)o(e)h(op)q(eration)f(carries)g(information)g(that)g(can)g(b)q(e)g (accessed)75 1118 y(using)21 b(the)f(functions)h(describ)q(ed)h(in)f(section) g(1.3.1.)68 b(The)21 b(status)e(ob)s(ject)h(for)f(a)h(send)h(op)q(eration) -1903 b Fk(\))75 1174 y FA(carries)22 b(information)h(that)e(can)h(b)q(e)h (accessed)f(b)o(y)g(a)g(call)h(to)f Fv(MPI)p 1283 1174 V 16 w(TEST)p 1413 1174 V 16 w(CANCEL)h FA(\(Section)f(1.9,)75 1231 y(b)q(elo)o(w\).)27 b(Th)o(us,)17 b(one)h(alw)o(a)o(ys)f(need)h(to)f(pro)o (vide)h(a)f(v)m(alid)i(status)d(argumen)o(t)h(to)g Fv(MPI)p 1577 1231 V 16 w(TEST)g FA(and)h(the)75 1287 y(other)d(related)g(functions.) 166 1347 y(It)21 b(is)11 b(allo)o(w)o(ed)g(to)f(call)i Fv(MPI)p 634 1347 V 15 w(TEST)f FA(with)g(a)f(n)o(ull)i Fv(request)g FA(argumen)o(t,)f(in)g(whic)o(h)h(case)e(the)h(op)q(eration)-1903 b Fk(\))75 1404 y FA(returns)17 b Fo(\015ag=true)p FA(,)h(and)f(the)g Fv(status)i FA(argumen)o(t)d(is)h(set)g(to)f(return)h Fv(tag)f(=)h(MPI)p 1465 1404 V 16 w(ANY)p 1573 1404 V 17 w(T)l(A)o(G)p FA(,)g Fv(source)h(=)75 1460 y(MPI)p 160 1460 V 16 w(ANY)p 268 1460 V 17 w(SOURCE)p FA(,)e(and)g Fv(count)g(=)g(0)p FA(.)166 1520 y(In)f(a)f(m)o(ulti-threaded)i(en)o(vironmen)o(t,)e(the)g(use)h(of)f(a)g(blo) q(c)o(king)i(receiv)o(e)f(op)q(eration)g(\()f Fv(MPI)p 1732 1520 V 15 w(W)l(AIT)p FA(\))75 1577 y(ma)o(y)k(allo)o(w)i(the)f(op)q(erating) g(system)g(to)g(de-sc)o(hedule)i(the)e(blo)q(c)o(k)o(ed)h(thread)f(and)h(sc)o (hedule)g(another)75 1633 y(thread)j(for)g(execution,)j(if)e(suc)o(h)g(is)f (a)o(v)m(ailable.)47 b(The)23 b(use)h(of)f(a)g(non)o(blo)q(c)o(king)h(receiv) o(e)h(op)q(eration)75 1690 y(\()p Fv(MPI)p 178 1690 V 15 w(TEST)p FA(\))e(allo)o(ws)f(the)g(user)g(to)g(sc)o(hedule)i(alternativ)o(e)e (activities)i(within)f(a)f(single)h(thread)f(of)75 1746 y(execution.)166 1806 y(Example:)75 1918 y Ft(CALL)h(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1974 y(IF\(rank.EQ.0\))123 2031 y(THEN)170 2087 y(CALL)h(MPI_ISEND\(a\(1\),)e (10,)h(MPI_REAL,)g(1,)g(tag,)h(comm,)f(request,)g(ierr\))170 2144 y(****)h(do)f(some)h(computation)e(to)i(mask)f(latency)g(****)170 2200 y(CALL)h(MPI_WAIT\(request,)d(status,)i(ierr\))123 2257 y(ELSE)170 2313 y(CALL)h(MPI_IRECV\(a\(1\),)e(15,)h(MPI_REAL,)g(0,)g(tag,)h (comm,)f(request,)g(ierr\))170 2370 y(****)h(do)f(some)h(computation)e(to)i (mask)f(latency)g(****)170 2426 y(CALL)h(MPI_WAIT\(request,)d(status,)i (ierr\))75 2483 y(END)g(IF)166 2591 y FA(The)f(functions)h Fv(MPI)p 554 2591 V 15 w(W)l(AIT)f FA(and)g Fv(MPI)p 881 2591 V 16 w(TEST)g FA(can)g(b)q(e)g(used)h(to)e(complete)h(b)q(oth)g(sends)h(and) 75 2647 y(receiv)o(es;)14 b(they)f(are)f(also)h(used)g(to)f(complete)h(an)o (y)g(other)f(non)o(blo)q(c)o(king)i(comm)o(unication)f(call)h(pro)o(vided)75 2704 y(b)o(y)h(MPI.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 25 25 bop 75 -100 a Fw(1.8.)29 b(NONBLOCKING)17 b(COMMUNICA)l(TION)868 b FA(25)75 49 y Fs(1.8.4)49 b(Semantics)13 b(of)k(Nonblo)q(cking)h(Comm)o (unications)75 135 y FA(The)11 b(seman)o(tics)f(of)h(non)o(blo)q(c)o(king)g (comm)o(unication)h(is)f(de\014ned)h(b)o(y)e(suitably)i(extending)g(the)e (de\014nitions)75 192 y(in)16 b(section)g(1.4)e(for)h(order)g(and)g (progress.)166 248 y(Non)o(blo)q(c)o(king)k(comm)o(unication)f(op)q(erations) g(are)g(ordered)g(according)g(to)f(the)h(execution)h(order)75 304 y(of)h(the)g(calls)h(that)e(initiate)j(the)e(comm)o(unication;)i(this)f (de\014nes)g(the)f(meaning)h(of)e(non-o)o(v)o(ertaking)75 361 y(comm)o(unication.)166 417 y(Example:)75 523 y Ft(CALL)k (MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 580 y(IF)h(\(RANK.EQ.0\))147 636 y(THEN)218 693 y(CALL)f(MPI_ISEND\(a,)g(1,)g(MPI_REAL,)g(1,)h(0,)f(comm,) g(h1,)h(ierr\))218 749 y(CALL)f(MPI_ISEND\(b,)g(1,)g(MPI_REAL,)g(1,)h(0,)f (comm,)g(h2,)h(ierr\))147 805 y(ELSE)94 b(!)24 b(rank.EQ.1)218 862 y(CALL)f(MPI_IRECV\(a,)g(1,)g(MPI_REAL,)g(0,)h(MPI_ANY_TAG,)e(comm,)h (h1,)g(ierr\))218 918 y(CALL)g(MPI_IRECV\(b,)g(1,)g(MPI_REAL,)g(0,)h(0,)f (comm,)g(h2,)h(ierr\))75 975 y(END)f(IF)75 1031 y(CALL)g (MPI_WAIT\(h1,status\))75 1088 y(CALL)g(MPI_WAIT\(h2,status\))166 1194 y FA(The)17 b(\014rst)f(send)h(of)f(pro)q(cess)h(zero)g(will)h(matc)o(h) e(the)h(\014rst)f(receiv)o(e)i(of)e(pro)q(cess)h(one,)g(ev)o(en)g(if)g(b)q (oth)75 1250 y(messages)e(are)h(sen)o(t)f(b)q(efore)h(pro)q(cess)g(one)g (executes)h(either)f(receiv)o(e.)22 b(This)17 b(precludes)g(an)f(implemen-)75 1306 y(tation)f(where)g(di\013eren)o(t)h(queues)g(are)f(used)g(to)g(p)q(ost)g (the)g(t)o(w)o(o)f(receiv)o(es.)166 1363 y(A)22 b(call)g(to)f Fv(MPI)p 458 1363 14 2 v 16 w(W)l(AIT)h FA(that)f(completes)h(a)f(receiv)o(e) i(will)g(ev)o(en)o(tually)g(terminate)e(and)h(return)75 1419 y(if)f(a)f(matc)o(hing)g(send)h(has)f(b)q(een)h(started,)g(unless)g(the)g (send)g(is)f(satis\014ed)h(b)o(y)f(another)g(receiv)o(e.)36 b(In)75 1476 y(particular,)20 b(if)f(the)g(matc)o(hing)g(send)g(is)g(non)o (blo)q(c)o(king,)i(then)e(the)g(receiv)o(e)g(should)h(complete)f(ev)o(en)g (if)75 1532 y(no)c(call)i(is)e(executed)i(b)o(y)e(the)g(sender)h(to)f (complete)h(the)f(send.)20 b(Similarly)l(,)e(a)d(call)h(to)f Fv(MPI)p 1651 1532 V 15 w(W)l(AIT)h FA(that)75 1589 y(completes)i(a)e(send)i (will)h(ev)o(en)o(tually)f(return)f(if)g(a)g(matc)o(hing)g(receiv)o(e)h(has)f (b)q(een)h(started,)f(unless)h(the)75 1645 y(receiv)o(e)e(is)g(satis\014ed)f (b)o(y)g(another)g(send,)g(and)g(ev)o(en)h(if)f(no)g(call)h(is)g(executed)g (to)e(complete)i(the)f(receiv)o(e.)166 1702 y(Example:)75 1807 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1864 y(IF)h(\(RANK.EQ.0\))147 1920 y(THEN)218 1977 y(CALL)f(MPI_ISEND\(a,)g(1,)g (MPI_REAL,)g(1,)h(0,)f(comm,)g(h,)h(ierr\))218 2033 y(CALL)f(MPI_RECV\(b,)g (1,)g(MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))218 2090 y(CALL)g(MPI_WAIT\(h,)g (status,)g(ierr\))147 2146 y(ELSE)94 b(!)24 b(rank.EQ.1)218 2203 y(CALL)f(MPI_RECV\(a,)g(1,)g(MPI_REAL,)g(0,)h(0,)f(comm,)g(ierr\))218 2259 y(CALL)g(MPI_SEND\(b,)g(1,)g(MPI_REAL,)g(0,)h(0,)f(comm,)g(ierr\))75 2316 y(END)g(IF)166 2421 y FA(This)16 b(co)q(de)f(should)i(not)d(deadlo)q(c)o (k)j(in)f(a)f(correct)f(MPI)h(implemen)o(tation.)166 2478 y(If)j Fv(MPI)p 299 2478 V 16 w(TEST)g FA(that)e(completes)j(a)e(receiv)o(e)i(is)f (rep)q(eatedly)h(called)g(with)f(the)g(same)f(argumen)o(ts,)75 2534 y(and)d(a)g(matc)o(hing)g(send)g(has)g(b)q(een)h(started,)e(then)h(the)g (call)h(will)g(ev)o(en)o(tually)g(return)f Fv(\015ag=true)p FA(,)h(unless)75 2591 y(the)i(send)g(is)g(satis\014ed)h(b)o(y)f(another)f (receiv)o(e.)25 b(If)17 b Fv(MPI)p 1020 2591 V 16 w(TEST)g FA(that)f(completes)i(a)e(send)i(is)f(rep)q(eatedly)75 2647 y(called)e(with)f(the)g(same)f(argumen)o(ts,)g(and)h(a)f(matc)o(hing)g (receiv)o(e)i(has)e(b)q(een)i(started,)e(then)h(the)g(call)g(will)75 2704 y(ev)o(en)o(tually)i(return)f Fv(\015ag=true)p FA(,)h(unless)g(the)g (receiv)o(e)g(is)f(satis\014ed)h(b)o(y)f(another)g(send.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 26 26 bop 75 -100 a FA(26)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(Non)o(blo)q(c)o(king)f(comm)o (unications)g(cannot)f(b)q(e)g(replaced)i(b)o(y)e(blo)q(c)o(king)h(comm)o (unications,)g(ev)o(en)f(in)75 106 y(the)h Fv(synchronous)i FA(comm)o(unication)f(mo)q(de.)k(Consider)c(the)f(follo)o(wing)h(example:)75 205 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 262 y(IF\(rank.EQ.0\))123 318 y(THEN)170 375 y(CALL)h(MPI_ISEND\(a\(1\),)e (count,)h(MPI_REAL,)f(1,)i(tag1,)f(comm,)g(request1,)g(ierr\))170 431 y(CALL)h(MPI_ISEND\(a\(2\),)e(count,)h(MPI_REAL,)f(1,)i(tag2,)f(comm,)g (request2,)g(ierr\))170 487 y(CALL)h(MPI_WAIT\(request1,)d(status,)i(ierr\)) 170 544 y(CALL)h(MPI_WAIT\(request2,)d(status,)i(ierr\))123 600 y(ELSE)170 657 y(CALL)h(MPI_IRECV\(a\(2\),)e(count,)h(MPI_REAL,)f(0,)i (tag2,)f(comm,)g(request2,)g(ierr\))170 713 y(CALL)h(MPI_IRECV\(a\(1\),)e (count,)h(MPI_REAL,)f(0,)i(tag1,)f(comm,)g(request1,)g(ierr\))170 770 y(CALL)h(MPI_WAIT\(request2,)d(status,)i(ierr\))170 826 y(CALL)h(MPI_WAIT\(request1,)d(status,)i(ierr\))75 883 y(END)g(IF)166 981 y FA(The)e(only)g(resource)g(requiremen)o(t)h(for)e(this)h(co)q(de)h(to)e (complete)h(is)h(the)f(abilit)o(y)h(to)e(ha)o(v)o(e)g(t)o(w)o(o)75 1037 y(p)q(ending)14 b(comm)o(unications.)19 b(If)13 b(one)f(replaces)h(the)f (non)o(blo)q(c)o(king)i(sends)f(\(receiv)o(es\))f(b)o(y)g(blo)q(c)o(king)h (sends)75 1094 y(\(receiv)o(es\),)18 b(and)f(deletes)h(the)g(calls)g(to)f Fv(MPI)p 861 1094 14 2 v 16 w(W)l(AIT)p FA(,)g(then)g(the)h(program)e(can)h (complete)h(only)g(if)g(the)75 1150 y(system)d(bu\013ers)h(at)e(least)i Fv(count)h FA(real)f(n)o(um)o(b)q(ers.)21 b(The)15 b(same)h(happ)q(ens)g(if)g (one)g(deletes)g(the)g(calls)h(that)75 1207 y(initiate)k(comm)o(unication)f (and)g(replaces)g(eac)o(h)g(call)g(to)f Fv(MPI)p 1165 1207 V 16 w(W)l(AIT)g FA(b)o(y)h(the)f(appropriate)h(blo)q(c)o(king)75 1263 y(send)c(or)e(receiv)o(e.)75 1391 y Fs(1.8.5)49 b(Multiple)17 b(Completions)75 1479 y FA(It)23 b(is)h(con)o(v)o(enien)o(t)g(to)f(b)q(e)h (able)g(to)e(w)o(ait)h(for)g(the)g(completion)i(of)d(an)o(y)h(or)g(all)h(the) g(op)q(erations)f(in)75 1536 y(a)g(set,)h(rather)e(than)h(ha)o(ving)g(to)f(w) o(ait)h(for)f(a)g(sp)q(eci\014c)j(message.)43 b(A)23 b(call)h(to)e Fv(MPI)p 1597 1536 V 16 w(W)l(AIT)l(ANY)h FA(or)75 1592 y Fv(MPI)p 160 1592 V 16 w(TEST)l(ANY)e FA(can)f(b)q(e)g(used)g(to)f(w)o(ait)h(for)f (the)h(completion)g(of)g(one)g(out)f(of)g(sev)o(eral)h(op)q(erations;)75 1649 y(a)14 b(call)h(to)27 b Fv(MPI)p 347 1649 V 15 w(W)l(AIT)l(ALL)14 b FA(or)f Fv(MPI)p 701 1649 V 16 w(TEST)l(ALL)h FA(can)g(b)q(e)g(used)h(to)e (w)o(ait)g(for)h(all)g(p)q(ending)i(op)q(erations)e(in)75 1705 y(a)h(list.)75 1810 y Fv(MPI)p 160 1810 V 16 w(W)l(AIT)l(ANY)h(\(count,)g(a)o (rra)o(y)p 640 1810 V 14 w(of)p 691 1810 V 16 w(requests,)h(index,)f (status\))117 1888 y Fu(IN)155 b Fv(count)482 b Fu(list)14 b(length)f(\(in)o(teger\))117 1966 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 1966 V 15 w(of)p 468 1966 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f(handles\))117 2043 y(OUT)108 b Fv(index)489 b Fu(index)12 b(of)f(handle)g(for)g(op)q(eration)h(that)f (completed)g(\(in)o(teger\))117 2121 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 2246 y Ft(int)23 b(MPI)p 245 2246 15 2 v 17 w(Waitany\(int)g(count,)g(MPI)p 788 2246 V 16 w(Comm)p 900 2246 V 17 w(request)g(*array)p 1252 2246 V 16 w(of)p 1316 2246 V 17 w(requests,)g(int)g(*index,)393 2303 y(MPI)p 468 2303 V 17 w(Status)g(*status\))75 2390 y(MPI)p 150 2390 V 17 w(WAITANY\(COUNT,)f(ARRAY)p 645 2390 V 16 w(OF)p 709 2390 V 17 w(REQUESTS,)h(INDEX,)g(STATUS,)g(IERROR\))170 2447 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2447 V 17 w(OF)p 716 2447 V 17 w(REQUESTS\(*\),)f(INDEX,)h(STATUS\(MPI)p 1450 2447 V 16 w(STATUS)p 1610 2447 V 16 w(SIZE\),)170 2503 y(IERROR)166 2591 y FA(Blo)q(c)o(ks)17 b(un)o(til)h(one)e(of)g(the)h(op)q(erations)f(asso) q(ciated)h(with)g(the)f(comm)o(unication)h(requests)g(in)g(the)75 2647 y(arra)o(y)c(has)h(completed.)20 b(Returns)15 b(in)g Fv(index)g FA(the)f(index)h(of)f(that)f(request)h(in)h(the)f(arra)o(y)f(and)h(returns)g (in)75 2704 y Fv(status)i FA(the)f(status)e(of)h(the)h(completing)g(comm)o (unication.)41 b(\(The)14 b(arra)o(y)f(is)i(indexed)h(from)e(zero,)g(b)q(oth) -1903 b Fk(\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 27 27 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(27)75 49 y(in)16 b(C)f(and)g(F)l(ortran.\))166 122 y(The)k(execution)h (of)e Fv(MPI)p 610 122 14 2 v 16 w(W)l(AIT)l(ANY\(count,)j(a)o(rra)o(y)p 1079 122 V 14 w(of)p 1130 122 V 16 w(requests,)h(index,)e(status\))g FA(has)f(the)f(same)75 179 y(e\013ect)10 b(as)g(the)g(execution)i(of)20 b Fv(MPI)p 655 179 V 16 w(W)l(AIT\(a)o(rra)o(y)p 897 179 V 14 w(of)p 948 179 V 16 w(requests[i],)13 b(status\))p FA(,)g(where)e Fv(i)f FA(is)h(the)f(v)m(alue)i(returned)75 235 y(b)o(y)h Fv(index)p FA(.)21 b(In)14 b(particular,)g(the)g(asso)q(ciated)f(comm)o(unication)i(ob)s (ject)d(is)i(deallo)q(cated,)h(and)f(the)g(handle)75 291 y(to)21 b(it)h(in)g Fv(a)o(rra)o(y)p 346 291 V 15 w(of)p 398 291 V 16 w(requests)i FA(is)e(set)f(to)g(n)o(ull.)41 b(If)22 b(more)f(then)h(one)g (op)q(eration)g(is)g(enabled)h(and)f(can)75 348 y(terminate,)d(one)g(is)g (arbitrarily)g(c)o(hosen.)30 b(There)19 b(is)g(no)g(requiremen)o(t)g(that)f (the)g(c)o(hoice)i(satisfy)e(an)o(y)75 404 y(fairness)e(criterion.)166 477 y Fv(MPI)p 251 477 V 16 w(W)l(AIT)30 b FA(is)16 b(equiv)m(alen)o(t)h(to)d Fv(MPI)p 809 477 V 16 w(W)l(AIT)l(ANY)i FA(with)g(an)f(arra)o(y)f(of)h (length)g(one.)348 b Fk(\()75 597 y Fv(MPI)p 160 597 V 16 w(TEST)l (ANY\(count,)17 b(a)o(rra)o(y)p 626 597 V 14 w(of)p 677 597 V 16 w(requests,)g(index,)f(\015ag,)f(status\))117 691 y Fu(IN)155 b Fv(count)482 b Fu(list)14 b(length)f(\(in)o(teger\))117 799 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 799 V 15 w(of)p 468 799 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f (handles\))117 907 y(OUT)108 b Fv(index)489 b Fu(index)9 b(of)g(op)q(eration) g(that)h(completed,)f(or)g Fo(MPI)p 1651 907 13 2 v 14 w(UNDEFINED)905 963 y Fu(if)k(none)h(completed)g(\(in)o(teger\))117 1071 y(OUT)108 b Fv(\015ag)518 b Fo(TRUE)13 b Fu(if)g(one)i(of)e(the)h(op)q(erations)h(is)e (complete)h(\(logical\))117 1179 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 1320 y Ft(int)23 b(MPI)p 245 1320 15 2 v 17 w(Testany\(int)g(count,)g(MPI)p 788 1320 V 16 w(Comm)p 900 1320 V 17 w(request)g(*array)p 1252 1320 V 16 w(of)p 1316 1320 V 17 w(requests,)g(int)g(*index,)393 1376 y(int)h(*flag,)f(MPI)p 731 1376 V 16 w(Status)g(*status\))75 1479 y(MPI)p 150 1479 V 17 w(TESTANY\(COUNT,)f(ARRAY)p 645 1479 V 16 w(OF)p 709 1479 V 17 w(REQUESTS,)h(INDEX,)g(FLAG,)g(STATUS,)g (IERROR\))170 1536 y(LOGICAL)g(FLAG)170 1592 y(INTEGER)g(COUNT,)g(ARRAY)p 651 1592 V 17 w(OF)p 716 1592 V 17 w(REQUESTS\(*\),)f(INDEX,)h(STATUS\(MPI)p 1450 1592 V 16 w(STATUS)p 1610 1592 V 16 w(SIZE\),)170 1649 y(IERROR)166 1751 y FA(Causes)11 b(either)h(one)f(or)g(none)h(of)f(the)g(op)q (erations)h(asso)q(ciated)f(with)h(the)f(comm)o(unication)h(requests)75 1808 y(to)21 b(return.)41 b(In)22 b(the)g(former)g(case,)h(it)f(returns)g Fv(\015ag)g(=)g(true)p FA(,)i(returns)e(in)h Fv(index)g FA(the)f(index)h(of)f (its)75 1864 y(request)12 b(in)h(the)f(arra)o(y)l(,)f(returns)g(in)i Fv(status)h FA(the)e(status)f(of)g(that)g(op)q(eration,)i(deallo)q(cates)g (the)f(asso)q(ciated)75 1921 y(comm)o(unication)f(ob)s(ject,)f(and)h(sets)f (the)g(request)h(to)f(n)o(ull.)38 b(\(The)10 b(arra)o(y)f(is)i(indexed)h (from)e(zero,)h(b)q(oth)f(in)70 b Fk(\()75 1977 y FA(C)12 b(and)h(F)l (ortran.\))18 b(In)13 b(the)f(latter)h(case,)f(it)h(returns)g Fv(\015ag)f(=)h(false)p FA(,)g(returns)f(a)h(v)m(alue)g(of)g Fo(MPI)p 1631 1977 13 2 v 14 w(UNDEFINED)75 2034 y FA(in)j Fv(index)g FA(and)g Fv(status)h FA(is)e(unde\014ned.)166 2107 y(The)k(execution)g(of)f Fv(MPI)p 609 2107 14 2 v 16 w(TEST)l(ANY\(count,)j (a)o(rra)o(y)p 1079 2107 V 15 w(of)p 1131 2107 V 16 w(requests,)g(index,)f (status\))g FA(has)f(the)f(same)75 2163 y(e\013ect)c(as)f(the)h(execution)h (of)e Fv(MPI)p 662 2163 V 16 w(TEST\(a)o(rra)o(y)p 905 2163 V 15 w(of)p 957 2163 V 16 w(requests[i],)j(\015ag,)d(status\))p FA(,)j(for)d Fv(i=0,)g(1)h(,...,)e(count-1)p FA(,)75 2220 y(in)i(some)e (arbitrary)h(order,)g(un)o(til)h(one)f(call)h(returns)f Fv(\015ag=true)p FA(,)h(or)e(all)i(fail.)20 b(In)14 b(the)f(former)f(case,)h Fv(index)75 2276 y FA(is)19 b(set)f(to)g(the)g(last)h(v)m(alue)g(of)f Fv(i)p FA(;)i(in)f(the)f(later)h(case,)g(it)f(is)h(set)f(to)g Fo(MPI)p 1314 2276 13 2 v 14 w(UNDEFINED)p FA(.)36 b(In)19 b(particular,)69 b Fk(\()75 2332 y Fv(MPI)p 160 2332 14 2 v 16 w(TEST)15 b FA(is)h(equiv)m(alen)o(t)h(to)d Fv(MPI)p 704 2332 V 16 w(TEST)l(ANY)j FA(with)e(an)g(arra)o(y)f(of)h(length)h(one.)166 2481 y Fq(Discussion:)48 b Fo(MPI)p 525 2481 13 2 v 15 w(TEST)p 644 2481 V 15 w(ANY)11 b Fu(pro)o(vides)h(t)o(w)o(o)e(indicators)i(of)e (success/failure:)19 b(the)12 b(v)n(alue)e(of)h Fo(\015ag)g Fu(and)68 b Fn(\()75 2531 y Fu(the)12 b(v)n(alue)g(of)f Fo(index)i Fu(\(i.e.,)e(whether)i Fo(index)g Fu(=)f Fo(MPI)p 870 2531 V 14 w(UNDEFINED)p Fu(\).)f(One)h(could)g(do)f(with)h(one)g(only;)f(e.g.,)g (get)h(rid)75 2581 y(of)i Fo(\015ag)p Fu(,)g(or)h(allo)o(w)e Fo(index)j Fu(to)e(b)q(e)i(unde\014ned)f(if)f(no)h(test)g(succeeded.)23 b(This)15 b(ma)o(y)e(buy)h(some)g(e\016ciency)h(and)g(ma)o(y)75 2631 y(sa)o(v)o(e)c(a)f(parameter,)g(at)g(the)h(exp)q(ense)i(of)c(increasing) i(the)g(di\013erence)h(b)q(et)o(w)o(een)g Fo(MPI)p 1384 2631 V 15 w(TEST)e Fu(and)h Fo(MPI)p 1666 2631 V 14 w(TEST)m(ANY)p Fu(.)166 2704 y(Opinions?)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 28 28 bop 75 -100 a FA(28)613 b Fw(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(W)l(AIT)l(ALL\()e(count,)i(a)o(rra)o(y)p 626 49 V 15 w(of)p 678 49 V 16 w(requests,)h(a)o(rra)o(y)p 971 49 V 15 w(of)p 1023 49 V 15 w(statuses\))117 128 y Fu(IN)155 b Fv(count)482 b Fu(lists)14 b(length)g(\(in)o(teger\))117 205 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 205 V 15 w(of)p 468 205 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f(handles\))117 283 y(OUT)108 b Fv(a)o(rra)o(y)p 416 283 V 15 w(of)p 468 283 V 16 w(statuses)276 b Fu(arra)o(y)14 b(of)f(status)i(ob)r(jects)g(\(arra)o(y) f(of)f(Status\))75 409 y Ft(int)23 b(MPI)p 245 409 15 2 v 17 w(Waitall\(int)g(count,)g(MPI)p 788 409 V 16 w(Comm)p 900 409 V 17 w(request)g(*array)p 1252 409 V 16 w(of)p 1316 409 V 17 w(requests,)393 465 y(MPI)p 468 465 V 17 w(Status)g(*array)p 796 465 V 16 w(of)p 860 465 V 17 w(statuses\))75 553 y(MPI)p 150 553 V 17 w(WAITALL\(COUNT,)f(ARRAY)p 645 553 V 16 w(OF)p 709 553 V 17 w(REQUESTS,)h(ARRAY)p 1085 553 V 16 w(OF)p 1149 553 V 17 w(STATUSES,)g(IERROR\))170 610 y(INTEGER)g(COUNT,)g(ARRAY)p 651 610 V 17 w(OF)p 716 610 V 17 w(REQUESTS\(*\),)170 666 y(ARRAY)p 293 666 V 17 w(OF)p 358 666 V 17 w(STATUSES\(MPI)p 663 666 V 15 w(STATUS)p 822 666 V 17 w(SIZE,*\),)g(IERROR)166 754 y FA(Blo)q(c)o(ks)12 b(un)o(til)h(all)g(comm)o(unication)g(op)q(erations)f (asso)q(ciated)g(with)g(requests)f(in)i(the)f(list)h(complete,)75 810 y(and)k(return)g(the)g(status)f(of)h(all)h(these)f(op)q(erations.)26 b(Both)16 b(arra)o(ys)g(ha)o(v)o(e)h(the)g(same)f(n)o(um)o(b)q(er)i(of)e(v)m (alid)75 867 y(en)o(tries.)24 b(The)16 b Fv(i)p FA(-th)h(en)o(try)f(in)h Fv(a)o(rra)o(y)p 694 867 14 2 v 14 w(of)p 745 867 V 16 w(statuses)i FA(is)e(set)f(to)g(the)h(return)f(status)f(of)h(the)h Fv(i)p FA(-th)f(op)q(eration.)75 923 y(All)g(comm)o(unication)g(ob)s(jects)f(are)g (deallo)q(cated)h(and)g(all)g(requests)f(are)g(set)g(to)g(n)o(ull.)166 981 y(The)f(execution)h(of)f Fv(MPI)p 596 981 V 16 w(W)l(AIT)l(ALL\(count,)g (a)o(rra)o(y)p 1046 981 V 15 w(of)p 1098 981 V 16 w(requests,)i(a)o(rra)o(y)p 1390 981 V 15 w(of)p 1442 981 V 16 w(statuses)g FA(has)e(the)g(same)75 1037 y(e\013ect)24 b(as)g(the)g(execution)h(of)f Fv(MPI)p 714 1037 V 16 w(W)l(AIT\(a)o(rra)o(y)p 956 1037 V 14 w(of)p 1007 1037 V 16 w(request[i],)k(a)o(rra)o(y)p 1331 1037 V 15 w(of)p 1383 1037 V 16 w(statuses[i]\))p FA(,)g(for)c Fv(i=0)g(,...,)75 1094 y(count-1)p FA(,)16 b(in)g(some)f(arbitrary)f(order.)75 1199 y Fv(MPI)p 160 1199 V 16 w(TEST)l(ALL\(count,)i(a)o(rra)o(y)p 613 1199 V 14 w(of)p 664 1199 V 16 w(requests,)h(\015ag,)e(a)o(rra)o(y)p 1054 1199 V 14 w(of)p 1105 1199 V 16 w(statuses\))117 1277 y Fu(IN)155 b Fv(count)482 b Fu(lists)14 b(length)g(\(in)o(teger\))117 1355 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 1355 V 15 w(of)p 468 1355 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f (handle\))117 1433 y(OUT)108 b Fv(\015ag)518 b Fu(\(logical\))117 1510 y(OUT)108 b Fv(a)o(rra)o(y)p 416 1510 V 15 w(of)p 468 1510 V 16 w(statuses)276 b Fu(arra)o(y)14 b(of)f(status)i(ob)r(jects)g (\(arra)o(y)f(of)f(Status\))75 1636 y Ft(int)23 b(MPI)p 245 1636 15 2 v 17 w(Testall\(int)g(count,)g(MPI)p 788 1636 V 16 w(Comm)p 900 1636 V 17 w(request)g(*array)p 1252 1636 V 16 w(of)p 1316 1636 V 17 w(requests,)g(int)g(*flag,)393 1693 y(MPI)p 468 1693 V 17 w(Status)g(*array)p 796 1693 V 16 w(of)p 860 1693 V 17 w(statuses\))75 1780 y(MPI)p 150 1780 V 17 w(TESTALL\(COUNT,)f (ARRAY)p 645 1780 V 16 w(OF)p 709 1780 V 17 w(REQUESTS,)h(FLAG,)g(ARRAY)p 1228 1780 V 16 w(OF)p 1292 1780 V 17 w(STATUSES,)g(IERROR\))170 1837 y(LOGICAL)g(FLAG)170 1893 y(INTEGER)g(COUNT,)g(ARRAY)p 651 1893 V 17 w(OF)p 716 1893 V 17 w(REQUESTS\(*\),)170 1950 y(ARRAY)p 293 1950 V 17 w(OF)p 358 1950 V 17 w(STATUSES\(MPI)p 663 1950 V 15 w(STATUS)p 822 1950 V 17 w(SIZE,*\),)g(IERROR)166 2037 y FA(Returns)18 b Fv(\015ag)g(=)g(true)h FA(if)f(all)h(comm)o (unications)g(asso)q(ciated)f(with)g(requests)g(in)h(the)f(arra)o(y)e(ha)o(v) o(e)75 2094 y(completed.)30 b(In)19 b(this)f(case,)h(eac)o(h)f(status)g(en)o (try)g(is)g(set)g(to)g(the)g(status)g(of)g(the)g(corresp)q(onding)h(com-)75 2150 y(m)o(unication.)32 b(All)21 b(comm)o(unication)e(ob)s(jects)g(are)g (deallo)q(cated,)i(and)e(all)h(requests)f(are)g(set)f(to)h(n)o(ull.)75 2207 y(Otherwise,)k Fv(\015ag)e(=)g(false)h FA(is)f(returned,)i(no)e(request) g(is)h(mo)q(di\014ed)g(and)f(the)h(v)m(alues)g(of)f(the)g(status)75 2263 y(en)o(tries)16 b(are)e(unde\014ned.)22 b(This)16 b(is)g(a)f(non)o(blo)q (c)o(king)h(op)q(eration)g(that)e(returns)h(immediately)l(.)166 2321 y(Example:)75 2478 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\)) 75 2534 y(IF\(rank)g(<)h(2\))123 2591 y(THEN)214 b(!)24 b(rank=0)f(or)g (rank=1)170 2647 y(CALL)h(MPI_ISEND\(a,)e(n,)i(MPI_REAL,)e(2,)i(tag,)f(comm,) g(request,)g(ierr\))170 2704 y(****)h(do)f(some)h(computation)e(to)i(mask)f (latency)g(****)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 29 29 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(29)170 49 y Ft(CALL)24 b(MPI_WAIT\(request,)d(status,)i(ierr\))123 106 y(ELSE)214 b(!)24 b(rank=2)170 162 y(CALL)g(MPI_IRECV\(a,)e(n,)i (MPI_REAL,)e(0,)i(tag,)f(comm,)g(request_list\(0\),)f(ierr\))170 219 y(CALL)i(MPI_IRECV\(b,)e(n,)i(MPI_REAL,)e(1,)i(tag,)f(comm,)g (request_list\(1\),)f(ierr\))170 275 y(****)i(do)f(some)h(computation)e(to)i (mask)f(latency)g(****)170 332 y(CALL)h(MPI_WAITANY\(2,)e(request_list,)g (index,)h(status,)g(ierr\))170 388 y(IF\(index.EQ.0\))218 444 y(THEN)266 501 y(****)g(handle)g(message)g(from)g(process)g(0)h(****)266 557 y(CALL)f(MPI_WAIT\(request_list\(1\),)e(status,)i(ierr\))266 614 y(****)g(handle)g(message)g(from)g(process)g(1)h(****)218 670 y(ELSE)266 727 y(****)f(handle)g(message)g(from)g(process)g(1)h(****)266 783 y(CALL)f(MPI_WAIT\(request_list\(0\),)e(status,)i(ierr\))266 840 y(****)g(handle)g(message)g(from)g(process)g(0)h(****)170 896 y(END)g(IF)75 953 y(END)f(IF)166 1055 y FA(The)13 b(calls)h(in)o(tro)q (duced)f(in)h(this)f(subsection)h(can)f(b)q(e)g(used)g(to)f(w)o(ait)g(or)h (test)f(for)g(the)h(completion)g(of)75 1111 y(an)h(arbitrary)f(mix)h(of)f (non)o(blo)q(c)o(king)i(comm)o(unication)f(calls,)h(including)h(a)e(mix)g(of) f(sends)h(and)g(receiv)o(es)75 1168 y(and)h(of)g(an)o(y)g(additional)i(non)o (blo)q(c)o(king)f(comm)o(unication)g(calls)g(pro)o(vided)g(b)o(y)f(MPI.)166 1226 y(An)21 b Fv(a)o(rra)o(y)p 344 1226 14 2 v 14 w(of)p 395 1226 V 16 w(requests)h FA(argumen)o(t)e(that)g(is)g(passed)h(to)f(an)o(y)g (of)g(these)g(functions)h(ma)o(y)f(con)o(tain)75 1283 y(n)o(ull)e(en)o (tries.)24 b(These)17 b(en)o(tries)g(are)g(ignored)g(and)f(the)h(call)h (executes)f(as)f(if)h(it)g(w)o(as)f(supplied)j(an)d(arra)o(y)75 1339 y(of)h(requests)f(con)o(taining)i(only)g(the)f(non-n)o(ull)i(requests)e (for)f(activ)o(e)h(comm)o(unications)h(\(ho)o(w)o(ev)o(er,)e(see)75 1395 y(paragraph)f(b)q(elo)o(w)i(on)f Fv(a)o(rra)o(y)p 585 1395 V 15 w(of)p 637 1395 V 16 w(statuses)p FA(\).)24 b(As)16 b(a)g(request)g(in)h Fv(a)o(rra)o(y)p 1271 1395 V 15 w(of)p 1323 1395 V 16 w(requests)h FA(is)f(set)e(to)h(n)o(ull)h(when)75 1452 y(the)f(corresp)q(onding)h(op)q(eration)f(completes,)h(this)f(allo)o(ws) g(to)g(reuse)g(the)g(same)g(arra)o(y)e(for)i(completing)75 1508 y(in)g(successiv)o(e)f(calls)h(the)f(op)q(erations)g(asso)q(ciated)g (with)g(this)g(arra)o(y)l(.)k(Th)o(us,)14 b(the)h(last)g(example)g(can)g(b)q (e)75 1565 y(rewritten)g(as)g(follo)o(ws:)75 1740 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1797 y(IF\(rank)g(<)h(2\))123 1853 y(THEN)214 b(!)24 b(rank=0)f(or)g(rank=1)170 1909 y(CALL)h (MPI_ISEND\(a,)e(n,)i(MPI_REAL,)e(2,)i(tag,)f(comm,)g(request,)g(ierr\))170 1966 y(CALL)h(MPI_WAIT\(request,)d(status,)i(ierr\))123 2022 y(ELSE)214 b(!)24 b(rank=2)170 2079 y(CALL)g(MPI_IRECV\(a,)e(n,)i(MPI_REAL,)e (0,)i(tag,)f(comm,)g(request_list\(0\),)f(ierr\))170 2135 y(CALL)i (MPI_IRECV\(b,)e(n,)i(MPI_REAL,)e(1,)i(tag,)f(comm,)g(request_list\(1\),)f (ierr\))170 2192 y(CALL)i(MPI_WAITANY\(2,)e(request_list,)g(index,)h(status,) g(ierr\))266 2248 y(****)g(handle)g(first)g(arriving)g(message)47 b(****)170 2305 y(CALL)24 b(MPI_WAITANY\(2,)e(request_list,)g(index,)h (status,)g(ierr\))266 2361 y(****)g(handle)g(second)g(arriving)g(message)47 b(****)166 2478 y FA(A)23 b(call)13 b(to)d Fv(MPI)p 439 2478 V 16 w(W)l(AIT)l(ALL)h FA(and)h(a)f(successful)i(call)g(to)e Fv(MPI)p 1191 2478 V 16 w(TEST)l(ALL)g FA(will)i(set)e(all)i(status)e(en)o (tries)69 b Fk(\()75 2534 y FA(in)14 b Fv(a)o(rra)o(y)p 224 2534 V 14 w(of)p 275 2534 V 16 w(statuses)p FA(,)i(including)g(those)d(that)f (corresp)q(ond)i(to)e(n)o(ull)j(requests)e(in)h Fv(a)o(rra)o(y)p 1567 2534 V 14 w(of)p 1618 2534 V 16 w(requests)p FA(;)i(the)75 2591 y(en)o(tries)11 b(are)f(set)h(to)f(the)g(parameters)g(of)g(the)h (completing)h(request,)f(if)g(the)f(request)h(is)g(non-n)o(ull,)i(or)d(set)h (to)75 2647 y(the)g(parameters)g(of)g(a)g(\\n)o(ull)h(comm)o(unication")g(\() p Fv(tag)f FA(=)h Fo(MPI)p 1121 2647 13 2 v 14 w(ANY)p 1220 2647 V 15 w(T)m(A)o(G)p FA(,)e Fv(source)h FA(=)h Fo(MPI)p 1590 2647 V 14 w(ANY)p 1689 2647 V 15 w(SOURCE)p FA(,)75 2704 y Fv(count)k(=)g(0)p FA(\))f(if)g(the)g(request)h(is)f(n)o(ull.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 30 30 bop 75 -100 a FA(30)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(If)21 b Fv(MPI)p 302 49 14 2 v 16 w(W)l(AIT)l(ANY)h FA(is)f(called)h(with)f(an)g(empt)o(y)g Fv(a)o(rra)o(y)p 1144 49 V 14 w(of)p 1195 49 V 16 w(requests)i FA(argumen)o(t)d(\(i.e.,)h(an)g(arra)o(y)75 106 y(of)h(length)h(zero,)h(or)e (an)h(arra)o(y)e(with)i(all)g(en)o(tries)g(n)o(ull\))h(then)f(it)f(returns)h (immediately)h(with)f(the)75 162 y(v)m(alue)d Fo(MPI)p 273 162 13 2 v 14 w(UNDEFINED)e FA(in)i Fv(index)p FA(;)i(no)c(status)h(is)g(up)q (dated.)32 b Fv(MPI)p 1272 162 14 2 v 16 w(TEST)l(ANY)20 b FA(with)f(an)g(empt)o(y)g Fv(a)o(r-)75 219 y(ra)o(y)p 136 219 V 15 w(of)p 188 219 V 16 w(requests)25 b FA(returns)d Fo(MPI)p 619 219 13 2 v 14 w(UNDEFINED)g FA(in)h Fv(index)g FA(and)g Fv(\015ag)f(=)h(false)p FA(;)j(no)c(status)g(is)g(up)q(dated.)75 275 y Fv(MPI)p 160 275 14 2 v 16 w(W)l(AIT)l(ALL)15 b FA(with)h(an)f(empt)o (y)g Fv(a)o(rra)o(y)p 781 275 V 14 w(of)p 832 275 V 16 w(requests)j FA(argumen)o(t)c(returns)i(immediately)h(with)e(all)i(sta-)75 332 y(tuses)j(up)q(dated)g(to)f(the)h(status)e(of)i(a)f(n)o(ull)i(comm)o (unication.)34 b Fv(MPI)p 1283 332 V 15 w(TEST)l(ALL)20 b FA(with)g(an)f (empt)o(y)h Fv(a)o(r-)75 388 y(ra)o(y)p 136 388 V 15 w(of)p 188 388 V 16 w(requests)f FA(argumen)o(t)d(returns)h Fv(\015ag=true)h FA(and)f(up)q(dates)g(all)h(statuses)e(to)h(the)g(status)f(of)g(a)h(n)o(ull) 75 444 y(comm)o(unication.)75 561 y Fv(Prop)q(osal)e(fo)o(r)f(additional)i (function)-28 599 y Fk(\))75 694 y Fv(MPI)p 160 694 V 16 w (TESTSOME\(incount,)h(a)o(rra)o(y)p 697 694 V 14 w(of)p 748 694 V 16 w(requests,)g(outcount,)f(a)o(rra)o(y)p 1237 694 V 15 w(of)p 1289 694 V 16 w(indices,)g(a)o(rra)o(y)p 1554 694 V 14 w(of)p 1605 694 V 16 w(statuses\))117 771 y Fu(IN)155 b Fv(incount)448 b Fu(length)14 b(of)f(arra)o(y)p 1176 771 13 2 v 15 w(of)p 1225 771 V 14 w(requests)j(\(in)o(teger\))117 842 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 842 14 2 v 15 w(of)p 468 842 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e (of)f(handle\))117 913 y(OUT)108 b Fv(outcount)420 b Fu(n)o(um)o(b)q(er)13 b(of)h(completed)f(requests)j(\(in)o(teger\))117 984 y(OUT)108 b Fv(a)o(rra)o(y)p 416 984 V 15 w(of)p 468 984 V 16 w(indices)299 b Fu(arra)o(y)12 b(of)f(indices)h(of)f(op)q(erations)g(that)h(completed)f (\(arra)o(y)h(of)905 1041 y(in)o(teger\))117 1112 y(OUT)108 b Fv(a)o(rra)o(y)p 416 1112 V 15 w(of)p 468 1112 V 16 w(statuses)276 b Fu(arra)o(y)15 b(of)g(status)h(ob)r(jects)g(for)f(op)q(erations)g(that)h (completed)905 1168 y(\(arra)o(y)e(of)f(Status\))75 1293 y Ft(int)23 b(MPI)p 245 1293 15 2 v 17 w(Testsome\(int)f(incount,)h(MPI)p 859 1293 V 17 w(Comm)p 972 1293 V 17 w(request)g(*array)p 1324 1293 V 16 w(of)p 1388 1293 V 17 w(requests,)393 1349 y(int)h(*outcount,)e (int)i(*array)p 994 1349 V 16 w(of)p 1058 1349 V 17 w(indices,)393 1406 y(MPI)p 468 1406 V 17 w(Status)f(*array)p 796 1406 V 16 w(of)p 860 1406 V 17 w(statuses\))75 1492 y(MPI)p 150 1492 V 17 w(TESTSOME\(INCOUNT,)e(ARRAY)p 716 1492 V 17 w(OF)p 781 1492 V 17 w(REQUESTS,)h(OUTCOUNT,)h(ARRAY)p 1395 1492 V 16 w(OF)p 1459 1492 V 17 w(INDICES,)393 1549 y(ARRAY)p 516 1549 V 17 w(OF)p 581 1549 V 17 w(STATUSES,)f(IERROR\))170 1605 y(INTEGER)h (INCOUNT,)g(ARRAY)p 699 1605 V 17 w(OF)p 764 1605 V 17 w(REQUESTS\(*\),)f (OUTCOUNT,)g(ARRAY)p 1449 1605 V 17 w(OF)p 1514 1605 V 17 w(INDICES\(*\),)170 1661 y(ARRAY)p 293 1661 V 17 w(OF)p 358 1661 V 17 w(STATUSES\(MPI)p 663 1661 V 15 w(STATUS)p 822 1661 V 17 w(SIZE,*\),)h(IERROR)166 1748 y FA(Returns)c(in)h Fv(outcount)h FA(the)e(n)o(um)o(b)q(er)h(of)e (requests)h(from)f(the)i(list)f Fv(a)o(rra)o(y)p 1445 1748 14 2 v 15 w(of)p 1497 1748 V 16 w(requests)i FA(that)d(ha)o(v)o(e)75 1804 y(completed.)i(Returns)14 b(in)g(the)f(\014rst)g Fv(outcount)i FA(lo)q(cations)f(of)f(the)g(arra)o(y)f Fv(a)o(rra)o(y)p 1411 1804 V 14 w(of)p 1462 1804 V 16 w(indices)j FA(the)e(indices)i(of)75 1861 y(these)j(op)q(erations)f(\(index)h(within)h(the)e(arra)o(y)g Fv(a)o(rra)o(y)p 1003 1861 V 14 w(of)p 1054 1861 V 16 w(requests)p FA(;)j(the)e(arra)o(y)e(is)i(indexed)h(from)d(zero,)75 1917 y(b)q(oth)f(in)h(C)e(and)h(F)l(ortran\).)k(Returns)c(on)g(the)g(\014rst)f Fv(outcount)j FA(lo)q(cations)f(of)e(the)h(arra)o(y)e Fv(a)o(rra)o(y)p 1697 1917 V 15 w(of)p 1749 1917 V 16 w(status)75 1974 y FA(the)i(status)g (for)f(these)i(completed)g(op)q(erations.)166 2113 y Fq(Discussion:)32 b Fu(Alternativ)o(e)13 b(c)o(hoice:)18 b(Returns)13 b(in)f Fo(a)o(rra)o(y)p 1078 2113 13 2 v 15 w(of)p 1127 2113 V 14 w(status[i])g Fu(the)h(status)h(of)d(the)i Fo(i)p Fu(-th)g(request,)g(for)75 2169 y(eac)o(h)h(request)i(that)e(completed.)166 2308 y FA(This)i(is)f(a)g (non)o(blo)q(c)o(king)i(op)q(eration)e(that)g(returns)g(immediately)l(.)166 2365 y(The)f(execution)h(of)f Fv(MPI)p 596 2365 14 2 v 16 w (TESTSOME\(incount,)i(a)o(rra)o(y)p 1132 2365 V 14 w(of)p 1183 2365 V 16 w(requests,)h(outcount,)e(a)o(rra)o(y)p 1671 2365 V 15 w(of)p 1723 2365 V 16 w(indices,)75 2421 y(status\))k FA(has)d(the)i(same)e(e\013ect)h(as)g(the)g(execution)h(of)e Fv(MPI)p 1104 2421 V 16 w(TEST\(a)o(rra)o(y)p 1347 2421 V 14 w(of)p 1398 2421 V 16 w(requests[i],)k(\015ag,)d(status\))p FA(,)75 2478 y(for)h Fv(i=0,)g(1)h(,...,)e(count-1)p FA(,)j(in)f(some)f (arbitrary)g(order.)29 b(Whenev)o(er)19 b(a)f(call)h(succeeds,)h(the)f(v)m (alue)g(of)f Fv(i)75 2534 y FA(is)h(app)q(ended)g(to)f Fv(a)o(rra)o(y)p 489 2534 V 14 w(of)p 540 2534 V 16 w(indices)i FA(and)e(the)g(status)f(of)h (the)g(succeeding)i(op)q(eration)e(is)h(app)q(ended)h(to)75 2591 y Fv(a)o(rra)o(y)p 173 2591 V 15 w(of)p 225 2591 V 16 w(statuses)p FA(.)166 2647 y(If)c Fv(MPI)p 297 2647 V 16 w(TESTSOME)g FA(is)g(called)h(with)f(an)g(empt)o(y)f Fv(a)o(rra)o(y)p 1145 2647 V 15 w(of)p 1197 2647 V 16 w(requests)i FA(argumen)o(t)e(then)h(it)g (returns)75 2704 y(immediatelly)h(with)f Fv(outcount)h(=)f(0)p FA(.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 31 31 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(31)189 49 y Fr(R)n(ationale.)70 b FA(The)21 b(use)f(of)g Fv(MPI)p 785 49 14 2 v 16 w(TESTSOME)h FA(a)o(v)o(oids)f(the)h(o)o(v)o (erhead)f(of)f(rep)q(eated)i(calls)h(to)189 106 y Fv(MPI)p 274 106 V 15 w(TEST)l(ANY)14 b FA(that)f(w)o(ould)g(b)q(e)g(otherwise)g (required.)20 b(Also,)14 b(it)f(allo)o(ws)g(to)f(ac)o(hiev)o(e)h(some)g (fair-)189 162 y(ness)g(in)h(the)f(servicing)i(of)e(p)q(ending)i(comm)o (unications.)k(A)14 b(serv)o(er)e(that)h(uses)g Fv(MPI)p 1622 162 V 16 w(TESTSOME)189 219 y FA(to)k(w)o(ait)g(on)g(service)h(requests)g (from)f(all)h(its)g(clien)o(t)h(will)g(not)e(starv)o(e)f(an)o(y)i(clien)o(t.) 28 b(The)17 b(formal)189 275 y(requiremen)o(t)c(is)g(that)e(is)i(a)f(comm)o (unication)h(is)g(enabled,)h(and)f Fv(MPI)p 1360 275 V 16 w(TESTSOME)g FA(is)g(rep)q(eatedly)189 332 y(called,)j(then)g(the)f(enabled)i(op)q (eration)e(will)i(ev)o(en)o(tually)f(complete.)21 b(\()p Fr(End)15 b(of)i(r)n(ationale.)p FA(\))75 509 y Fv(Alternative)f(p)o(rop)q(osal)189 615 y Fr(R)n(ationale.)457 598 y Fj(1)498 615 y FA(Some)g(applications)i(ma)o (y)d(p)q(ost)g(a)h(large)g(n)o(um)o(b)q(er)g(of)g(receiv)o(es)g(\(10's)f({)h (100's\))67 b Fk(\()189 671 y FA(and)18 b(w)o(ait)f(for)g(the)h(completion)h (of)e(an)o(y)l(.)27 b(This,)19 b(in)g(order)e(to)g(tak)o(e)g(adv)m(an)o(tage) g(of)h(ready)f(send)189 728 y(proto)q(col)k(and)g(amortize)g(for)f(load)h(im) o(balances.)39 b(The)21 b(use)g(of)g(a)f(sequen)o(tial)j(structure)d(\(an)189 784 y(arra)o(y)12 b(of)h(requests\))g(is)h(ine\016cien)o(t)i(as)d(it)h(has)f (to)g(b)q(e)h(searc)o(hed)g(sequen)o(tially)h(for)e(eac)o(h)h(call.)20 b(This)189 840 y(results)d(in)h(a)f(linear)h(o)o(v)o(erhead)f(p)q(er)g(in)o (v)o(o)q(cation)h(and)f(a)g(total)f(of)h Fi(O)q FA(\()p Fi(n)1429 824 y Fj(2)1447 840 y FA(\))g(time)h(for)e(handling)j Fi(n)189 897 y FA(comm)o(unications.)k(Instead,)17 b(if)f(the)h(list)g(of)f(requests)g (is)h(an)f(opaque)g(MPI)g(ob)s(ject,)g(then)g(more)189 953 y(e\016cien)o(t)k(data)f(structures)g(can)h(b)q(e)g(used,)h(reducing)g(the)f (o)o(v)o(erhead)f(p)q(er)h(comm)o(unication)h(to)189 1010 y Fi(O)q FA(\(log)8 b(log)g Fi(n)p FA(\),)20 b(where)g Fi(n)h FA(is)f(the)g(total)f(size)i(of)e(the)h(lists)h(\(constan)o(t)e(time)h(is)g (p)q(ossible,)j(if)d(the)189 1066 y(lists)14 b(are)g(disjoin)o(t\).)20 b(Also,)14 b(if)h(the)f(list)h(of)e(requests)h(is)h(an)f(opaque)g(ob)s(ject,) f(one)i(ma)o(y)e(reconsider)189 1123 y(the)20 b(decision)j(to)d(ha)o(v)o(e)g (functions)h(that)f(complete)h(comm)o(unications)h(n)o(ullify)g(nonp)q (ersisten)o(t)189 1179 y(requests:)d(this)d(b)q(eha)o(vior)g(is)g(essen)o (tial)g(only)f(for)g(arra)o(y)f(of)h(requests.)189 1275 y(An)d(in)o(terface)h (whic)o(h)g(allo)o(ws)g(incremen)o(tal)g(mo)q(di\014cations)h(to)e(these)g (lists)h(of)f(requests)h(is)g(prefer-)189 1331 y(able)j(to)e(rep)q(eated)i (uses)g(of)e(a)h(commit-lik)o(e)i(function)f(on)f(a)g(static)g(arra)o(y)f (since)j(these)e(lists)h(will)189 1387 y(c)o(hange)11 b(gradually)g(in)h(man) o(y)e(applications)j(\(e.g.)k(n-b)q(o)q(dy)12 b(tree)f(algorithm\))g(\()p Fr(End)g(of)i(r)n(ationale.)p FA(\))166 1544 y(A)23 b(new)f(opaque)h(ob)s (ject,)h(a)e Fo(collection)i FA(is)f(added)g(to)f(MPI.)g(The)h(functions)h Fv(MPI)p 1648 1544 V 15 w(W)l(AIT)l(ANY)p FA(,)75 1601 y Fv(MPI)p 160 1601 V 16 w(W)l(AIT)l(ALL)p FA(,)17 b Fv(MPI)p 477 1601 V 16 w(TEST)l(ANY)j FA(and)e Fv(MPI)p 888 1601 V 16 w(TEST)l(ALL)g FA(accept)h(a)f Fv(collection)g FA(argumen)o(t,)g(whic)o(h)h(is)75 1657 y(a)i(handle)i(to)e(a)g(set)g(of)g(requests,)i(instead)f(of)f(the)g Fv(a)o(rra)o(y)p 1113 1657 V 15 w(of)p 1165 1657 V 16 w(requests)j FA(argumen)o(t)c(they)i(ha)o(v)o(e)f(no)o(w.)75 1713 y(Whenev)o(er)13 b(a)f(nonp)q(ersisten)o(t)h(request)f(is)h(satis\014ed)f(b)o(y)h(a)e(call)j (to)e Fv(MPI)p 1282 1713 V 15 w(W)l(AIT)l(ANY)i FA(or)d Fv(MPI)p 1646 1713 V 16 w(TEST)l(ANY)p FA(,)75 1770 y(it)16 b(is)g(dropp)q(ed)h(from)e (the)g(collection.)23 b(The)16 b(user)g(sp)q(eci\014es)h(an)f(index)h(for)e (eac)o(h)h(item)g(in)g(a)g(collection.)75 1826 y(When)f(a)f(call)h(to)f Fv(MPI)p 466 1826 V 16 w(W)l(AIT)p 595 1826 V 16 w(ANY)h FA(or)f(other)g (similar)i(function)f(satis\014es)f(a)g(request,)h(then)f(the)h(index)75 1883 y(asso)q(ciated)d(with)g(this)h(request)e(is)i(returned)f(b)o(y)g(the)g (call.)20 b(This)12 b(index)h(can)f(b)q(e)h(an)f(address)g(\(a)f(p)q(oin)o (ter\))75 1939 y(in)16 b(C,)f(and)g(can)g(b)q(e)h(used)g(directly)h(to)d (access)h(the)h(data)e(that)h(w)o(as)f(receiv)o(ed.)75 2053 y Fv(MPI)p 160 2053 V 16 w(COLLECTION)p 453 2053 V 16 w(CREA)l (TE\(collection\))117 2140 y Fu(OUT)108 b Fv(collection)409 b Fu(collection)14 b(\(handle\))75 2275 y Ft(int)23 b(int)h(MPI)p 341 2275 15 2 v 17 w(Collection)p 598 2275 V 15 w(create\(MPI)p 853 2275 V 16 w(Comm)p 965 2275 V 17 w(collection)e(*collection\))75 2371 y(MPI)p 150 2371 V 17 w(COLLECTION)p 407 2371 V 16 w (CREATE\(COLLECTION,)f(IERROR\))170 2428 y(INTEGER)i(COLLECTION,)g(IERROR)166 2524 y FA(The)14 b(function)h Fv(MPI)p 520 2524 14 2 v 16 w(COLLECTION)p 813 2524 V 16 w(CREA)l(TE)g FA(creates)f(a)f(collection)j(ob)s(ject)d(for)g (m)o(ultiple)j(com-)75 2581 y(pletion)h(tests.)p 75 2661 720 2 v 127 2688 a Fy(1)144 2704 y Fx(Prop)q(osal)e(and)f(rationale)h(b)o(y)e (Salmon)h(and)g(Edelsohn)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 32 32 bop 75 -100 a FA(32)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(ADD)p 272 49 V 16 w(TO)p 352 49 V 17 w(COLLECTION\(request,)g (collection,)g(k)o(ey\))117 142 y Fu(IN)155 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))117 249 y(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 357 y(IN)155 b Fv(k)o(ey)525 b Fu(user-sp)q(eci\014ed)17 b(k)o(ey)d(for)f (request)j(\(address\))75 497 y Ft(int)23 b(int)h(MPI)p 341 497 15 2 v 17 w(Add)p 430 497 V 16 w(to)p 494 497 V 17 w(collection\(MPI)p 847 497 V 16 w(Comm)p 959 497 V 16 w(request)f(request,)g(MPI)p 1453 497 V 17 w(Comm)p 1566 497 V 16 w(collection)393 553 y(collection,)g (MPI)p 755 553 V 16 w(Aint)h(key\))75 656 y(MPI)p 150 656 V 17 w(ADD)p 239 656 V 17 w(TO)p 304 656 V 16 w(COLLECTION\(REQUEST,)e (COLLECTION,)g(KEY,)h(IERROR\))170 712 y(INTEGER)g(REQUEST,)g(COLLECTION,)f (KEY,)i(IERROR)166 815 y FA(The)15 b(function)g Fv(MPI)p 521 815 14 2 v 16 w(ADD)p 633 815 V 16 w(TO)p 713 815 V 17 w(COLLECTION)f FA(inserts)h Fv(request)h FA(in)o(to)f(the)f(set)g(of)g(requests)h(asso-)75 871 y(ciated)h(with)f Fv(collection)h FA(and)f(asso)q(ciates)h(the)f(user-sp) q(eci\014ed)i Fv(k)o(ey)f FA(with)f(it.)75 991 y Fv(MPI)p 160 991 V 16 w(DELETE)p 346 991 V 16 w(FROM)p 490 991 V 17 w (COLLECTION\(request,)h(collection\))117 1084 y Fu(IN)155 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))117 1191 y(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))75 1331 y Ft(int)23 b(int)h(MPI)p 341 1331 15 2 v 17 w(Delete)p 502 1331 V 16 w(from)p 614 1331 V 17 w(collection\(MPI)p 967 1331 V 15 w(Comm)p 1078 1331 V 17 w(request)e(request,)393 1388 y(MPI)p 468 1388 V 17 w(Comm)p 581 1388 V 17 w(collection)g (collection\))75 1490 y(MPI)p 150 1490 V 17 w(DELETE)p 311 1490 V 16 w(FROM)p 423 1490 V 17 w(COLLECTION\(REQUEST,)f(COLLECTION,)h (IERROR\))170 1547 y(INTEGER)h(REQUEST,)g(COLLECTION,)f(IERROR)166 1649 y FA(The)d(function)g Fv(MPI)p 529 1649 14 2 v 16 w(DELETE)p 715 1649 V 16 w(FROM)p 859 1649 V 17 w(COLLECTION)f FA(remo)o(v)o(es)g Fv(request)i FA(from)e(the)h(set)f(of)g(re-)75 1706 y(quests)d(asso)q(ciated) h(with)f Fv(collection)p FA(.)75 1825 y Fv(MPI)p 160 1825 V 16 w(W)l(AIT)l(ANY\(collection,)h(value,)f(status\))117 1918 y Fu(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 2025 y(OUT)108 b Fv(k)o(ey)525 b Fu(user-sp)q(eci\014ed)17 b(k)o(ey)d(\(address\))117 2133 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 2273 y Ft(int)23 b(int)h(MPI)p 341 2273 15 2 v 17 w(Waitany\(MPI)p 622 2273 V 15 w(Comm)p 733 2273 V 17 w(collection)e(collection,)h(MPI)p 1371 2273 V 17 w(Aint)g(*key,)393 2329 y(MPI)p 468 2329 V 17 w(Status)g(*status\))75 2432 y(MPI)p 150 2432 V 17 w(WAITANY\(COLLECTION,)e(KEY,)i(STATUS,)g (IERROR\))170 2488 y(INTEGER)g(COLLECTION,)g(KEY,)g(STATUS\(MPI)p 1010 2488 V 16 w(STATUS)p 1170 2488 V 16 w(SIZE\),)g(IERROR)166 2591 y FA(The)11 b(function)g Fv(MPI)p 513 2591 14 2 v 16 w(W)l(AIT)l(ANY)h FA(w)o(aits)e(for)g(the)h(completion)h(of)e(one)h(request)g(from)f(the)h (collection.)75 2647 y(If)k(this)g(request)g(is)h(nonp)q(ersisten)o(t,)f (then)g(its)g(is)g(dropp)q(ed)h(from)e(the)h(collection.)22 b(The)15 b(k)o(ey)g(asso)q(ciated)75 2704 y(with)h(the)f(request)g(is)h (returned)f(in)h Fv(k)o(ey)p FA(.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 33 33 bop 75 -100 a Fw(1.8.)34 b(NONBLOCKING)17 b(COMMUNICA)l(TION)863 b FA(33)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(TEST)l(ANY\(collection,)16 b(\015ag,)f(k)o(ey)l(,)g(status\))117 129 y Fu(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 209 y(OUT)108 b Fv(\015ag)518 b Fo(true)11 b Fu(if)d(one)i(of)f(the)h(op)q(erations)g(is)f(complete)g Fo(false)p Fu(,)h(otherwise)905 266 y(\(logical\))117 346 y(OUT)108 b Fv(k)o(ey)525 b Fu(user-sp)q(eci\014ed)17 b(k)o(ey)d(\(address\))117 426 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 553 y Ft(int)23 b(int)h(MPI)p 341 553 15 2 v 17 w(Testany\(MPI)p 622 553 V 15 w(Comm)p 733 553 V 17 w(collection)e(collection,)h(int)g(*flag,) g(MPI)p 1633 553 V 17 w(Aint)393 610 y(*key,)g(MPI)p 611 610 V 17 w(Status)g(*status\))75 699 y(MPI)p 150 699 V 17 w(TESTANY\(COLLECTION,) e(FLAG,)i(KEY,)h(STATUS,)e(IERROR\))170 755 y(INTEGER)h(COLLECTION,)g(FLAG,)g (KEY,)g(STATUS\(MPI)p 1153 755 V 16 w(STATUS)p 1313 755 V 16 w(SIZE\),)g(IERROR)166 844 y Fv(MPI)p 251 844 14 2 v 16 w(TEST)l(ANY)c FA(returns)f Fv(\015ag)g(=)g(true)g FA(if)h(an)o(y)e(of)h(the)g(op)q (erations)g(asso)q(ciated)g(with)g Fv(collection)75 901 y FA(has)f (completed.)28 b(In)19 b(this)f(case,)f(it)h(returns)g(in)g Fv(k)o(ey)g FA(the)f(user-sp)q(eci\014ed)j(v)m(alue)f(asso)q(ciated)f(with)g (the)75 957 y(request)d(handle)g(whic)o(h)h(completed,)f(in)h Fv(status)g FA(the)f(status)f(of)g(that)g(op)q(eration,)g(and)h(deallo)q (cates)h(the)75 1014 y(asso)q(ciated)f(comm)o(unications)h(ob)s(ject)f(if)h (ephemeral.)21 b(It)15 b(returns)g Fv(\015ag)g(=)g(false)p FA(,)g(otherwise.)75 1120 y Fv(MPI)p 160 1120 V 16 w(W)l(AIT)l (ALL\(collection,)g(a)o(rra)o(y)p 684 1120 V 14 w(of)p 735 1120 V 16 w(k)o(eys,)g(a)o(rra)o(y)p 953 1120 V 15 w(of)p 1005 1120 V 16 w(statuses\))117 1200 y Fu(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 1280 y(OUT)108 b Fv(a)o(rra)o(y)p 416 1280 V 15 w(of)p 468 1280 V 16 w(k)o(eys)345 b Fu(arra)o(y)14 b(of)f(user-sp)q(eci\014ed)k(k)o(eys)d(\(arra)o(y)g(of)f(addresses\))117 1360 y(OUT)108 b Fv(a)o(rra)o(y)p 416 1360 V 15 w(of)p 468 1360 V 16 w(statuses)276 b Fu(arra)o(y)14 b(of)f(status)i(ob)r(jects)g (\(arra)o(y)f(of)f(Status\))75 1487 y Ft(int)23 b(int)h(MPI)p 341 1487 15 2 v 17 w(Waitall\(MPI)p 622 1487 V 15 w(Comm)p 733 1487 V 17 w(collection)e(collection,)h(MPI)p 1371 1487 V 17 w(Aint)g(*array)p 1651 1487 V 16 w(of)p 1715 1487 V 17 w(keys,)393 1544 y(MPI)p 468 1544 V 41 w(Status)g(*array)p 820 1544 V 16 w(of)p 884 1544 V 17 w(statuses\))75 1633 y(MPI)p 150 1633 V 17 w(WAITALL\(COLLECTION,)e(ARRAY)p 764 1633 V 17 w(OF)p 829 1633 V 16 w(KEYS,)j(ARRAY)p 1109 1633 V 16 w(OF)p 1173 1633 V 17 w(STATUSES,)f(IERROR\))170 1689 y(INTEGER)g(COLLECTION,)g (ARRAY)p 771 1689 V 16 w(OF)p 835 1689 V 17 w(KEYS,)g(ARRAY)p 1115 1689 V 17 w(OF)p 1180 1689 V 17 w(STATUSES\(MPI)p 1485 1689 V 15 w(STATUS)p 1644 1689 V 17 w(SIZE,*\),)170 1746 y(IERROR)166 1835 y FA(The)14 b(function)g Fv(MPI)p 519 1835 14 2 v 16 w(W)l(AIT)l(ALL)f FA(blo)q(c)o(ks)h(un)o(til)h(all)f(comm)o(unication)g(op)q(erations)g(asso)q (ciated)g(with)75 1891 y(the)g(requests)g(in)h Fv(collection)g FA(complete.)20 b Fv(MPI)p 866 1891 V 16 w(W)l(AIT)l(ALL)13 b FA(returns)h(the)g(user-sp)q(eci\014ed)j(k)o(ey)d(asso)q(ciated)75 1948 y(with)21 b(eac)o(h)g(request)g(handle)h(in)f Fv(a)o(rra)o(y)p 764 1948 V 15 w(of)p 816 1948 V 16 w(k)o(eys)g FA(and)g(the)g(resp)q(ectiv)o (e)g(status)f(of)g(eac)o(h)h(op)q(eration)g(in)75 2004 y Fv(a)o(rra)o(y)p 173 2004 V 15 w(of)p 225 2004 V 16 w(statuses)p FA(.)h(All)16 b(ephemeral)h(comm)o(unication)f(ob)s(jects)e(are)h(deallo)q(cated.)75 2110 y Fv(MPI)p 160 2110 V 16 w(TEST)l(ALL\(collection,)g(\015ag,)g(a)o(rra)o (y)p 782 2110 V 14 w(of)p 833 2110 V 16 w(k)o(eys,)g(a)o(rra)o(y)p 1051 2110 V 15 w(of)p 1103 2110 V 16 w(statuses\))117 2190 y Fu(INOUT)62 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 2271 y(OUT)108 b Fv(\015ag)518 b Fo(true)15 b Fu(if)e(all)g(of)g(the)i(op)q (erations)f(are)g(complete)f(\(logical\))117 2351 y(OUT)108 b Fv(a)o(rra)o(y)p 416 2351 V 15 w(of)p 468 2351 V 16 w(k)o(eys)345 b Fu(arra)o(y)14 b(of)f(user-sp)q(eci\014ed)k(k)o(eys)d(\(arra)o(y)g(of)f (addresses\))117 2431 y(OUT)108 b Fv(a)o(rra)o(y)p 416 2431 V 15 w(of)p 468 2431 V 16 w(statuses)276 b Fu(arra)o(y)14 b(of)f(status)i(ob) r(jects)g(\(arra)o(y)f(of)f(Status\))75 2558 y Ft(int)23 b(int)h(MPI)p 341 2558 15 2 v 17 w(Testall\(MPI)p 622 2558 V 15 w(Comm)p 733 2558 V 17 w(collection)e(collection,)h(int)g(*flag,)g(MPI)p 1633 2558 V 17 w(Aint)393 2615 y(*array)p 540 2615 V 17 w(of)p 605 2615 V 16 w(keys,)h(MPI)p 837 2615 V 16 w(Status)f(*array)p 1164 2615 V 17 w(of)p 1229 2615 V 17 w(statuses\))75 2704 y(MPI)p 150 2704 V 17 w(TESTALL\(COLLECTION,)e(FLAG,)i(ARRAY)p 907 2704 V 17 w(OF)p 972 2704 V 17 w(KEYS,)g(ARRAY)p 1252 2704 V 16 w(OF)p 1316 2704 V 17 w(STATUSES,)g(IERROR\))-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 34 34 bop 75 -100 a FA(34)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)170 49 y Ft(INTEGER)23 b(COLLECTION,)g(FLAG,) g(ARRAY)p 914 49 15 2 v 16 w(OF)p 978 49 V 17 w(KEYS,)170 106 y(ARRAY)p 293 106 V 17 w(OF)p 358 106 V 17 w(STATUSES\(MPI)p 663 106 V 15 w(STATUS)p 822 106 V 17 w(SIZE,*\),)g(IERROR)166 192 y Fv(MPI)p 251 192 14 2 v 16 w(TEST)l(ALL)12 b FA(returns)g Fv(\015ag)h(=)g(true)g FA(if)f(all)i(of)e(the)g(op)q(erations)h(asso)q (ciated)g(with)f Fv(collection)h FA(ha)o(v)o(e)75 249 y(completed.)31 b(In)19 b(this)g(case,)g(it)f(returns)h(in)g(the)g(user-sp)q(eci\014ed)i(k)o (ey)d(asso)q(ciated)h(with)f(eac)o(h)h(request)75 305 y(handle)f(in)f Fv(a)o(rra)o(y)p 375 305 V 14 w(of)p 426 305 V 16 w(k)o(eys)g FA(and)f(the)h(resp)q(ectiv)o(e)g(status)f(of)f(eac)o(h)i(op)q(eration)f(in)h Fv(a)o(rra)o(y)p 1564 305 V 15 w(of)p 1616 305 V 16 w(statuses)p FA(.)26 b(All)75 362 y(ephemeral)17 b(comm)o(unication)g(ob)s(jects)f(are)g (deallo)q(cated.)25 b(It)16 b(returns)g Fv(\015ag)g(=)h(false)p FA(,)f(otherwise.)23 b(This)75 418 y(is)16 b(a)f(non)o(blo)q(c)o(king)h(op)q (erations)f(that)g(returns)g(immediately)l(.)75 522 y Fv(MPI)p 160 522 V 16 w(COLLECTION)p 453 522 V 16 w(SIZE\(collection,)g(count\))117 599 y Fu(IN)155 b Fv(collection)409 b Fu(collection)14 b(\(handle\))117 673 y(OUT)108 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(requests)i(in)d (collection)g(\(in)o(teger\))75 797 y Ft(int)23 b(int)h(MPI)p 341 797 15 2 v 17 w(Collection)p 598 797 V 15 w(size\(MPI)p 805 797 V 17 w(Comm)p 918 797 V 16 w(collection)f(collection,int)f(*count\)) 75 883 y(MPI)p 150 883 V 17 w(COLLECTION)p 407 883 V 16 w(SIZE\(COLLECTION,)f (COUNT,)i(IERROR\))p Fv(I)p Ft(NTEGER)f(COLLECTION,)g(COUNT,)393 940 y(IERROR)166 1026 y Fv(MPI)p 251 1026 14 2 v 16 w(COLLECTION)p 544 1026 V 16 w(SIZE)f FA(returns)h(the)f(n)o(um)o(b)q(er)h(of)f(requests)h (in)g(the)g(collection)h(for)e(whic)o(h)75 1083 y(there)16 b(is)h(an)f(incomplete)i(comm)o(unication.)24 b(The)16 b(coun)o(t)g(ma)o(y)g (b)q(e)g(used)h(as)f(the)g(size)i(of)d(the)i(arra)o(y)e(to)75 1139 y(pass)g(to)g Fv(MPI)p 315 1139 V 15 w(TEST)l(ALL)h FA(and)f Fv(MPI)p 706 1139 V 16 w(W)l(AIT)l(ALL)p FA(.)75 1243 y Fv(MPI)p 160 1243 V 16 w(COLLECTION)p 453 1243 V 16 w(FREE\(collection\))117 1320 y Fu(IN)155 b Fv(collection)409 b Fu(collection)14 b(\(handle\))75 1445 y Ft(int)23 b(int)h(MPI)p 341 1445 15 2 v 17 w(Collection)p 598 1445 V 15 w(free\(MPI)p 805 1445 V 17 w(Comm)p 918 1445 V 16 w(collection)f(collection\))75 1531 y(MPI)p 150 1531 V 17 w(COLLECTION)p 407 1531 V 16 w(FREE\(COLLECTION,)e(IERROR\))170 1588 y(INTEGER)i(COLLECTION,)g(IERROR)166 1674 y FA(The)12 b(function)g Fv(MPI)p 515 1674 14 2 v 16 w(COLLECTION)p 808 1674 V 16 w(FREE)h FA(deallo)q(cates)f(the)g(collection)h(ob)s(ject.)18 b(It)12 b(is)g(erroneous)75 1730 y(to)j(initiate)h(a)f(m)o(ultiple)i (completion)f(test)f(with)h(a)e(collection)j(that)e(has)g(b)q(een)h(freed.) 166 1870 y Fq(Discussion:)34 b Fu(Some)13 b(\(less)h(p)q(o)o(w)o(erful\))g (alternativ)o(es:)75 1926 y(Drop)f(the)g(k)o(ey:)18 b(the)13 b(user)h(will)e(need)h(to)g(use)h(message)e(tags,)h(in)f(order)i(to)f(iden)o (tify)f(requests)j(\(to)e(the)g(least,)g(this)75 1982 y(requires)i Fo(MPI)p 308 1982 13 2 v 14 w(GET)p 403 1982 V 15 w(T)m(A)o(G)e Fu(to)h(w)o(ork)f(after)h(a)g(send,)g(to)q(o\).)75 2039 y(Drop)g(the)p 255 2039 V 29 w(ADD)f(and)p 459 2039 V 29 w(DELETE.)h(Basically)m(,)e(this)i (amoun)o(ts)e(to)i(ha)o(ving)f(a)g(\\commit)p 1482 2039 V 12 w(arra)o(y")h(function.)75 2263 y FB(1.9)59 b(Prob)r(e)20 b(and)f(Cancel)75 2365 y FA(The)d Fv(MPI)p 254 2365 14 2 v 16 w(PROBE)g FA(and)g Fv(MPI)p 605 2365 V 16 w(IPROBE)g FA(op)q(erations)g(allo)o(w)g(incoming)h (messages)e(to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)75 2421 y(without)g (actually)h(receiving)h(them.)k(The)17 b(user)f(can)g(then)h(decide)g(ho)o(w) f(to)g(receiv)o(e)h(them,)f(based)g(on)75 2478 y(the)i(information)f (returned)h(b)o(y)f(the)h(prob)q(e)g(\(basically)l(,)h(the)e(information)h (returned)g(b)o(y)f Fv(status)p FA(\).)28 b(In)75 2534 y(particular,)16 b(the)g(user)g(ma)o(y)f(allo)q(cate)h(memory)f(for)g(the)h(receiv)o(e)h (bu\013er,)e(according)h(to)f(the)h(length)g(of)75 2591 y(the)f(prob)q(ed)h (message.)166 2647 y(The)i Fv(MPI)p 347 2647 V 16 w(CANCEL)g FA(op)q(eration)g(allo)o(ws)g(p)q(ending)i(comm)o(unications)e(to)g(b)q(e)g (canceled.)30 b(This)18 b(is)75 2704 y(required)i(for)f(clean)o(up.)33 b(P)o(osting)19 b(a)g(send)h(or)f(a)g(receiv)o(e)h(ties)f(up)h(user)f (resources)h(\(send)f(or)g(receiv)o(e)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 35 35 bop 75 -100 a Fw(1.9.)34 b(PR)o(OBE)16 b(AND)f(CANCEL)1161 b FA(35)75 49 y(bu\013ers\),)14 b(and)i(a)f(cancel)h(ma)o(y)f(b)q(e)g(needed) i(to)d(free)i(these)f(resources)g(gracefully)l(.)75 165 y Fv(MPI)p 160 165 14 2 v 16 w(IPROBE\(source,)h(tag,)f(comm)m(,)c(\015ag,)k(status\)) 117 255 y Fu(IN)155 b Fv(source)468 b Fu(source)15 b(rank,)f(or)27 b Fo(MPI)p 1281 255 13 2 v 15 w(ANY)p 1381 255 V 14 w(SOURCE)13 b Fu(\(in)o(teger\))117 355 y(IN)155 b Fv(tag)526 b Fu(tag)14 b(v)n(alue)f(or)28 b Fo(MPI)p 1226 355 V 14 w(ANY)p 1325 355 V 15 w(T)m(A)o(G)12 b Fu(\(in)o(teger\))117 455 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 555 y(OUT)108 b Fv(\015ag)518 b Fu(\(logical\))117 655 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 792 y Ft(int)23 b(MPI)p 245 792 15 2 v 17 w(Iprobe\(int)g(source,)g(int)g(tag,)g(MPI)p 1002 792 V 17 w(Comm)g(comm,)h(int)f(*flag,)393 849 y(MPI)p 468 849 V 17 w(Status)g(*status\))75 948 y(MPI)p 150 948 V 17 w(IPROBE\(SOURCE,)f(TAG,)h(COMM,)g(FLAG,)g(STATUS,)g(IERROR\))170 1004 y(LOGICAL)g(FLAG)170 1061 y(INTEGER)g(SOURCE,)g(TAG,)h(COMM,)f (STATUS\(MPI)p 1058 1061 V 16 w(STATUS)p 1218 1061 V 16 w(SIZE\),)g(IERROR) 166 1160 y Fv(MPI)p 251 1160 14 2 v 16 w(IPROBE)c FA(returns)f Fv(\015ag)g(=)h(true)f FA(if)h(there)g(is)f(a)g(message)g(that)g(can)g(b)q(e) h(receiv)o(ed)g(and)g(that)75 1216 y(matc)o(hes)i(the)g(pattern)f(sp)q (eci\014ed)j(b)o(y)e(the)h(argumen)o(ts)e Fv(source)p FA(,)i Fv(tag)p FA(,)g(and)g Fv(comm)m FA(.)34 b(It)21 b(returns)g Fv(\015ag)75 1273 y(=)i(false)p FA(,)h(otherwise.)41 b(If)23 b Fv(MPI)p 633 1273 V 16 w(IPROBE)g FA(returns)f Fv(\015ag)g(=)h(true)p FA(,)h(then)f(the)f(status)g(ob)s(ject)g(can)g(b)q(e)75 1329 y(subsequen)o(tly)c(queried)g(using)g(the)f(functions)h(describ)q(ed)g(in)g (section)g(1.3.1)d(to)h(\014nd)i(the)f(source,)g(tag)75 1385 y(and)f(length)h(of)f(the)g(message.)23 b(The)16 b(v)m(alues)i(returned)e(b)o (y)h(these)f(queries)h(are)f(the)g(same)g(v)m(alues)i(that)75 1442 y(w)o(ould)12 b(ha)o(v)o(e)e(b)q(een)j(returned)e(b)o(y)h(a)f(call)h(to) e Fv(MPI)p 897 1442 V 16 w(RECV)i FA(executed)g(at)f(the)g(same)g(p)q(oin)o (t)h(in)g(the)f(program.)75 1498 y(The)k(return)h(status)e(ob)s(ject)h(is)g (unde\014ned)i(if)f Fv(\015ag)f(=)h(false)p FA(.)166 1567 y(A)d(subsequen)o (t)i(receiv)o(e)f(executed)g(with)g(the)g(same)f(con)o(text,)g(and)g(the)h (source)f(and)h(tag)f(returned)75 1624 y(b)o(y)g(querying)h(the)f(status)f (returned)i(b)o(y)f Fv(MPI)p 858 1624 V 16 w(IPROBE)h FA(will)h(receiv)o(e)f (the)f(message)f(that)h(w)o(as)f(matc)o(hed)75 1680 y(b)o(y)21 b(the)g(prob)q(e,)h(if)g(no)f(other)f(in)o(terv)o(ening)i(receiv)o(e)g(o)q (ccurred)f(after)g(the)g(prob)q(e.)37 b(If)21 b(the)g(receiving)75 1737 y(pro)q(cess)e(is)h(m)o(ulti-threaded,)h(it)f(is)f(the)h(user's)f(resp)q (onsibilit)o(y)i(to)e(ensure)h(that)e(the)i(last)f(condition)75 1793 y(holds.)166 1862 y(The)d Fv(source)h FA(argumen)o(t)e(of)h Fv(MPI)p 738 1862 V 16 w(PROBE)h FA(can)f(b)q(e)h Fo(MPI)p 1140 1862 13 2 v 14 w(ANY)p 1239 1862 V 15 w(SOURCE)p FA(,)d(and)i(the)h Fv(tag)f FA(argumen)o(t)75 1919 y(can)g(b)q(e)g Fo(MPI)p 298 1919 V 15 w(ANY)p 398 1919 V 14 w(T)m(A)o(G)p FA(,)f(so)g(that)g(one)h(can)g (prob)q(e)g(for)f(messages)h(from)f(an)h(arbitrary)f(source)h(and/or)75 1975 y(with)g(an)g(arbitrary)f(tag.)20 b(Ho)o(w)o(ev)o(er,)15 b(a)g(sp)q(eci\014c)j(comm)o(unication)e(con)o(text)f(m)o(ust)g(pro)o(vided)i (with)f(the)75 2031 y Fv(comm)9 b FA(argumen)o(t.)166 2100 y(A)32 b(call)17 b(to)e Fv(MPI)p 457 2100 14 2 v 16 w(PROBE)i FA(will)g(not)f(matc)o(h)f(a)g(message)h(that)f(has)h(already)g(b)q(een)h (matc)o(hed)e(b)o(y)h(a)69 b Fk(\()75 2157 y FA(previous)17 b(receiv)o(e)g(op)q(eration.)23 b(It)16 b(ma)o(y)f(matc)o(h)h(a)f(message)h (that)f(has)h(b)q(een)h(matc)o(hed)f(b)o(y)g(a)g(previous)75 2213 y(prob)q(e)f(op)q(eration)g(or)f(b)o(y)h(a)f(concurren)o(t)h(receiv)o(e) g(\(e.g.,)e(a)h(receiv)o(e)i(executed)g(b)o(y)e(a)g(concurren)o(t)h(thread)75 2270 y(in)h(a)f(m)o(ultithreaded)h(en)o(vironmen)o(t.)166 2421 y Fq(Implemen)o(tati)o(on)g(note:)47 b Fu(MPI)17 b(guaran)o(tees)h(that)f (successiv)o(e)i(messages)d(sen)o(t)i(from)d(a)i(source)h(to)e(a)75 2478 y(destination)10 b(within)f(the)i(same)f(con)o(text)h(are)f(receiv)o(ed) i(in)d(the)i(order)g(they)g(are)g(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)g(supp)q(ort,)75 2534 y(either)k(explicitly)e(or)h (implicitly)l(,)d(a)j(FIF)o(O)g(structure)i(to)e(manage)f(messages)h(b)q(et)o (w)o(een)h(eac)o(h)g(pair)e(of)h(pro)q(cesses,)75 2591 y(for)k(eac)o(h)g(con) o(text.)56 b Fo(MPI)p 516 2591 13 2 v 14 w(PROBE)16 b Fu(returns)i (information)c(on)j(the)h(\014rst)g(matc)o(hing)d(message)i(in)f(this)h(FIF)o (O;)75 2647 y(this)d(will)e(also)i(b)q(e)g(the)h(message)e(receiv)o(ed)i(b)o (y)f(the)g(\014rst)h(subsequen)o(t)h(receiv)o(e)f(with)e(the)i(same)e (source,)i(tag)e(and)75 2704 y(con)o(text)i(as)f(the)g(message)g(matc)o(hed)f (b)o(y)g Fo(MPI)p 806 2704 V 15 w(PROBE)p Fu(.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 36 36 bop 75 -100 a FA(36)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(PROBE\(source,)g(tag,)f(comm)m(,)d(status\))117 126 y Fu(IN)155 b Fv(source)468 b Fu(source)15 b(rank,)f(or)27 b Fo(MPI)p 1281 126 13 2 v 15 w(ANY)p 1381 126 V 14 w(SOURCE)13 b Fu(\(in)o(teger\))117 200 y(IN)155 b Fv(tag)526 b Fu(tag)14 b(v)n(alue,)e(or)28 b Fo(MPI)p 1237 200 V 14 w(ANY)p 1336 200 V 15 w(T)m(A)o(G)13 b Fu(\(in)o(teger\))117 274 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 347 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 472 y Ft(int)23 b(MPI)p 245 472 15 2 v 17 w(Probe\(int)g(source,)g(int)g (tag,)h(MPI)p 979 472 V 16 w(Comm)g(comm,)f(MPI)p 1330 472 V 17 w(Status)g(*status\))75 558 y(MPI)p 150 558 V 17 w(PROBE\(SOURCE,)f (TAG,)h(COMM,)g(STATUS,)g(IERROR\))170 615 y(INTEGER)g(SOURCE,)g(TAG,)h (COMM,)f(STATUS\(MPI)p 1058 615 V 16 w(STATUS)p 1218 615 V 16 w(SIZE\),)g(IERROR)166 701 y Fv(MPI)p 251 701 14 2 v 16 w(PROBE)17 b FA(b)q(eha)o(v)o(es)g(lik)o(e)g Fv(MPI)p 769 701 V 16 w(IPROBE)g FA(except)g(that)f(it)g(is)h(a)f(blo)q(c)o(king)i(call)f (whic)o(h)g(returns)75 757 y(only)f(after)e(a)h(matc)o(hing)g(message)g(has)g (b)q(een)h(found.)166 814 y(The)10 b(MPI)g(implemen)o(tation)i(of)e Fv(MPI)p 805 814 V 15 w(PROBE)i FA(and)e Fv(MPI)p 1145 814 V 16 w(IPROBE)h FA(needs)g(to)e(guaran)o(tee)h(progress:)75 870 y(if)17 b(a)g(call)h(to)e Fv(MPI)p 385 870 V 16 w(PROBE)i FA(has)f(b)q(een)h(issued)g(b)o(y)f(a)f(pro)q(cess,)i(and)f(a)f(send)i(that)e (matc)o(hes)h(the)f(prob)q(e)75 927 y(has)j(b)q(een)i(initiated)g(b)o(y)f (some)f(pro)q(cess,)i(then)f(the)f(call)i(to)e Fv(MPI)p 1244 927 V 16 w(PROBE)i FA(will)g(return,)f(unless)h(the)75 983 y(message)c(is)i(receiv)o(ed)g(b)o(y)f(another)f(concurren)o(t)h(receiv)o(e)h (op)q(eration)f(\(that)e(is)j(executed)g(b)o(y)e(another)75 1040 y(thread)e(at)g(the)h(probing)g(pro)q(cess\).)21 b(Similarly)l(,)d(if)e (a)f(pro)q(cess)h(busy)g(w)o(aits)f(with)h Fv(MPI)p 1573 1040 V 16 w(IPROBE)g FA(and)g(a)75 1096 y(matc)o(hing)e(message)g(has)g(b)q(een)h (issued,)g(then)f(the)g(call)h(to)f Fv(MPI)p 1179 1096 V 16 w(IPROBE)g FA(will)i(ev)o(en)o(tually)f(return)f Fv(\015ag)75 1153 y(=)i(true)f FA(unless)i(the)e(message)g(is)g(receiv)o(ed)i(b)o(y)e (another)g(concurren)o(t)g(receiv)o(e)h(op)q(eration.)75 1256 y Fv(MPI)p 160 1256 V 16 w(CANCEL\(request\))117 1334 y Fu(IN)155 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))75 1458 y Ft(int)23 b(MPI)p 245 1458 15 2 v 17 w(Cancel\(MPI)p 502 1458 V 16 w(Comm)p 614 1458 V 17 w(request)g(*request\))75 1544 y(MPI)p 150 1544 V 17 w(CANCEL\(REQUEST,)f(IERROR\))170 1601 y(INTEGER)h(REQUEST,)g(IERROR)166 1687 y FA(A)17 b(call)h(to)e Fv(MPI)p 444 1687 14 2 v 15 w(CANCEL)i FA(marks)e(for)g(cancellation)i(a)f(p) q(ending)h(non)o(blo)q(c)o(king)g(comm)o(unication)75 1744 y(op)q(eration)f(\(send)g(or)g(receiv)o(e\).)25 b(The)18 b(cancel)g(call)g (is)f(non)o(blo)q(c)o(king,)h(and)f(lo)q(cal.)27 b(It)17 b(returns)f(immedi-) 75 1800 y(ately)l(,)f(p)q(ossibly)i(b)q(efore)e(the)h(comm)o(unication)g(is)f (actually)h(canceled.)166 1857 y(It)c(is)g(still)h(necessary)f(to)g(complete) g(a)g(comm)o(unication)g(that)f(has)h(b)q(een)h(mark)o(ed)e(for)h (cancellation,)75 1913 y(using)i(a)g(call)h(to)e Fv(MPI)p 450 1913 V 15 w(W)l(AIT)h FA(or)f Fv(MPI)p 728 1913 V 16 w(TEST)h FA(\(or)f(an)o(y)g(of)g(the)h(deriv)o(ed)h(op)q(erations\).)k(If)14 b(the)g(op)q(eration)75 1970 y(has)e(b)q(een)h(canceled,)h(then)f (information)f(to)g(that)f(e\013ect)h(will)i(b)q(e)f(returned)f(in)h(the)g (status)e(argumen)o(t)g(of)75 2026 y(the)j(op)q(eration)g(that)f(completes)h (the)g(comm)o(unication.)20 b(If)14 b(a)g(comm)o(unication)g(is)g(mark)o(ed)g (for)f(cancel-)75 2083 y(lation,)j(then)h(a)e Fv(MPI)p 445 2083 V 16 w(W)l(AIT)h FA(call)h(for)f(that)f(comm)o(unication)h(is)h(guaran)o (teed)e(to)h(return,)f(irresp)q(ectiv)o(e)75 2139 y(of)i(the)g(activities)i (of)e(other)g(pro)q(cesses)h(\(i.e.,)f Fv(MPI)p 970 2139 V 16 w(W)l(AIT)g FA(b)q(eha)o(v)o(es)h(as)f(a)g(lo)q(cal)i(function\);)f (similarly)75 2195 y(if)g Fv(MPI)p 204 2195 V 16 w(TEST)f FA(is)h(rep)q (eatedly)g(called)h(in)f(a)f(busy)h(w)o(ait)e(lo)q(op)i(for)f(a)g(canceled)i (comm)o(unication,)f(then)75 2252 y Fv(MPI)p 160 2252 V 16 w(TEST)d FA(will)i(ev)o(en)o(tually)f(b)q(e)g(successful.)166 2308 y(Either)j(the)f(cancellation)i(succeeds,)f(or)f(the)g(comm)o(unication) h(succeeds,)h(but)e(not)g(b)q(oth.)29 b(If)18 b(a)75 2365 y(send)h(is)h(mark) o(ed)e(for)g(cancellation,)j(then)e(it)g(m)o(ust)f(b)q(e)i(the)f(case)g(that) f(either)h(the)g(send)g(completes)75 2421 y(normally)l(,)f(in)h(whic)o(h)f (case)f(the)h(message)f(sen)o(t)g(w)o(as)g(receiv)o(ed)h(at)f(the)h (destination)g(pro)q(cess,)g(or)f(that)75 2478 y(the)f(send)g(is)g (successfully)i(canceled,)f(in)g(whic)o(h)f(case)g(no)g(part)f(of)g(the)h (message)f(w)o(as)g(receiv)o(ed)i(at)e(the)75 2534 y(destination.)21 b(Then,)15 b(an)o(y)f(matc)o(hing)h(receiv)o(e)h(has)e(to)h(b)q(e)g (satis\014ed)h(b)o(y)e(another)h(send.)20 b(If)15 b(a)g(receiv)o(e)g(is)75 2591 y(mark)o(ed)d(for)f(cancellation,)k(then)d(it)h(m)o(ust)e(b)q(e)i(the)g (case)f(that)f(either)i(the)g(receiv)o(e)g(completes)g(normally)l(,)75 2647 y(or)18 b(that)g(the)g(receiv)o(e)i(is)f(successfully)h(canceled,)h(in)e (whic)o(h)h(case)e(no)h(part)f(of)g(the)g(receiv)o(e)i(bu\013er)e(is)75 2704 y(altered.)i(Then,)c(an)o(y)e(matc)o(hing)i(send)g(has)f(to)f(b)q(e)i (satis\014ed)g(b)o(y)f(another)g(receiv)o(e.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 37 37 bop 75 -100 a Fw(1.10.)34 b(PERSISTENT)16 b(COMMUNICA)l(TION)g(REQUESTS)638 b FA(37)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(TEST)p 290 49 V 16 w(CANCELLED\(status,)17 b(\015ag\))117 143 y Fu(IN)155 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))117 250 y(OUT)108 b Fv(\015ag)518 b Fu(\(logical\))75 391 y Ft(int)23 b(MPI)p 245 391 15 2 v 17 w(Test)p 358 391 V 17 w(cancelled\(MPI)p 687 391 V 15 w(Status)g(status,)g(int)h(*flag\))75 494 y(MPI)p 150 494 V 17 w(TEST)p 263 494 V 16 w(CANCELLED\(STATUS,)e(FLAG,)h(IERROR\)) 170 550 y(LOGICAL)g(FLAG)170 607 y(INTEGER)g(STATUS\(MPI)p 604 607 V 16 w(STATUS)p 764 607 V 17 w(SIZE\),)g(IERROR)166 709 y FA(Returns)12 b Fv(\015ag)f(=)h(true)g FA(if)f(the)h(comm)o(unication)g (asso)q(ciated)f(with)h(the)g(status)e(ob)s(ject)h(w)o(as)f(canceled)75 766 y(successfully)l(.)27 b(In)18 b(suc)o(h)f(a)g(case,)g(all)g(other)g (\014elds)h(of)e Fv(status)j FA(\(suc)o(h)e(as)g Fv(count)h FA(or)e Fv(tag)p FA(\))h(are)f(unde\014ned.)75 822 y(Returns)i Fv(\015ag)f(=)h(false)p FA(,)f(otherwise.)27 b(If)18 b(a)f(receiv)o(e)h(op)q (eration)g(migh)o(t)f(b)q(e)h(canceled)h(then)e(one)h(should)75 879 y(call)41 b Fv(MPI)p 268 879 14 2 v 16 w(TEST)p 398 879 V 17 w(CANCELLED)20 b FA(\014rst,)h(to)e(c)o(hec)o(k)i(whether)f(the)g(op)q (eration)h(w)o(as)e(canceled,)k(b)q(efore)75 935 y(c)o(hec)o(king)16 b(on)f(the)g(other)g(\014elds)i(of)d(the)i(return)f(status.)166 1090 y Fq(Implemen)o(tati)o(on)h(note:)51 b Fu(If)18 b(a)f(send)h(op)q (eration)g(uses)h(an)e(\\eager")h(proto)q(col)f(\(data)h(is)f(transferred)75 1147 y(to)e(the)h(receiv)o(er)h(b)q(efore)f(a)g(matc)o(hing)d(receiv)o(e)k (is)e(p)q(osted\),)i(then)f(the)g(cancelation)f(of)g(this)g(send)i(ma)o(y)c (require)75 1203 y(comm)o(unication)j(with)j(the)h(in)o(tended)g(receiv)o (er,)h(in)e(order)h(to)f(free)h(allo)q(cated)f(bu\013ers.)36 b(On)19 b(some)g(systems)75 1260 y(this)14 b(ma)o(y)e(require)j(an)f(in)o (terrupt)h(to)e(the)i(in)o(tended)g(receiv)o(er.)20 b(Therefore,)15 b(it)e(is)h(exp)q(ected)i(that)e Fo(MPI)p 1705 1260 13 2 v 15 w(CANCEL)75 1316 y Fu(will)h(b)q(e)i(an)f(exp)q(ensiv)o(e)i(op)q(eration)e (on)g(man)o(y)f(systems,)i(to)f(b)q(e)h(used)g(only)f(exceptionally)m(.)25 b(Note)17 b(that,)f(while)75 1373 y(comm)o(unication)f(ma)o(y)h(b)q(e)i (needed)i(to)e(implemen)o(t)d Fo(MPI)p 1006 1373 V 14 w(CANCEL)p Fu(,)i(this)h(is)g(still)f(a)h(lo)q(cal)f(op)q(eration,)i(in)e(the)75 1429 y(sense)h(that)e(its)g(completion)e(do)q(es)j(not)f(dep)q(end)h(on)f (the)g(co)q(de)h(executed)h(b)o(y)e(other)g(pro)q(cesses.)27 b(If)16 b(pro)q(cessing)75 1486 y(is)d(required)i(on)e(another)h(pro)q(cess,) h(this)e(should)g(b)q(e)i(transparen)o(t)f(to)f(the)i(application)d(\(hence)j (the)f(need)g(for)f(an)75 1542 y(in)o(terrupt)i(and)e(an)h(in)o(terrupt)h (handler\).)75 1861 y FB(1.10)60 b(P)n(ersistent)21 b(comm)o(unication)c (requests)75 1994 y FA(Often)22 b(a)g(comm)o(unication)h(with)f(the)g(same)g (argumen)o(t)f(list)i(is)g(rep)q(eatedly)g(executed)g(within)g(the)75 2050 y(inner)c(lo)q(op)g(of)e(a)h(parallel)i(computation.)28 b(In)18 b(suc)o(h)h(a)f(situation,)g(it)h(ma)o(y)e(b)q(e)i(p)q(ossible)g(to)f (optimize)75 2107 y(the)h(comm)o(unication)g(b)o(y)g(binding)h(the)f(list)g (of)g(comm)o(unication)g(argumen)o(ts)f(to)g(a)g(comm)o(unication)75 2163 y(request)11 b(once)g(and,)g(then,)h(rep)q(eatedly)g(using)g(the)f(comm) o(unication)g(request)g(to)f(initiate)i(and)f(complete)75 2219 y(messages.)19 b(The)d(comm)o(unication)f(request)h(th)o(us)f(created)g(can)g (b)q(e)h(though)o(t)e(of)h(as)f(a)h(comm)o(unication)75 2276 y(p)q(ort)c(or)f(a)h(\\half-c)o(hannel")h(.)18 b(It)12 b(do)q(es)f(not)f(pro) o(vide)i(the)f(full)h(functionalit)o(y)h(of)d(a)h(con)o(v)o(en)o(tional)g(c)o (hannel,)75 2332 y(since)i(there)g(is)g(no)f(binding)i(of)e(the)g(send)h(p)q (ort)f(to)f(the)i(receiv)o(e)g(p)q(ort:)18 b(this)13 b(construct)f(allo)o(ws) g(reduction)75 2389 y(of)18 b(the)g(o)o(v)o(erhead)f(for)h(comm)o(unication)g (b)q(et)o(w)o(een)h(pro)q(cessor)e(and)h(comm)o(unication)h(con)o(troller,)g (but)75 2445 y(not)13 b(of)g(the)g(o)o(v)o(erhead)g(for)g(comm)o(unication)h (b)q(et)o(w)o(een)g(one)f(comm)o(unication)h(con)o(troller)g(and)g(another.) 166 2518 y(A)j(comm)o(unication)g(request)g(is)g(created)g(using)h(one)f(of)f (the)h(four)f(follo)o(wing)i(calls.)26 b(These)17 b(calls)75 2574 y(in)o(v)o(olv)o(e)f(no)f(comm)o(unication.)166 2647 y(The)10 b(function)i Fv(MPI)p 513 2647 14 2 v 15 w(CREA)l(TE)p 697 2647 V 18 w(SEND)f FA(creates)f(a)g(comm)o(unication)h(request)f(for)g(a)g (standard)g(mo)q(de)75 2704 y(send)16 b(op)q(eration,)f(and)g(binds)i(to)d (it)i(all)g(the)f(argumen)o(ts)f(of)h(a)g(send)h(op)q(eration.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 38 38 bop 75 -100 a FA(38)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(CREA)l(TE)p 345 49 V 17 w(SEND\(buf,)g(count,)g(datat)o(yp)q(e,)h (dest,)f(tag,)f(comm)m(,)d(request\))117 140 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\)) 117 242 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (sen)o(t)i(\(in)o(teger\))117 345 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 447 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 549 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 652 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 754 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 892 y Ft(int)23 b(MPI)p 245 892 15 2 v 17 w(Create)p 406 892 V 17 w(send\(void*)f(buf,)h(int)h(count,)f(MPI)p 1139 892 V 17 w(Datatype)f(datatype,)h(int)h(dest,)393 948 y(int)g(tag,)f(MPI)p 683 948 V 17 w(Comm)g(comm,)g(MPI)p 1034 948 V 17 w(Comm)p 1147 948 V 17 w(request)g(*request\))75 1048 y(MPI)p 150 1048 V 17 w(CREATE)p 311 1048 V 16 w(SEND\(BUF,)g(COUNT,)g(DATATYPE,)f(DEST,)i (TAG,)f(COMM,)g(REQUEST,)g(IERROR\))170 1105 y()g(BUF\(*\))170 1161 y(INTEGER)g(REQUEST,)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)g(COMM,)g (REQUEST,)g(IERROR)166 1261 y FA(The)16 b(function)h Fv(MPI)p 524 1261 14 2 v 16 w(CREA)l(TE)p 709 1261 V 17 w(RSEND)g FA(creates)e(a)h (comm)o(unication)h(ob)s(ject)e(for)g(a)h(ready)g(mo)q(de)75 1318 y(send)g(op)q(eration.)75 1435 y Fv(MPI)p 160 1435 V 16 w(CREA)l(TE)p 345 1435 V 17 w(RSEND\(buf,)g(count,)g(datat)o(yp)q(e,)h(dest,) f(tag,)f(comm)m(,)d(request\))117 1526 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 1628 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (sen)o(t)i(\(in)o(teger\))117 1730 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 1833 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\)) 117 1935 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 2037 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 2140 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 2278 y Ft(int)23 b(MPI)p 245 2278 15 2 v 17 w(Create)p 406 2278 V 17 w(rsend\(void*)f(buf,)h(int)h(count,)f(MPI)p 1163 2278 V 17 w(Datatype)f(datatype,)h(int)g(dest,)393 2334 y(int)h(tag,)f(MPI)p 683 2334 V 17 w(Comm)g(comm,)g(MPI)p 1034 2334 V 17 w(Comm)p 1147 2334 V 17 w(request)g(*request\))75 2434 y(MPI)p 150 2434 V 17 w(CREATE)p 311 2434 V 16 w(RSEND\(BUF,)g(COUNT,)g (DATATYPE,)f(DEST,)h(TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 2491 y()g(BUF\(*\))170 2547 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (TAG,)h(COMM,)f(REQUEST,)f(IERROR)166 2647 y FA(The)15 b(function)i Fv(MPI)p 523 2647 14 2 v 15 w(CREA)l(TE)p 707 2647 V 18 w(SSEND)f FA(creates)f(a)g(comm)o(unication)h(ob)s(ject)e(for)h(a)g(sync)o(hronous)75 2704 y(mo)q(de)g(send)h(op)q(eration.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 39 39 bop 75 -100 a Fw(1.10.)34 b(PERSISTENT)16 b(COMMUNICA)l(TION)g(REQUESTS)638 b FA(39)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(CREA)l(TE)p 345 49 V 17 w(SSEND\(buf,)16 b(count,)g(datat)o(yp)q(e,)h(dest,)f(tag,)f(comm)m (,)d(request\))117 126 y Fu(IN)155 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o(hoice\))117 202 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o (teger\))117 277 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 352 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\))117 427 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(\(in)o(teger\))117 502 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 577 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 702 y Ft(int)23 b(MPI)p 245 702 15 2 v 17 w(Create)p 406 702 V 17 w(ssend\(void*)f(buf,)h(int)h(count,)f(MPI)p 1163 702 V 17 w(Datatype)f(datatype,)h(int)g(dest,)393 758 y(int)h(tag,)f(MPI)p 683 758 V 17 w(Comm)g(comm,)g(MPI)p 1034 758 V 17 w(Comm)p 1147 758 V 17 w(request)g(*request\))75 845 y(MPI)p 150 845 V 17 w(CREATE)p 311 845 V 16 w(SSEND\(BUF,)g(COUNT,)g(DATATYPE,)f(DEST,)h (TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 901 y()g(BUF\(*\))170 958 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g(TAG,)h(COMM,)f(REQUEST,)f (IERROR)166 1044 y FA(The)15 b(function)h Fv(MPI)p 522 1044 14 2 v 15 w(CREA)l(TE)p 706 1044 V 18 w(RECV)g FA(creates)e(a)h(comm)o (unication)g(ob)s(ject)f(for)h(a)f(receiv)o(e)i(op)q(era-)75 1100 y(tion.)75 1204 y Fv(MPI)p 160 1204 V 16 w(CREA)l(TE)p 345 1204 V 17 w(RECV\(buf,)g(count,)h(datat)o(yp)q(e,)f(source,)f(tag,)g (comm)m(,)d(request\))117 1281 y Fu(OUT)108 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 1356 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f (receiv)o(ed)j(\(in)o(teger\))117 1432 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(eac)o(h)h(elemen)o(t)g(\(handle\))117 1507 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(source)j(or)d(MPI)p 1309 1507 13 2 v 16 w(ANY)p 1418 1507 V 15 w(SOUR)o(CE)h(\(in)o(teger\))117 1582 y(IN)155 b Fv(tag)526 b Fu(message)14 b(tag)f(or)h(MPI)p 1270 1582 V 15 w(ANY)p 1378 1582 V 16 w(T)m(A)o(G)f(\(in)o(teger\))117 1657 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 1732 y(OUT)108 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k (\(handle\))75 1857 y Ft(int)23 b(MPI)p 245 1857 15 2 v 17 w(Create)p 406 1857 V 17 w(recv\(void*)f(buf,)h(int)h(count,)f(MPI)p 1139 1857 V 17 w(Datatype)f(datatype,)h(int)h(source,)393 1913 y(int)g(tag,)f(MPI)p 683 1913 V 17 w(Comm)g(comm,)g(MPI)p 1034 1913 V 17 w(Comm)p 1147 1913 V 17 w(request)g(*request\))75 1999 y(MPI)p 150 1999 V 17 w(CREATE)p 311 1999 V 16 w(RECV\(BUF,)g(COUNT,)g (DATATYPE,)f(SOURCE,)h(TAG,)h(COMM,)f(REQUEST,)g(IERROR\))170 2056 y()g(BUF\(*\))170 2112 y(INTEGER)g(COUNT,)g(DATATYPE,)g(SOURCE,)g (TAG,)g(COMM,)g(REQUEST,)g(IERROR)166 2199 y FA(A)18 b(comm)o(unication)g (\(send)g(or)g(receiv)o(e\))g(that)f(uses)h(a)g(prede\014ned)h(request)f(is)h (initiated)g(b)o(y)f(the)75 2255 y(function)e Fv(MPI)p 338 2255 14 2 v 16 w(ST)l(ART)p FA(.)75 2359 y Fv(MPI)p 160 2359 V 16 w(ST)l(ART\(request\))117 2436 y Fu(INOUT)62 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))75 2561 y Ft(int)23 b(MPI)p 245 2561 15 2 v 17 w(Start\(MPI)p 478 2561 V 16 w(Comm)p 590 2561 V 17 w(request)g(*request\))75 2647 y(MPI)p 150 2647 V 17 w(START\(REQUEST,)f(IERROR\))170 2704 y(INTEGER)h(REQUEST,)g(IERROR)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 40 40 bop 75 -100 a FA(40)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(The)11 b(comm)o(unication)h(ob)s (ject)f(asso)q(ciated)g(with)h Fv(request)h FA(should)f(b)q(e)g(an)f(ob)s (ject)g(that)g(w)o(as)f(created)75 106 y(b)o(y)k(one)g(of)g(the)g(previous)h (four)e(functions,)i(so)f(that)f(all)i(the)f(comm)o(unication)h(argumen)o(ts) e(are)h(already)75 162 y(de\014ned.)23 b(A)16 b(send)g(can)g(b)q(e)h(started) e(pro)o(vided)h(that)f(the)h(previous)h(send)f(using)h(the)e(same)h(ob)s (ject)f(has)75 219 y(completed,)d(or)f(as)f(so)q(on)h(as)f(the)h(ob)s(ject)f (is)i(created,)f(if)g(it)g(has)g(not)g(y)o(et)f(b)q(een)i(used)f(in)h(a)f (comm)o(unication.)75 275 y(In)k(addition,)h(if)f(the)f(comm)o(unication)i (mo)q(de)e(is)h Fv(ready)g FA(then)g(a)f(matc)o(hing)h(receiv)o(e)g(should)h (b)q(e)f(p)q(osted.)75 332 y(The)j(send)h(bu\013er)f(should)g(not)g(b)q(e)h (accessed)f(after)f(the)h(send)h(is)f(started,)g(and)g(un)o(til)h(the)f(op)q (eration)75 388 y(completes.)166 446 y(A)c(receiv)o(e)g(can)g(b)q(e)g (started)f(pro)o(vided)i(that)e(the)h(preceding)h(receiv)o(e)f(using)h(the)f (same)f(ob)s(ject)g(has)75 502 y(completed,)f(or)f(as)f(so)q(on)h(as)f(the)h (ob)s(ject)f(is)i(created,)f(if)g(it)g(has)g(not)g(y)o(et)f(b)q(een)i(used)f (in)h(a)f(comm)o(unication.)75 559 y(The)18 b(receiv)o(e)h(bu\013er)f(should) h(not)e(b)q(e)i(accessed)g(after)e(the)h(receiv)o(e)h(is)f(started,)g(un)o (til)h(the)f(op)q(eration)75 615 y(completes.)166 673 y(The)23 b(call)g(is)g(non)o(blo)q(c)o(king,)i(with)e(similar)h(seman)o(tics)e(to)g (the)g(non)o(blo)q(c)o(king)i(comm)o(unication)75 729 y(op)q(erations)e (describ)q(ed)i(in)e(section)g(1.8.)39 b(I.e.,)23 b(a)e(call)i(to)e Fv(MPI)p 1218 729 14 2 v 16 w(ST)l(ART)h FA(starts)f(a)g(comm)o(unication)75 786 y(in)g(the)e(same)h(manner)g(as)f(a)g(call)i(to)e Fv(MPI)p 838 786 V 16 w(ISEND)p FA(,)h Fv(MPI)p 1099 786 V 15 w(IRSEND)p FA(,)g Fv(MPI)p 1388 786 V 16 w(ISSEND)g FA(or)f Fv(MPI)p 1721 786 V 16 w(IRECV)p FA(,)75 842 y(resp)q(ectiv)o(ely)l(.)166 982 y Fq(Discussion:)34 b Fu(It)14 b(has)g(b)q(een)h(suggested)g(that)f Fo(MPI)p 1005 982 13 2 v 14 w(IST)m(ART)f Fu(w)o(ould)g(mak)o(e)g(this)h(p)q (oin)o(t)f(b)q(etter)75 1170 y Fv(MPI)p 160 1170 14 2 v 16 w(ST)l(ART)l(ALL\(count,)j(a)o(rra)o(y)p 641 1170 V 15 w(of)p 693 1170 V 16 w(requests\))117 1249 y Fu(IN)155 b Fv(count)482 b Fu(list)14 b(length)f(\(in)o(teger\))117 1326 y(INOUT)62 b Fv(a)o(rra)o(y)p 416 1326 V 15 w(of)p 468 1326 V 16 w(requests)272 b Fu(arra)o(y)14 b(of)f(requests)j(\(arra)o(y)e(of)f(handle\))75 1452 y Ft(int)23 b(MPI)p 245 1452 15 2 v 17 w(Startall\(int)f(count,)h(MPI)p 811 1452 V 17 w(Comm)p 924 1452 V 17 w(request)g(*array)p 1276 1452 V 16 w(of)p 1340 1452 V 17 w(requests\))75 1540 y(MPI)p 150 1540 V 17 w(STARTALL\(COUNT,)f(ARRAY)p 669 1540 V 16 w(OF)p 733 1540 V 17 w(REQUESTS,)h(IERROR\))170 1596 y(INTEGER)g(COUNT,)g(ARRAY)p 651 1596 V 17 w(OF)p 716 1596 V 17 w(REQUESTS\(*\),)f(IERROR)166 1684 y FA(Start)i(all)j(comm)o(unications)f(asso)q(ciated)g(with)f(requests)h (in)g Fv(a)o(rra)o(y)p 1399 1684 14 2 v 15 w(of)p 1451 1684 V 16 w(requests)p FA(.)52 b(A)26 b(call)h(to)75 1740 y Fv(MPI)p 160 1740 V 16 w(ST)l(ART)l(ALL\(count,)76 b(a)o(rra)o(y)p 701 1740 V 14 w(of)p 752 1740 V 16 w(requests\))65 b FA(has)d(the)h(same)f (e\013ect)h(as)f(calls)i(to)75 1797 y Fv(MPI)p 160 1797 V 16 w(ST)l(ART\(a)o(rra)o(y)p 431 1797 V 15 w(of)p 483 1797 V 16 w(requests[i]\))p FA(,)17 b(executed)f(for)e Fv(i=0)i(,...,)d(count-1)p FA(,)j(in)g(some)f(arbitrary)f(order.)166 1854 y(A)j(comm)o(unication)g (started)f(with)h(a)g(call)h(to)e Fv(MPI)p 1063 1854 V 16 w(ST)l(ART)h FA(or)f Fv(MPI)p 1377 1854 V 16 w(ST)l(ART)l(ALL)i FA(is)f(completed)75 1911 y(b)o(y)11 b(a)h(call)g(to)f Fv(MPI)p 385 1911 V 16 w(W)l(AIT)p FA(,)g Fv(MPI)p 620 1911 V 16 w(TEST)p FA(,)h(or)f(one)g(of)g(the)h(deriv)o (ed)h(functions)f(describ)q(ed)h(in)g(section)f(1.8.5.)75 1967 y(These)k(comm)o(unication)h(completion)g(functions)g(do)f(not)f(deallo)q (cate)j(the)e(request,)g(and)g(this)g(can)g(b)q(e)75 2024 y(reused)g(anew)g (b)o(y)f(a)h Fv(MPI)p 521 2024 V 15 w(ST)l(ART)h FA(call.)22 b(The)16 b(ob)s(ject)f(needs)h(to)f(b)q(e)i(explicitly)h(deallo)q(cated)f(b)o (y)f(a)f(call)75 2080 y(to)j(the)g(function)i Fv(MPI)p 482 2080 V 16 w(REQUEST)p 701 2080 V 17 w(FREE)p FA(,)f(b)q(elo)o(w.)30 b(W)l(e)19 b(th)o(us)f(ha)o(v)o(e)g(t)o(w)o(o)g(t)o(yp)q(es)g(of)g(comm)o (unication)75 2137 y(requests:)34 b Fz(p)q(ersisten)o(t)23 b FA(requests,)h(whic)o(h)f(are)f(allo)q(cated)i(b)o(y)e(a)g(call)i(to)e Fv(MPI)p 1503 2137 V 15 w(CREA)l(TE)p 1687 2137 V 18 w(xxx)p FA(,)i(and)75 2193 y(are)18 b(explicitly)j(deallo)q(cated)e(b)o(y)f Fv(MPI)p 746 2193 V 16 w(REQUEST)p 965 2193 V 18 w(FREE)p FA(,)g(and)g Fz(ephemeral)g FA(requests)g(that)f(p)q(ersist)75 2249 y(for)f(one)i(comm)o (unication)g(only:)24 b(they)17 b(are)g(created)g(b)o(y)g(a)g(non)o(blo)q(c)o (king)h(comm)o(unication)g(initiation)75 2306 y(function,)e(and)f(are)g (freed)g(b)o(y)h(the)f(comm)o(unication)h(completion)g(call.)166 2364 y(If)40 b(a)20 b(call)h(to)e Fv(MPI)p 512 2364 V 16 w(W)l(AIT)p FA(,)g Fv(MPI)p 755 2364 V 16 w(TEST)p FA(,)h(or)f(an)o(y)h(of)f(the)h(deriv) o(ed)h(functions,)h(is)e(executed)h(on)-1903 b Fk(\))75 2420 y FA(a)22 b(p)q(ersisten)o(t)h(comm)o(unication)g(request)f(for)g(whic)o(h)h (there)g(is)g(no)f(p)q(ending)i(comm)o(unication)f(\(e.g.)75 2477 y(immediately)13 b(after)e(it)h(w)o(as)f(created)h(or)f(immediately)i (after)e(a)h(call)g(to)f Fv(MPI)p 1378 2477 V 16 w(W)l(AIT)p FA(\))h(then)g(the)f(outcome)75 2533 y(is)18 b(similar)h(to)e(that)g(whic)o (h)i(obtains)f(when)g(a)f(n)o(ull)i(request)f(w)o(as)f(passed)h(as)f(argumen) o(t:)24 b Fv(MPI)p 1749 2533 V 16 w(W)l(AIT)75 2589 y FA(returns)13 b(immediately)l(,)i Fv(MPI)p 581 2589 V 16 w(TEST)e FA(returns)g(with)g Fo(\015ag)f(=)g(true)p FA(,)i(etc.)19 b(As)13 b(for)g(calls)h(with)f(n)o(ull) h(requests,)75 2646 y(the)h(status)g(is)g(up)q(dated)h(to)f(the)g(status)g (of)f(a)h(n)o(ull)i(comm)o(unication.)166 2704 y(In)g(particular,)g(one)f (can)g(pass)g(to)g Fv(MPI)p 858 2704 V 16 w(W)l(AIT)l(ANY)h FA(and)f(the)h(other)f(functions)h(in)g(section)g(1.8.5)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 41 41 bop 75 -100 a Fw(1.11.)34 b(SEND-RECEIVE)1277 b FA(41)75 49 y(an)23 b Fv(a)o(rra)o(y)p 244 49 14 2 v 14 w(of)p 295 49 V 16 w(request)i FA(argumen)o(t)c(where)i(some)g(of)f(the)h(requests)g(are)f(p) q(ersisten)o(t)h(ones,)h(and)f(ha)o(v)o(e)75 106 y(the)c(desired)h(outcome:) 28 b(the)19 b(call)h(will)h(ignore)e(those)g(requests)g(in)h(the)f(arra)o(y)f (for)g(whic)o(h)i(there)f(are)75 162 y(no)e(p)q(ending)i(comm)o(unications)e (\(in)h(the)f(same)g(manner)g(that)f(these)h(functions)h(ignore)f(n)o(ull)i (request)75 219 y(en)o(tries\).)h(Th)o(us,)15 b(the)g(last)g(example)h(on)f (page)g(29)g(w)o(orks)f(correctly)h(with)h(p)q(ersisten)o(t)g(requests,)e(to) q(o.)75 335 y Fv(MPI)p 160 335 V 16 w(REQUEST)p 379 335 V 18 w(FREE\(request\))117 425 y Fu(INOUT)62 b Fv(request)452 b Fu(comm)o(unication)11 b(request)k(\(handle\))75 562 y Ft(int)23 b(MPI)p 245 562 15 2 v 17 w(Request)p 430 562 V 16 w(free\(MPI)p 638 562 V 17 w(Comm)p 751 562 V 16 w(request)g(*request\))75 661 y(MPI)p 150 661 V 17 w(REQUEST)p 335 661 V 16 w(FREE\(REQUEST,)f (IERROR\))170 718 y(INTEGER)h(REQUEST,)g(IERROR)166 817 y FA(Mark)c Fv(request)i FA(for)e(deallo)q(cation.)35 b(The)20 b(request)f(ob)s(ject)h (is)g(deallo)q(cated)h(when)f(there)g(are)f(no)75 874 y(p)q(ending)e(comm)o (unications)f(in)o(v)o(olving)g(this)g(ob)s(ject,)e(at)h(whic)o(h)h(p)q(oin)o (t)f(the)h(request)f(b)q(ecomes)h(n)o(ull.)166 943 y(The)j(call)g(is)g(non)o (blo)q(c)o(king.)31 b(It)18 b(is)h(p)q(ermissible)i(to)d(call)h Fv(MPI)p 1249 943 14 2 v 16 w(REQUEST)p 1468 943 V 18 w(FREE\(request\))h FA(after)75 999 y(a)f(comm)o(unication)g(that)f(uses)h Fv(request)i FA(has)e(b)q(een)h(initiated,)h(but)e(b)q(efore)g(it)g(has)g(completed.)32 b(The)75 1056 y(ob)s(ject)15 b(will)j(b)q(e)f(deallo)q(cated)g(after)e(the)h (comm)o(unication)h(completes.)22 b(It)16 b(is)h(erroneous)f(to)f(initiate)i (a)75 1112 y(comm)o(unication)f(with)f(a)g(request)h(that)e(has)h(b)q(een)h (freed.)166 1181 y(A)f(correct)f(in)o(v)o(o)q(cation)i(of)e(the)h(functions)h (describ)q(ed)g(in)g(this)f(section)h(will)g(o)q(ccur)g(in)f(a)g(sequence)75 1238 y(of)g(the)g(form)166 1307 y Fz(Create)i FA(\()p Fz(Start)h(Complete)p FA(\))725 1290 y Fh(\003)761 1307 y Fz(F)l(ree)f Fi(;)166 1376 y FA(where)f Fl(\003)g FA(indicates)i(zero)e(or)f(more)h(rep)q(etitions.)24 b(If)16 b(the)g(same)g(comm)o(unication)h(ob)s(ject)f(is)g(used)75 1433 y(in)k(sev)o(eral)f(concurren)o(t)g(threads,)g(it)g(is)h(the)f(user)g (resp)q(onsibilit)o(y)i(to)e(co)q(ordinate)g(calls)h(so)f(that)f(the)75 1489 y(correct)d(sequence)h(is)g(ob)q(ey)o(ed.)166 1558 y(A)c(send)g(op)q (eration)f(initiated)j(with)d Fv(MPI)p 874 1558 V 16 w(ST)l(ART)i FA(can)e(b)q(e)i(matc)o(hed)e(with)h(an)o(y)f(receiv)o(e)i(op)q(eration)75 1615 y(and,)g(lik)o(ewise,)h(a)e(receiv)o(e)i(op)q(eration)e(initiated)i (with)f Fv(MPI)p 1098 1615 V 16 w(ST)l(ART)g FA(can)g(receiv)o(e)g(messages)f (generated)75 1671 y(b)o(y)j(an)o(y)g(send)h(op)q(eration.)75 1887 y FB(1.11)60 b(Send-receive)75 2013 y FA(The)18 b Fz(send-receiv)o(e)g FA(op)q(erations)g(com)o(bine)h(in)g(one)f(call)h(the)f(sending)i(of)d(a)h (message)g(to)f(one)h(desti-)75 2070 y(nation)e(and)g(the)g(receiving)i(of)d (another)h(message,)f(from)g(another)h(destination,)g(p)q(ossibly)i(the)e (same.)75 2126 y(A)k(send-receiv)o(e)j(op)q(eration)d(is)h(v)o(ery)f(useful)i (for)d(executing)j(a)e(shift)h(op)q(eration)f(across)g(a)g(c)o(hain)h(of)75 2183 y(pro)q(cesses.)37 b(If)20 b(blo)q(c)o(king)i(sends)g(and)e(receiv)o(es) i(are)e(used)h(for)f(suc)o(h)h(shift,)h(then)f(one)g(needs)g(to)f(or-)75 2239 y(der)c(the)h(sends)f(and)g(receiv)o(es)h(correctly)g(\(e.g.,)d(ev)o(en) j(pro)q(cesses)f(send,)h(next)f(receiv)o(e,)h(o)q(dd)g(pro)q(cesses)75 2296 y(receiv)o(e)j(\014rst,)g(next)f(send\))g(so)g(as)g(to)g(prev)o(en)o(t)g (cyclic)i(dep)q(endencies)h(that)c(ma)o(y)h(lead)h(to)e(deadlo)q(c)o(k.)75 2352 y(When)g(a)g(send-receiv)o(e)h(op)q(eration)f(is)g(used,)h(the)f(comm)o (unication)g(subsystem)g(tak)o(es)f(care)g(of)h(these)75 2409 y(issues.)j(The)14 b(send-receiv)o(e)j(op)q(eration)e(can)f(b)q(e)i(used)f (in)g(conjunction)h(with)f(the)g(functions)g(describ)q(ed)75 2465 y(in)h(Chapter)f Fz(??)k FA(in)d(order)f(to)f(p)q(erform)h(shifts)g(on)g (v)m(arious)h(logical)g(top)q(ologies.)k(Also,)c(a)e(send-receiv)o(e)75 2521 y(op)q(eration)h(is)h(useful)g(for)f(implemen)o(ting)i(remote)e(pro)q (cedure)h(calls.)166 2591 y(A)29 b(message)14 b(sen)o(t)h(b)o(y)f(a)h (send-receiv)o(e)h(op)q(eration)f(can)f(b)q(e)i(receiv)o(ed)g(b)o(y)e(a)g (regular)h(receiv)o(e)h(op)q(er-)69 b Fk(\()75 2647 y FA(ation)16 b(or)f(prob)q(ed)i(b)o(y)f(a)g(prob)q(e)g(op)q(eration;)g(a)g(send-receiv)o (e)i(op)q(eration)e(can)g(receiv)o(e)h(a)f(message)f(sen)o(t)75 2704 y(b)o(y)g(a)g(regular)g(send)h(op)q(eration.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 42 42 bop 75 -100 a FA(42)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(SENDRECV\(sendbuf,)24 b(sendcount,)f(sendt)o(yp)q(e,)g(dest,)f (sendtag,)g(recvbuf,)g(recvcount,)f(recvt)o(yp)q(e,)75 106 y(source,)15 b(recvtag,)g(comm)m(,)d(status\))117 183 y Fu(IN)155 b Fv(sendbuf)444 b Fu(initial)12 b(address)j(of)f(send)g(bu\013er)h(\(c)o (hoice\))117 254 y(IN)155 b Fv(sendcount)399 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(send)h(bu\013er)f(\(in)o(teger\))117 325 y(IN)155 b Fv(sendt)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h (in)f(send)i(bu\013er)g(\(handle\))117 397 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\))117 468 y(IN)155 b Fv(sendtag)443 b Fu(send)15 b(tag)e(\(in)o(teger\))117 539 y(OUT)108 b Fv(recvbuf)450 b Fu(initial)12 b(address)j(of)f(receiv)o(e)h (bu\013er)g(\(c)o(hoice\))117 610 y(IN)155 b Fv(recvcount)405 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(receiv)o(e)h(bu\013er)g (\(in)o(teger\))117 682 y(IN)155 b Fv(recvt)o(yp)q(e)430 b Fu(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(receiv)o(e)i(bu\013er)g (\(handle\))117 753 y(IN)155 b Fv(source)468 b Fu(rank)14 b(of)f(source)j (\(in)o(teger\))117 824 y(IN)155 b Fv(recvtag)449 b Fu(receiv)o(e)15 b(tag)f(\(in)o(teger\))117 895 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 966 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 1091 y Ft(int)23 b(MPI)p 245 1091 15 2 v 17 w(Sendrecv\(void)f(*sendbuf,)h(int)g(sendcount,)g(MPI)p 1265 1091 V 17 w(Datatype)f(sendtype,)393 1147 y(int)i(dest,)f(int)g (sendtag,)g(void)g(*recvbuf,)g(int)g(recvcount,)393 1204 y(MPI)p 468 1204 V 17 w(Datatype)g(recvtype,)f(int)i(source,)f(MPI)p 1297 1204 V 16 w(Datatype)g(recvtag,)393 1260 y(MPI)p 468 1260 V 17 w(Comm)g(comm,)g(MPI)p 819 1260 V 17 w(Status)g(*status\))75 1347 y(MPI)p 150 1347 V 17 w(SENDRECV\(SENDBUF,)e(SENDCOUNT,)i(SENDTYPE,)f (DEST,)i(SENDTAG,)e(RECVBUF,)393 1403 y(RECVCOUNT,)h(RECVTYPE,)f(SOURCE,)h (RECVTAG,)g(COMM,)g(STATUS,)g(IERROR\))170 1460 y()g(SENDBUF\(*\),)g (RECVBUF\(*\))170 1516 y(INTEGER)g(SENDCOUNT,)g(SENDTYPE,)f(DEST,)i(SENDTAG,) e(RECVCOUNT,)h(RECVTYPE,)170 1573 y(SOURCE,)g(RECV)h(TAG,)f(COMM,)g (STATUS\(MPI)p 986 1573 V 16 w(STATUS)p 1146 1573 V 16 w(SIZE\),)g(IERROR)166 1659 y FA(Execute)c(a)e(blo)q(c)o(king)j(send)f(and)f(receiv)o(e)h(op)q (eration.)29 b(Both)18 b(send)h(and)f(receiv)o(e)h(use)g(the)f(same)75 1715 y(comm)o(unicator,)c(but)g(p)q(ossibly)i(di\013eren)o(t)f(tags.)j(The)d (send)g(bu\013er)f(and)h(receiv)o(e)g(bu\013ers)g(are)f(disjoin)o(t,)75 1772 y(and)h(ma)o(y)g(ha)o(v)o(e)g(di\013eren)o(t)g(lengths)h(and)f(datat)o (yp)q(es.)75 1876 y Fv(MPI)p 160 1876 14 2 v 16 w(SENDRECV)p 408 1876 V 17 w(REPLA)o(CE\(buf,)k(count,)f(datat)o(yp)q(e,)h(dest,)f (sendtag,)h(source,)f(recvtag,)f(comm)m(,)d(sta-)75 1932 y(tus\))117 2009 y Fu(INOUT)62 b Fv(buf)527 b Fu(initial)12 b(address)j(of)f(send)g(and)g (receiv)o(e)h(bu\013er)g(\(c)o(hoice\))117 2081 y(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)9 b(of)g(elemen)o(ts)h(in)f(send)i(and)e (receiv)o(e)i(bu\013er)g(\(in)o(teger\))117 2152 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(t)o(yp)q(e)15 b(of)e(elemen)o(ts)h(in)f(send)i (and)f(receiv)o(e)h(bu\013er)g(\(handle\))117 2223 y(IN)155 b Fv(dest)511 b Fu(rank)14 b(of)f(destination)h(\(in)o(teger\))117 2294 y(IN)155 b Fv(sendtag)443 b Fu(send)15 b(message)f(tag)f(\(in)o(teger\)) 117 2365 y(IN)155 b Fv(source)468 b Fu(rank)14 b(of)f(source)j(\(in)o (teger\))117 2437 y(IN)155 b Fv(recvtag)449 b Fu(receiv)o(e)15 b(message)f(tag)f(\(in)o(teger\))117 2508 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(\(handle\))117 2579 y(OUT)108 b Fv(status)476 b Fu(status)15 b(ob)r(ject)g(\(Status\))75 2704 y Ft(int)23 b(MPI)p 245 2704 15 2 v 17 w(Sendrecv)p 454 2704 V 16 w(replace\(void*)f (buf,)i(int)f(count,)g(MPI)p 1258 2704 V 17 w(Datatype)g(datatype,)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 43 43 bop 75 -100 a Fw(1.12.)34 b(NULL)16 b(PR)o(OCESSES)1218 b FA(43)393 49 y Ft(int)24 b(dest,)f(int)g(sendtag,)g(int)g(source,)g(int)h(recvtag,)f (MPI)p 1614 49 15 2 v 16 w(Comm)h(comm,)393 106 y(MPI)p 468 106 V 17 w(Status)f(*status\))75 192 y(MPI)p 150 192 V 17 w(SENDRECV)p 359 192 V 16 w(REPLACE\(BUF,)f(COUNT,)h(DATATYPE,)g(DEST,)g(SENDTAG,)g (SOURCE,)g(RECVTAG,)393 249 y(COMM,)g(STATUS,)g(IERROR\))170 305 y()g(BUF\(*\))170 362 y(INTEGER)g(COUNT,)g(DATATYPE,)g(DEST,)g (SENDTAG,)g(SOURCE,)g(RECVTAG,)g(COMM,)170 418 y(STATUS\(MPI)p 413 418 V 16 w(STATUS)p 573 418 V 17 w(SIZE\),)g(IERROR)166 504 y FA(Execute)15 b(a)f(blo)q(c)o(king)i(send)f(and)g(receiv)o(e;)g(the)g (same)f(bu\013er)g(is)h(used)h(b)q(oth)e(for)g(the)h(send)g(and)g(for)75 561 y(the)g(receiv)o(e,)h(so)f(that)f(the)i(message)e(sen)o(t)h(is)h (replaced)g(b)o(y)g(the)f(message)g(receiv)o(ed.)166 617 y(The)f(seman)o (tics)g(of)f(a)h(send-receiv)o(e)h(op)q(eration)f(is)g(what)g(w)o(ould)g (obtain)g(if)g(the)g(caller)h(fork)o(ed)e(t)o(w)o(o)75 674 y(concurren)o(t)19 b(threads,)f(one)h(to)f(execute)h(the)g(send,)g(and)g(one) g(to)e(execute)j(the)e(receiv)o(e,)i(follo)o(w)o(ed)f(b)o(y)75 730 y(a)e(join)g(of)g(these)g(t)o(w)o(o)f(threads.)25 b(Additional)19 b(in)o(termediate)f(bu\013ering)g(is)f(needed)i(for)d(the)i(\\replace")75 787 y(v)m(arian)o(t.)166 843 y(A)11 b(send-receiv)o(e)i(cannot)f(b)q(e)g (implemen)o(ted)h(b)o(y)e(a)g(blo)q(c)o(king)i(send)f(follo)o(w)o(ed)g(b)o(y) f(a)g(blo)q(c)o(king)i(receiv)o(e)75 900 y(or)i(a)g(blo)q(c)o(king)i(receiv)o (e,)f(follo)o(w)o(ed)g(b)o(y)g(a)f(blo)q(c)o(king)i(send.)22 b(Consider)16 b(the)f(follo)o(wing)i(co)q(de,)e(where)h(t)o(w)o(o)75 956 y(pro)q(cesses)g(exc)o(hange)f(messages:)75 1050 y Ft(CALL)23 b(MPI_COMM_RANK\(comm,)f(rank,)h(ierr\))75 1107 y(IF)h(\(rank.EQ.0\))123 1163 y(THEN)170 1220 y(CALL)g(MPI_SENDRECV\(send_buff,)c(send_count,)j (MPI_REAL,)f(1,)i(send_tag,)242 1276 y(recv_buff,)f(recv_count,)f(MPI_REAL,)h (1,)g(recv_tag,)g(comm,)g(status,)g(ierr\))123 1333 y(ELSE)170 1389 y(CALL)h(MPI_SENDRECV\(send_buff,)c(send_count,)j(MPI_REAL,)f(0,)i (send_tag,)242 1446 y(recv_buff,)f(recv_count,)f(MPI_REAL,)h(0,)g(tag,)g (comm,)h(status,)f(ierr\))75 1502 y(END)g(IF)166 1596 y FA(If)16 b(the)f(send)h(receiv)o(es)h(are)e(replaced)i(either)f(b)o(y)f(blo)q(c)o (king)i(send,)f(follo)o(w)o(ed)g(b)o(y)g(blo)q(c)o(king)g(receiv)o(e,)75 1653 y(or)g(blo)q(c)o(king)h(receiv)o(e,)g(follo)o(w)o(ed)g(b)o(y)f(blo)q(c)o (king)i(send,)f(then)f(the)h(co)q(de)g(ma)o(y)e(deadlo)q(c)o(k.)24 b(On)17 b(the)f(other)75 1709 y(hand,)f(send-receiv)o(e)i(can)f(b)q(e)f (implemen)o(ted)i(using)f(non)o(blo)q(c)o(king)h(sends)e(and)h(receiv)o(es.) 75 1851 y FB(1.12)60 b(Null)19 b(p)n(ro)r(cesses)75 1952 y FA(In)c(man)o(y)e(instances,)i(it)f(is)h(con)o(v)o(enien)o(t)g(to)e(sp)q (ecify)j(a)d(\\dumm)o(y")h(source)g(or)g(destination)h(for)e(comm)o(u-)75 2009 y(nication.)29 b(This)18 b(simpli\014es)i(the)e(co)q(de)g(that)f(is)h (needed)i(for)d(dealing)i(with)f(b)q(oundaries,)h(e.g.,)e(in)i(the)75 2065 y(case)c(of)g(a)g(non-circular)i(shift)e(done)h(with)f(calls)i(to)d (send-receiv)o(e.)166 2122 y(The)f(sp)q(ecial)j(v)m(alue)26 b Fo(MPI)p 608 2122 13 2 v 15 w(PROCNULL)12 b FA(can)i(b)q(e)g(used)g (instead)g(of)f(a)g(rank)g(wherev)o(er)g(a)g(source)g(or)g(a)75 2178 y(destination)j(argumen)o(t)f(is)g(required)i(in)f(a)f(call.)21 b(A)15 b(comm)o(unication)h(with)g(pro)q(cess)29 b Fo(MPI)p 1649 2178 V 15 w(PROCNULL)75 2235 y FA(has)13 b(no)h(e\013ect:)k(a)13 b(send)h(to)25 b Fo(MPI)p 633 2235 V 14 w(PROCNULL)13 b FA(succeeds)i(and)e (returns)g(as)g(so)q(on)h(as)f(p)q(ossible.)21 b(A)13 b(receiv)o(e)75 2291 y(from)26 b Fo(MPI)p 271 2291 V 14 w(PROCNULL)13 b FA(succeeds)i(and)f (returns)g(as)f(so)q(on)h(as)f(p)q(ossible)j(with)e(no)g(mo)q(di\014cations)h (to)e(the)75 2348 y(receiv)o(e)18 b(bu\013er.)25 b(When)17 b(a)g(receiv)o(e)h(with)f Fv(source)h FA(=)32 b Fo(MPI)p 1090 2348 V 15 w(PROCNULL)16 b FA(is)i(executed)f(then)h(the)f(status)75 2404 y(ob)s(ject)e(returns)30 b Fv(source)16 b FA(=)29 b Fo(MPI)p 661 2404 V 14 w(PROCNULL)p FA(,)14 b Fv(tag)h FA(=)30 b Fo(MPI)p 1132 2404 V 14 w(ANY)p 1231 2404 V 15 w(T)m(A)o(G)14 b FA(and)h Fv(count=0)p FA(.)75 2546 y FB(1.13)60 b(Derived)19 b(datat)n(yp)r(es)75 2647 y FA(Up)c(to)f(no)o(w,)g(all)i(p)q(oin)o(t)f(to)f(p)q(oin)o(t)h(comm)o (unication)h(in)o(v)o(olv)o(ed)f(only)g(con)o(tiguous)g(bu\013ers)g(con)o (taining)g(a)75 2704 y(sequence)g(of)e(elemen)o(ts)h(of)f(the)g(same)g(t)o (yp)q(e.)19 b(This)14 b(is)g(to)q(o)f(constraining)h(on)f(t)o(w)o(o)f(accoun) o(ts:)19 b(One)14 b(often)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 44 44 bop 75 -100 a FA(44)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y FA(w)o(an)o(ts)d(to)h(pass)g (messages)g(that)f(con)o(tain)i(v)m(alues)g(with)g(di\013eren)o(t)f(datat)o (yp)q(es)g(\(e.g.,)f(an)h(in)o(teger)h(coun)o(t,)75 106 y(follo)o(w)o(ed)i(b) o(y)f(a)g(sequence)i(of)e(real)h(n)o(um)o(b)q(ers\);)f(and)h(one)g(often)f(w) o(an)o(ts)f(to)h(send)h(noncon)o(tiguous)g(data)75 162 y(\(e.g.)h(a)12 b(sub-blo)q(c)o(k)i(of)e(a)g(matrix\).)18 b(One)13 b(solution)g(is)g(to)f (pro)o(vide)h(functions)g(that)e(pac)o(k)i(noncon)o(tiguous)75 219 y(data)f(in)o(to)h(a)g(con)o(tiguous)g(bu\013er)g(at)f(the)h(sender)h (site)f(and)g(unpac)o(k)h(it)f(bac)o(k)g(at)f(the)h(receiv)o(er)h(site.)19 b(This)75 275 y(has)f(the)g(disadv)m(an)o(tage)g(of)g(requiring)h(additional) h(memory-to-memory)c(cop)o(y)i(op)q(erations)g(at)g(b)q(oth)75 332 y(sites,)12 b(ev)o(en)f(when)g(the)g(comm)o(unication)g(subsystem)g(has)f (scatter-gather)g(capabilities.)20 b(Instead,)12 b(MPI)75 388 y(pro)o(vides)22 b(mec)o(hanisms)f(to)g(sp)q(ecify)h(more)f(general,)i(mixed) f(and)f(noncon)o(tiguous)g(comm)o(unication)75 444 y(bu\013ers.)29 b(It)19 b(is)g(up)g(to)f(the)g(implemen)o(tation)i(to)e(decide)i(whether)e (data)g(should)h(b)q(e)h(\014rst)e(pac)o(k)o(ed)g(in)75 501 y(a)e(con)o(tiguous)h(bu\013er)g(b)q(efore)g(b)q(eing)h(transmitted,)f(or)f (whether)h(it)g(can)g(b)q(e)g(collected)i(directly)f(from)75 557 y(where)d(it)h(resides.)166 614 y(The)e(general)h(mec)o(hanisms)g(pro)o (vided)g(here)f(allo)o(w)h(one)f(to)g(transfer)f(directly)l(,)j(without)e (cop)o(ying,)75 670 y(ob)s(jects)j(of)g(v)m(arious)h(shap)q(e)h(and)f(size.) 28 b(It)17 b(is)i(not)e(assumed)h(that)f(the)g(MPI)h(library)g(is)h(cognizan) o(t)e(of)75 727 y(the)f(ob)s(jects)g(declared)h(in)g(the)f(host)g(language;)g (th)o(us,)g(if)g(one)g(w)o(an)o(ts)f(to)h(transfer)f(a)h(structure,)f(or)h (an)75 783 y(arra)o(y)d(section,)i(it)f(will)i(b)q(e)f(necessary)f(to)g(pro)o (vide)h(in)g(MPI)f(a)g(de\014nition)i(of)e(a)g(comm)o(unication)h(bu\013er)75 840 y(that)e(mimics)j(the)e(de\014nition)i(of)e(the)g(structure)g(or)f(arra)o (y)g(section)i(in)g(question.)20 b(These)15 b(facilities)h(can)75 896 y(b)q(e)j(used)g(b)o(y)f(library)h(designers)h(to)d(de\014ne)j(comm)o (unication)f(functions)g(that)f(can)g(transfer)g(ob)s(jects)75 953 y(de\014ned)g(in)f(the)f(host)g(language)g({)g(b)o(y)h(deco)q(ding)g (their)g(de\014nitions)h(as)e(a)o(v)m(ailable)i(in)f(a)f(sym)o(b)q(ol)h (table)75 1009 y(or)e(a)g(dop)q(e)g(v)o(ector.)20 b(Suc)o(h)15 b(higher-lev)o(el)j(comm)o(unication)e(functions)g(are)f(not)g(part)f(of)h (MPI.)166 1065 y(More)d(general)g(comm)o(unication)i(bu\013ers)e(are)g(sp)q (eci\014ed)i(b)o(y)f(replacing)g(the)g(basic)g(datat)o(yp)q(es)f(that)75 1122 y(ha)o(v)o(e)e(b)q(een)i(used)f(so)f(far)g(with)h(deriv)o(ed)g(datat)o (yp)q(es)f(that)g(are)g(constructed)g(from)g(basic)h(datat)o(yp)q(es)f(using) 75 1178 y(the)j(constructors)e(describ)q(ed)j(in)g(this)f(section.)19 b(These)13 b(metho)q(ds)f(of)g(constructing)h(deriv)o(ed)h(datat)o(yp)q(es)75 1235 y(can)h(b)q(e)h(applied)h(recursiv)o(ely)l(.)166 1291 y(A)e Fz(general)j(datat)o(yp)q(e)e FA(is)g(an)f(opaque)h(ob)s(ject)e(that)h (sp)q(eci\014es)i(t)o(w)o(o)d(things:)143 1384 y Fl(\017)23 b FA(A)15 b(sequence)h(of)f(basic)h(datat)o(yp)q(es)143 1478 y Fl(\017)23 b FA(A)15 b(sequence)h(of)f(in)o(teger)g(\(b)o(yte\))g (displacemen)o(ts)166 1571 y(The)22 b(displacemen)o(ts)g(are)g(not)f (required)h(to)f(b)q(e)h(p)q(ositiv)o(e,)i(distinct,)g(or)d(in)h(increasing)h (order;)75 1628 y(therefore)17 b(the)g(order)g(of)f(items)i(need)g(not)f (coincide)i(with)e(their)h(order)f(in)g(store,)g(and)g(an)g(item)h(ma)o(y)75 1684 y(app)q(ear)c(more)g(than)g(once.)20 b(W)l(e)14 b(call)h(suc)o(h)g(pair) f(of)g(sequences)h(\(or)e(sequence)j(of)d(pairs\))h(a)g Fz(t)o(yp)q(e)i(map)p FA(.)75 1740 y(The)22 b(sequence)g(of)f(basic)h(datat)o(yp)q(es)f (\(displacemen)o(ts)h(ignored\))g(is)f(the)h Fz(t)o(yp)q(e)i(signature)e FA(of)f(the)75 1797 y(datat)o(yp)q(e.)166 1853 y(Let)476 1910 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 1917 y Fj(0)867 1910 y Fi(;)8 b(disp)972 1917 y Fj(0)990 1910 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 1917 y Fg(n)p Fh(\000)p Fj(1)1253 1910 y Fi(;)h(disp)1358 1917 y Fg(n)p Fh(\000)p Fj(1)1421 1910 y FA(\))p Fl(g)p Fi(;)75 1993 y FA(b)q(e)16 b(suc)o(h)f(a)g(t)o(yp)q(e)h(map,)e(where)i Fi(ty)r(pe)709 2000 y Fg(i)738 1993 y FA(are)f(basic)h(t)o(yp)q(es,)f(and)g Fi(disp)1233 2000 y Fg(i)1261 1993 y FA(are)g(displacemen)o(ts.)22 b(Let)676 2094 y Fi(T)6 b(y)r(pesig)14 b FA(=)f Fl(f)p Fi(ty)r(pe)1005 2101 y Fj(0)1023 2094 y Fi(;)8 b(:::;)g(ty)r(pe)1188 2101 y Fg(n)p Fh(\000)p Fj(1)1251 2094 y Fl(g)75 2195 y FA(b)q(e)14 b(the)f(asso)q(ciated)h(t)o(yp)q(e)f(signature.)19 b(This)14 b(t)o(yp)q(e)f(map,)g(together)g(with)g(a)g(base)g(address)h Fr(buf)p FA(,)f(sp)q(eci\014es)75 2252 y(a)19 b(comm)o(unication)i(bu\013er:) 28 b(the)20 b(comm)o(unication)h(bu\013er)e(that)g(consists)h(of)f Fi(n)h FA(en)o(tries,)h(where)f(the)75 2308 y Fi(i)p FA(-th)e(en)o(try)g(is)h (at)e(address)h Fi(buf)f FA(+)c Fi(disp)778 2315 y Fg(i)809 2308 y FA(and)19 b(has)f(t)o(yp)q(e)g Fi(ty)r(pe)1174 2315 y Fg(i)1188 2308 y FA(.)29 b(A)18 b(message)g(assem)o(bled)h(from)f(suc)o(h) 75 2365 y(comm)o(unication)e(bu\013er)f(will)i(consist)e(of)g Fi(n)h FA(v)m(alues,)g(of)e(the)i(t)o(yp)q(es)f(de\014ned)h(b)o(y)g Fi(T)6 b(y)r(pesig)r FA(.)166 2421 y(W)l(e)11 b(can)f(use)h(a)f(handle)i(to)e (a)g(general)h(datat)o(yp)q(e)f(as)h(an)f(argumen)o(t)g(in)h(a)f(send)i(or)e (receiv)o(e)h(op)q(eration,)75 2478 y(in)j(replacemen)o(t)f(of)f(a)h(basic)g (datat)o(yp)q(e)g(argumen)o(t.)18 b(The)13 b(op)q(eration)g Fv(MPI)p 1359 2478 14 2 v 16 w(SEND\(buf,)h(1,)e(datat)o(yp)q(e,...\))75 2534 y FA(will)j(use)e(the)h(send)g(bu\013er)f(de\014ned)h(b)o(y)g(the)f (base)g(address)h Fv(buf)g FA(and)f(the)g(general)h(datat)o(yp)q(e)f(asso)q (ciated)75 2591 y(with)f Fv(datat)o(yp)q(e)p FA(;)i(it)e(will)i(generate)d(a) h(message)f(with)h(the)g(t)o(yp)q(e)g(signature)g(determined)h(b)o(y)e(the)h Fv(datat)o(yp)q(e)75 2647 y FA(argumen)o(t.)28 b Fv(MPI)p 390 2647 V 16 w(RECV\(buf,)20 b(1,)e(datat)o(yp)q(e,...\))29 b FA(will)20 b(use)e(the)h(receiv)o(e)g(bu\013er)f(de\014ned)h(b)o(y)g(the)f (base)75 2704 y(address)d Fv(buf)h FA(and)g(the)f(general)h(datat)o(yp)q(e)e (asso)q(ciated)i(with)f Fv(datat)o(yp)q(e)p FA(.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 45 45 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(45)166 49 y(General)16 b(datat)o(yp)q(es)e(can)i(b)q(e)g(used)f(in)h (all)h(send)e(and)h(receiv)o(e)g(op)q(erations.)k(W)l(e)c(address)f(later)g (in)75 106 y(section)h(1.13.5)d(the)i(case)h(where)f(the)g(second)h(argumen)o (t)e Fv(count)j FA(has)e(v)m(alue)h Fi(>)d FA(1.)166 162 y(The)e(basic)g (datat)o(yp)q(es)f(presen)o(ted)i(in)f(section)g(1.2.1)f(are)g(particular)h (cases)g(of)f(a)h(general)g(datat)o(yp)q(e,)75 219 y(and)19 b(are)f(prede\014ned.)32 b(Th)o(us,)19 b Fv(MPI)p 711 219 14 2 v 16 w(INT)f FA(is)h(a)g(prede\014ned)h(handle)g(to)e(a)g(datat)o(yp)q(e)g (with)h(t)o(yp)q(e)g(map)75 275 y Fl(f)p FA(\()p Fv(int)p Fi(;)8 b FA(0\))p Fl(g)p FA(,)j(with)i(one)g(en)o(try)f(of)g(t)o(yp)q(e)h Fv(int)g FA(and)f(displacemen)o(t)i(zero.)19 b(And)13 b(similarly)i(for)d (all)h(other)f(basic)75 332 y(datat)o(yp)q(es.)166 388 y(The)j Fz(exten)o(t)f FA(of)g(a)g(datat)o(yp)q(e)g(is)h(de\014ned)g(to)f(b)q(e)h (the)g(span)f(from)g(the)g(\014rst)g(b)o(yte)h(to)e(the)i(last)f(b)o(yte)75 444 y(o)q(ccupied)19 b(b)o(y)d(en)o(tries)h(in)h(this)f(datat)o(yp)q(e,)f (rounded)h(up)g(to)f(satisfy)h(alignmen)o(t)g(requiremen)o(ts.)25 b(That)75 501 y(is,)15 b(if)476 557 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 564 y Fj(0)867 557 y Fi(;)8 b(disp)972 564 y Fj(0)990 557 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 564 y Fg(n)p Fh(\000)p Fj(1)1253 557 y Fi(;)h(disp)1358 564 y Fg(n)p Fh(\000)p Fj(1)1421 557 y FA(\))p Fl(g)p Fi(;)75 635 y FA(then)501 726 y Fi(l)q(b)p FA(\()p Fi(T)e(y)r(pemap)p FA(\))39 b(=)j(min)906 752 y Fg(j)959 726 y Fi(disp)1043 733 y Fg(j)1060 726 y Fi(;)489 812 y(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))40 b(=)i(max)911 839 y Fg(j)961 812 y FA(\()p Fi(disp)1063 819 y Fg(j)1089 812 y FA(+)11 b Fi(siz)r(eof)5 b FA(\()p Fi(ty)r(pe)1367 819 y Fg(j)1385 812 y FA(\)\))p Fi(;)22 b FA(and)406 899 y Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))42 b(=)g Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))j Fl(\000)h Fi(l)q(b)p FA(\()p Fi(T)c(y)r(pemap)p FA(\))i(+)j Fi(\017:)238 b FA(\(1.1\))75 990 y(If)17 b Fi(ty)r(pe)206 997 y Fg(i)237 990 y FA(requires)g(alignmen)o(t) g(to)f(a)h(b)o(yte)f(address)h(that)f(is)h(is)g(a)f(m)o(ultiple)j(of)d Fi(k)1474 997 y Fg(i)1487 990 y FA(,)h(then)g Fi(\017)g FA(is)g(the)g(least) 75 1046 y(nonnegativ)o(e)f(incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))16 b(to)e(the)h(next)h(m)o (ultiple)h(of)d(max)1794 1053 y Fg(i)1815 1046 y Fi(k)1839 1053 y Fg(i)1853 1046 y FA(.)166 1103 y(Example:)23 b(Assume)17 b(that)f Fi(T)6 b(y)r(pe)14 b FA(=)h Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)15 b FA(\(a)h Fv(double)h FA(at)f(displacemen)o(t)i(zero,)75 1159 y(follo)o(w)o(ed)g(b)o(y)g(a)f Fv(cha)o(r)g FA(at)g(displacemen)o(t)j (eigh)o(t\).)27 b(Assume,)18 b(furthermore,)f(that)g(doubles)i(ha)o(v)o(e)e (to)g(b)q(e)75 1216 y(strictly)e(aligned)g(at)f(addresses)h(that)e(are)h(m)o (ultiple)i(of)e(eigh)o(t.)20 b(Then,)15 b(the)f(exten)o(t)g(of)g(this)h (datat)o(yp)q(e)e(is)75 1272 y(16)d(\(9)g(rounded)h(to)f(the)h(next)g(m)o (ultiple)h(of)f(8\).)17 b(A)11 b(datat)o(yp)q(e)f(that)g(consists)h(of)f(a)g (c)o(haracter)g(immediately)75 1329 y(follo)o(w)o(ed)16 b(b)o(y)f(a)g(double) h(will)h(also)e(ha)o(v)o(e)g(an)g(exten)o(t)g(of)g(16.)189 1421 y Fr(R)n(ationale.)39 b FA(The)15 b(de\014nition)h(of)e(exten)o(t)g(is)h (motiv)m(ated)f(b)o(y)h(the)f(assumption)h(that)e(the)i(amoun)o(t)189 1478 y(of)i(padding)i(added)g(at)e(the)h(end)g(of)g(eac)o(h)g(structure)f(in) i(an)f(arra)o(y)e(of)i(structures)f(is)i(the)f(least)189 1534 y(needed)d(to)e(ful\014ll)j(alignmen)o(t)e(constrain)o(ts.)19 b(More)14 b(explicit)i(con)o(trol)d(of)g(the)h(exten)o(t)g(is)g(pro)o(vided) 189 1591 y(in)i(section)g(1.13.3.)k(Suc)o(h)c(explicit)i(con)o(trol)e(is)g (needed)h(in)f(cases)g(where)g(the)g(assumption)g(do)q(es)189 1647 y(not)e(hold,)i(e.g.)j(where)d(union)g(t)o(yp)q(es)f(are)g(used.)21 b(\()p Fr(End)15 b(of)i(r)n(ationale.)p FA(\))75 1767 y Fs(1.13.1)49 b(Datat)o(yp)q(e)17 b(constructo)o(rs)75 1853 y Fv(Contiguous)47 b FA(The)16 b(simplest)i(datat)o(yp)q(e)e(constructor)f(is)i Fv(MPI)p 1171 1853 V 16 w(TYPE)p 1304 1853 V 17 w(CONTIGUOUS)h FA(whic)o(h)f(allo)o(ws)75 1909 y(replication)g(of)e(a)g(datat)o(yp)q(e)f(in) o(to)h(con)o(tiguous)h(lo)q(cations.)75 2013 y Fv(MPI)p 160 2013 V 16 w(TYPE)p 293 2013 V 17 w(CONTIGUOUS\(count,)h(oldt)o(yp)q(e,)f (newt)o(yp)q(e\))117 2090 y Fu(IN)155 b Fv(count)482 b Fu(replication)14 b(coun)o(t)g(\(nonnegativ)o(e)g(in)o(teger\))117 2164 y(IN)155 b Fv(oldt)o(yp)q(e)450 b Fu(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2237 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2361 y Ft(int)23 b(MPI)p 245 2361 15 2 v 17 w(Type)p 358 2361 V 17 w(contiguous\(int)f(count,)h(MPI)p 972 2361 V 17 w(Datatype)f(oldtype,)393 2418 y(MPI)p 468 2418 V 17 w(Datatype)h(*newtype\))75 2504 y(MPI)p 150 2504 V 17 w(TYPE)p 263 2504 V 16 w(CONTIGUOUS\(COUNT,)f(OLDTYPE,)h(NEWTYPE,)f(IERROR\))170 2561 y(INTEGER)h(COUNT,)g(OLDTYPE,)g(NEWTYPE,)g(IERROR)166 2647 y Fv(newt)o(yp)q(e)c FA(is)f(the)f(datat)o(yp)q(e)g(obtained)h(b)o(y)f (concatenating)h Fv(count)h FA(copies)f(of)f Fv(oldt)o(yp)q(e)p FA(.)27 b(Concate-)75 2704 y(nation)15 b(is)h(de\014ned)h(using)f Fr(extent)e FA(as)h(the)h(size)g(of)e(the)i(concatenated)f(copies.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 46 46 bop 75 -100 a FA(46)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fq(Discussion:)55 b Fu(Here,)15 b(and)f(in)g(the)h(other)f(datat)o(yp)q(e)h(constructors,)h (the)f(input)f(datat)o(yp)q(e)g(argumen)o(t)g(of)-1905 b Fn(\))75 106 y Fu(the)16 b(C)g(function)f(is)h(passed)h(b)o(y)e(v)n(alue,)g(and)h(the) g(output)g(datat)o(yp)q(e)g(argumen)o(t)f(is)h(passed)g(b)o(y)g(reference.)26 b(I.e.,)75 162 y(\(...,oldt)o(yp)q(e,)12 b(*newt)o(yp)q(e\).)18 b(It)c(has)g(b)q(een)h(argued)g(that)f(it)f(is)h(more)f(con)o(v)o(enien)o(t)h (to)g(pass)g(b)q(oth)g(via)f(a)h(p)q(oin)o(ter.)166 301 y FA(Example:)31 b(let)21 b Fv(oldt)o(yp)q(e)h FA(ha)o(v)o(e)e(t)o(yp)q(e)h(map)f Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)p Fi(;)18 b FA(with)j(exten)o(t)g(16,)g (and)f(let)75 358 y Fv(count)14 b FA(=)f(3.)19 b(The)d(t)o(yp)q(e)f(map)g(of) g(the)g(datat)o(yp)q(e)g(returned)g(b)o(y)h Fv(newt)o(yp)q(e)h FA(is)308 458 y Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fi(;)g FA(\()p Fv(double)n Fi(;)g FA(16\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(24\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g FA(32\))p Fi(;)g FA(\()p Fv(cha)o(r)m Fi(;)g FA(40\))p Fl(g)p FA(;)75 558 y(i.e.,)15 b(alternating)g Fv(double)i FA(and)e Fv(cha)o(r)g FA(elemen)o(ts,)h(with)f (displacemen)o(ts)i(0)p Fi(;)8 b FA(8)p Fi(;)g FA(16)p Fi(;)f FA(24)p Fi(;)g FA(32)o Fi(;)h FA(4)o(0.)166 615 y(In)16 b(general,)f(assume)g (that)g(the)g(t)o(yp)q(e)g(map)g(of)g Fv(oldt)o(yp)q(e)h FA(is)600 715 y Fl(f)p FA(\()p Fi(ty)r(pe)725 722 y Fj(0)743 715 y Fi(;)8 b(disp)848 722 y Fj(0)865 715 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 722 y Fg(n)p Fh(\000)p Fj(1)1128 715 y Fi(;)g(disp)1233 722 y Fg(n)p Fh(\000)p Fj(1)1297 715 y FA(\))p Fl(g)p Fi(;)75 815 y FA(with)16 b(exten)o(t)e Fi(extent)p FA(.)22 b(Then)15 b Fv(newt)o(yp)q(e)i FA(has)e(a)g(t)o(yp)q(e)h(map)f(with)g Fv(count)c Fl(\001)f Fv(n)16 b FA(en)o(tries)f(de\014ned)i(b)o(y:)75 915 y Fl(f)p FA(\()p Fi(ty)r(pe)200 922 y Fj(0)218 915 y Fi(;)8 b(disp)323 922 y Fj(0)341 915 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)541 922 y Fg(n)p Fh(\000)p Fj(1)604 915 y Fi(;)g(disp)709 922 y Fg(n)p Fh(\000)p Fj(1)773 915 y FA(\))p Fi(;)g FA(\()p Fi(ty)r(pe)914 922 y Fj(0)931 915 y Fi(;)g(disp)1036 922 y Fj(0)1061 915 y FA(+)h Fi(extent)p FA(\))p Fi(;)f(:::;)g FA(\()p Fi(ty)r(pe)1433 922 y Fg(n)p Fh(\000)p Fj(1)1496 915 y Fi(;)g(disp)1601 922 y Fg(n)p Fh(\000)p Fj(1)1673 915 y FA(+)g Fi(extent)p FA(\))p Fi(;)145 1015 y(:::;)g FA(\()p Fi(ty)r(pe)307 1022 y Fj(0)324 1015 y Fi(;)g(disp)429 1022 y Fj(0)456 1015 y FA(+)j Fi(extent)g Fl(\001)f FA(\()p Fv(count)h Fl(\000)g FA(1\)\))p Fi(;)d(:::;)f FA(\()p Fi(ty)r(pe)1084 1022 y Fg(n)p Fh(\000)p Fj(1)1146 1015 y Fi(;)h(disp)1251 1022 y Fg(n)p Fh(\000)p Fj(1)1325 1015 y FA(+)j Fi(extent)g Fl(\001)f FA(\()p Fv(count)h Fl(\000)f FA(1\)\))p Fl(g)p Fi(:)75 1135 y Fv(V)o(ecto)o(r)45 b FA(The)16 b(function)g Fv(MPI)p 600 1135 14 2 v 16 w(TYPE)p 733 1135 V 17 w(VECTOR)g FA(is)g(a)f(more)g(general)h(constructor)e(that)h(allo)o(ws)g(repli-)75 1192 y(cation)20 b(of)g(a)g(datat)o(yp)q(e)f(in)o(to)h(lo)q(cations)h(that)e (consist)i(of)e(equally)j(spaced)f(blo)q(c)o(ks.)35 b(Eac)o(h)20 b(blo)q(c)o(k)g(is)75 1248 y(obtained)g(b)o(y)f(concatenating)h(the)f(same)g (n)o(um)o(b)q(er)h(of)f(copies)h(of)f(the)g(old)h(datat)o(yp)q(e.)32 b(The)19 b(spacing)75 1304 y(b)q(et)o(w)o(een)d(blo)q(c)o(ks)f(is)h(a)f(m)o (ultiple)i(of)e(the)g(exten)o(t)g(of)g(the)g(old)h(datat)o(yp)q(e.)75 1408 y Fv(MPI)p 160 1408 V 16 w(TYPE)p 293 1408 V 17 w(VECTOR\()g(count,)g (blo)q(cklength,)g(stride,)g(oldt)o(yp)q(e,)g(newt)o(yp)q(e\))117 1485 y Fu(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g (\(nonnegativ)o(e)f(in)o(teger\))117 1560 y(IN)155 b Fv(blo)q(cklength)371 b Fu(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o(h)i(blo)q(c)o(k)e (\(nonnegativ)o(e)h(in)o(te-)905 1617 y(ger\))117 1691 y(IN)155 b Fv(stride)484 b Fu(n)o(um)o(b)q(er)13 b(of)f(elemen)o(ts)h(b)q(et)o(w)o (een)h(start)g(of)f(eac)o(h)g(blo)q(c)o(k)g(\(in)o(te-)905 1748 y(ger\))117 1823 y(IN)155 b Fv(oldt)o(yp)q(e)450 b Fu(old)13 b(datat)o(yp)q(e)i(\(handle\))117 1898 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f(\(handle\))75 2022 y Ft(int)23 b(MPI)p 245 2022 15 2 v 17 w(Type)p 358 2022 V 17 w(vector\(int)f(count,)h (int)h(blocklength,)e(int)i(stride,)393 2078 y(MPI)p 468 2078 V 17 w(Datatype)f(oldtype,)f(MPI)p 986 2078 V 17 w(Datatype)h(*newtype\))75 2165 y(MPI)p 150 2165 V 17 w(TYPE)p 263 2165 V 16 w(VECTOR\(COUNT,)f (BLOCKLENGTH,)h(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h(IERROR\))170 2221 y(INTEGER)g(COUNT,)g(BLOCKLENGTH,)g(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h (IERROR)166 2308 y FA(Example:)c(Assume,)12 b(again,)g(that)f Fv(oldt)o(yp)q(e)i FA(has)f(t)o(yp)q(e)g(map)g Fl(f)p FA(\()p Fv(double)p Fi(;)c FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)p Fi(;)i FA(with)i(exten)o(t)75 2364 y(16.)20 b(A)c(call)h(to)d Fv(MPI)p 428 2364 14 2 v 16 w(TYPE)p 561 2364 V 17 w(VECTOR\()i(2,)f(3,)g(4,)g(oldt)o(yp)q(e,)i(newt)o(yp)q(e\))g FA(will)g(create)e(the)h(datat)o(yp)q(e)f(with)75 2421 y(t)o(yp)q(e)g(map)320 2521 y Fl(f)p FA(\()p Fv(double)p Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g FA(16\))p Fi(;)g FA(\()p Fv(cha)o(r)m Fi(;)g FA(24\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g FA(32\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(40\))p Fi(;)279 2621 y FA(\()p Fv(double)q Fi(;)g FA(64\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(72\))p Fi(;)g FA(\()p Fv(double)n Fi(;)g FA(80\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(88\))p Fi(;)g FA(\()p Fv(double)n Fi(;)g FA(96\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(104\))p Fl(g)i FA(:)75 2704 y(t)o(w)o(o)k(blo)q(c)o(ks)i(with)f(three)h(copies)g(eac)o(h)f (of)g(the)g(old)h(t)o(yp)q(e,)f(starting)f(4*16)g(apart.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 47 47 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(47)166 49 y(A)15 b(call)h(to)f Fv(MPI)p 439 49 14 2 v 16 w(TYPE)p 572 49 V 17 w(VECTOR\(3,)g(1,)g(-2,)f(oldt)o(yp)q(e,)i(newt)o(yp) q(e\))h FA(will)g(create)e(the)g(datat)o(yp)q(e)237 154 y Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g Fl(\000)p FA(32\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g Fl(\000)p FA(24\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g Fl(\000)p FA(64\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g Fl(\000)p FA(56\))p Fl(g)p Fi(:)166 260 y FA(In)16 b(general,)f(assume)g(that)g Fv(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)600 365 y Fl(f)p FA(\()p Fi(ty)r(pe)725 372 y Fj(0)743 365 y Fi(;)8 b(disp)848 372 y Fj(0)865 365 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 372 y Fg(n)p Fh(\000)p Fj(1)1128 365 y Fi(;)g(disp)1233 372 y Fg(n)p Fh(\000)p Fj(1)1297 365 y FA(\))p Fl(g)p Fi(;)75 470 y FA(with)15 b(exten)o(t)e Fi(extent)p FA(.)21 b(The)15 b(newly)g(created)f(datat)o(yp)q(e)f(has)i(a)e (t)o(yp)q(e)i(map)f(with)g Fv(count)9 b Fl(\001)f Fv(blo)q(cklength)i Fl(\001)e Fi(n)75 526 y FA(en)o(tries:)611 586 y Fl(f)p FA(\()p Fi(ty)r(pe)736 593 y Fj(0)754 586 y Fi(;)g(disp)859 593 y Fj(0)877 586 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)1077 593 y Fg(n)p Fh(\000)p Fj(1)1140 586 y Fi(;)g(disp)1245 593 y Fg(n)p Fh(\000)p Fj(1)1309 586 y FA(\))p Fi(;)409 675 y FA(\()p Fi(ty)r(pe)511 682 y Fj(0)530 675 y Fi(;)g(disp)635 682 y Fj(0)662 675 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)1036 682 y Fg(n)p Fh(\000)p Fj(1)1099 675 y Fi(;)g(disp)1204 682 y Fg(n)p Fh(\000)p Fj(1)1278 675 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)75 765 y FA(\()p Fi(ty)r(pe)177 772 y Fj(0)196 765 y Fi(;)g(disp)301 772 y Fj(0)328 765 y FA(+)i(\()p Fv(blo)q(cklength)i Fl(\000)e FA(1\))f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1065 772 y Fg(n)p Fh(\000)p Fj(1)1128 765 y Fi(;)g(disp)1233 772 y Fg(n)p Fh(\000)p Fj(1)1307 765 y FA(+)j(\()p Fv(blo)q(cklength)g Fl(\000)f FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)273 854 y FA(\()p Fi(ty)r(pe)375 861 y Fj(0)393 854 y Fi(;)f(disp)498 861 y Fj(0)526 854 y FA(+)i Fv(stride)h Fl(\001)f Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1036 861 y Fg(n)p Fh(\000)p Fj(1)1099 854 y Fi(;)g(disp)1204 861 y Fg(n)p Fh(\000)p Fj(1)1278 854 y FA(+)j Fv(stride)g Fl(\001)f Fi(extent)p FA(\))p Fi(;)e(:::;)455 944 y FA(\()p Fi(ty)r(pe)557 951 y Fj(0)576 944 y Fi(;)g(disp)681 951 y Fj(0)708 944 y FA(+)j(\()p Fv(stride)g FA(+)f Fv(blo)q(cklength)i Fl(\000)f FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)403 1033 y FA(\()p Fi(ty)r(pe)505 1040 y Fg(n)p Fh(\000)p Fj(1)570 1033 y Fi(;)g(disp)675 1040 y Fg(n)p Fh(\000)p Fj(1)749 1033 y FA(+)i(\()p Fv(stride)h FA(+)f Fv(blo)q(cklength)i Fl(\000)f FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(::::;)523 1123 y FA(\()p Fi(ty)r(pe)625 1130 y Fj(0)643 1123 y Fi(;)g(disp)748 1130 y Fj(0)776 1123 y FA(+)i Fv(stride)h Fl(\001)f FA(\()p Fv(count)h Fl(\000)f FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)e(:::;)476 1213 y FA(\()p Fi(ty)r(pe)578 1220 y Fg(n)p Fh(\000)p Fj(1)643 1213 y Fi(;)g(disp)748 1220 y Fg(n)p Fh(\000)p Fj(1)822 1213 y FA(+)j Fv(stride)g Fl(\001)e FA(\()p Fv(count)i Fl(\000)g FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)329 1302 y FA(\()p Fi(ty)r(pe)431 1309 y Fj(0)450 1302 y Fi(;)g(disp)555 1309 y Fj(0)582 1302 y FA(+)j(\()p Fv(stride)f Fl(\001)g FA(\()p Fv(count)h Fl(\000)g FA(1\))e(+)h Fv(blo)q(cklength)j Fl(\000)d FA(1\))f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)307 1392 y FA(\()p Fi(ty)r(pe)409 1399 y Fg(n)p Fh(\000)p Fj(1)474 1392 y Fi(;)g(disp)579 1399 y Fg(n)p Fh(\000)p Fj(1)653 1392 y FA(+)i(\()p Fv(stride)h Fl(\001)e FA(\()p Fv(count)j Fl(\000)e FA(1\))f(+)i Fv(blo)q(cklength)h Fl(\000)e FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fl(g)166 1479 y FA(A)16 b(call)i(to)e Fv(MPI)p 443 1479 V 16 w(TYPE)p 576 1479 V 16 w(CONTIGUOUS\(count,)k(oldt)o(yp)q(e,)d (newt)o(yp)q(e\))h FA(is)f(equiv)m(alen)o(t)h(to)e(a)g(call)i(to)75 1536 y Fv(MPI)p 160 1536 V 16 w(TYPE)p 293 1536 V 17 w(VECTOR\(count,)g(1,)f (1,)f(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))p FA(,)h(or)e(to)g(a)h(call)g(to)f Fv(MPI)p 1495 1536 V 16 w(TYPE)p 1628 1536 V 17 w(VECTOR\(1,)75 1592 y(count,)g(n,)f(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))p FA(,)h Fv(n)e FA(arbitrary)l(.)75 1718 y Fv(Hvecto)o(r)45 b FA(The)19 b(function)h Fv(MPI)p 631 1718 V 16 w(TYPE)p 764 1718 V 17 w(HVECTOR)g FA(is)f(iden)o(tical)i(to)d Fv(MPI)p 1389 1718 V 16 w(TYPE)p 1522 1718 V 17 w(VECTOR)p FA(,)h(except)75 1774 y(that)h Fv(stride)h FA(is)g(giv)o(en)g(in)g(b)o(ytes,)g(rather)f(then)g(in)i (elemen)o(ts.)36 b(The)20 b(use)h(for)f(b)q(oth)g(t)o(yp)q(es)h(of)f(v)o (ector)75 1831 y(constructors)14 b(is)i(illustrated)h(in)f(section)f(1.13.7.) j(\()p Fv(H)d FA(is)h(used)g(for)e(\\heterogeneous"\).)75 1935 y Fv(MPI)p 160 1935 V 16 w(TYPE)p 293 1935 V 17 w(HVECTOR\()i(count,)g(blo)q (cklength,)h(stride,)e(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))117 2013 y Fu(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(blo)q(c)o(ks)g (\(nonnegativ)o(e)f(in)o(teger\))117 2090 y(IN)155 b Fv(blo)q(cklength)371 b Fu(n)o(um)o(b)q(er)16 b(of)h(elemen)o(ts)g(in)f(eac)o(h)i(blo)q(c)o(k)e (\(nonnegativ)o(e)h(in)o(te-)905 2147 y(ger\))117 2224 y(IN)155 b Fv(stride)484 b Fu(n)o(um)o(b)q(er)13 b(of)g(b)o(ytes)h(b)q(et)o(w)o(een)h (start)f(of)f(eac)o(h)h(blo)q(c)o(k)f(\(in)o(teger\))117 2301 y(IN)155 b Fv(oldt)o(yp)q(e)450 b Fu(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2378 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2503 y Ft(int)23 b(MPI)p 245 2503 15 2 v 17 w(Type)p 358 2503 V 17 w(hvector\(int)f(count,)h(int)h(blocklength,)e(MPI)p 1306 2503 V 17 w(Aint)h(stride,)393 2560 y(MPI)p 468 2560 V 17 w(Datatype)g(oldtype,)f(MPI)p 986 2560 V 17 w(Datatype)h(*newtype\))75 2647 y(MPI)p 150 2647 V 17 w(TYPE)p 263 2647 V 16 w(HVECTOR\(COUNT,)f (BLOCKLENGTH,)h(STRIDE,)f(OLDTYPE,)h(NEWTYPE,)g(IERROR\))170 2704 y(INTEGER)g(COUNT,)g(BLOCKLENGTH,)g(STRIDE,)g(OLDTYPE,)f(NEWTYPE,)h (IERROR)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 48 48 bop 75 -100 a FA(48)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(Assume)f(that)g Fv(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)600 155 y Fl(f)p FA(\()p Fi(ty)r(pe)725 162 y Fj(0)743 155 y Fi(;)8 b(disp)848 162 y Fj(0)865 155 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 162 y Fg(n)p Fh(\000)p Fj(1)1128 155 y Fi(;)g(disp)1233 162 y Fg(n)p Fh(\000)p Fj(1)1297 155 y FA(\))p Fl(g)p Fi(;)75 260 y FA(with)15 b(exten)o(t)e Fi(extent)p FA(.)21 b(The)15 b(newly)g(created)f (datat)o(yp)q(e)f(has)i(a)e(t)o(yp)q(e)i(map)f(with)g Fv(count)9 b Fl(\001)f Fv(blo)q(cklength)i Fl(\001)e Fi(n)75 317 y FA(en)o(tries:)611 377 y Fl(f)p FA(\()p Fi(ty)r(pe)736 384 y Fj(0)754 377 y Fi(;)g(disp)859 384 y Fj(0)877 377 y FA(\))p Fi(;)g(:::;)g FA(\()o Fi(ty)r(pe)1077 384 y Fg(n)p Fh(\000)p Fj(1)1140 377 y Fi(;)g(disp)1245 384 y Fg(n)p Fh(\000)p Fj(1)1309 377 y FA(\))p Fi(;)409 467 y FA(\()p Fi(ty)r(pe)511 474 y Fj(0)530 467 y Fi(;)g(disp)635 474 y Fj(0)662 467 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)1036 474 y Fg(n)p Fh(\000)p Fj(1)1099 467 y Fi(;)g(disp)1204 474 y Fg(n)p Fh(\000)p Fj(1)1278 467 y FA(+)j Fi(extent)p FA(\))p Fi(;)d(:::;)75 558 y FA(\()p Fi(ty)r(pe)177 565 y Fj(0)196 558 y Fi(;)g(disp)301 565 y Fj(0)328 558 y FA(+)i(\()p Fv(blo)q(cklength)i Fl(\000)e FA(1\))f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1065 565 y Fg(n)p Fh(\000)p Fj(1)1128 558 y Fi(;)g(disp)1233 565 y Fg(n)p Fh(\000)p Fj(1)1307 558 y FA(+)j(\()p Fv(blo)q(cklength)g Fl(\000)f FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)434 649 y FA(\()p Fi(ty)r(pe)536 656 y Fj(0)555 649 y Fi(;)f(disp)660 656 y Fj(0)687 649 y FA(+)i Fv(stride)q FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1037 656 y Fg(n)p Fh(\000)p Fj(1)1099 649 y Fi(;)g(disp)1204 656 y Fg(n)p Fh(\000)p Fj(1)1278 649 y FA(+)j Fv(stride)q FA(\))p Fi(;)d(:::;)455 740 y FA(\()p Fi(ty)r(pe)557 747 y Fj(0)576 740 y Fi(;)g(disp)681 747 y Fj(0)708 740 y FA(+)j Fv(stride)g FA(+)f(\()p Fv(blo)q(cklength)i Fl(\000)f FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)403 831 y FA(\()p Fi(ty)r(pe)505 838 y Fg(n)p Fh(\000)p Fj(1)570 831 y Fi(;)g(disp)675 838 y Fg(n)p Fh(\000)p Fj(1)749 831 y FA(+)i Fv(stride)h FA(+)g(\()p Fv(blo)q(cklength)g Fl(\000)g FA(1\))e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(::::;)603 921 y FA(\()p Fi(ty)r(pe)705 928 y Fj(0)724 921 y Fi(;)g(disp)829 928 y Fj(0)856 921 y FA(+)j Fv(stride)g Fl(\001)e FA(\()p Fv(count)j Fl(\000)e FA(1\)\))p Fi(;)e(:::;)557 1012 y FA(\()p Fi(ty)r(pe)659 1019 y Fg(n)p Fh(\000)p Fj(1)724 1012 y Fi(;)g(disp)829 1019 y Fg(n)p Fh(\000)p Fj(1)903 1012 y FA(+)i Fv(stride)h Fl(\001)f FA(\()p Fv(count)h Fl(\000)f FA(1\)\))p Fi(;)e(:::;)329 1103 y FA(\()p Fi(ty)r(pe)431 1110 y Fj(0)450 1103 y Fi(;)g(disp)555 1110 y Fj(0)582 1103 y FA(+)j Fv(stride)g Fl(\001)e FA(\()p Fv(count)i Fl(\000)g FA(1\))e(+)i(\()p Fv(blo)q(cklength)h Fl(\000)e FA(1\))f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)307 1194 y FA(\()p Fi(ty)r(pe)409 1201 y Fg(n)p Fh(\000)p Fj(1)474 1194 y Fi(;)g(disp)579 1201 y Fg(n)p Fh(\000)p Fj(1)653 1194 y FA(+)i Fv(stride)h Fl(\001)f FA(\()p Fv(count)h Fl(\000)f FA(1\))g(+)g(\()p Fv(blo)q(cklength)i Fl(\000)e FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fl(g)75 1320 y Fv(Indexed)47 b FA(The)19 b(function)h Fv(MPI)p 629 1320 14 2 v 16 w(TYPE)p 762 1320 V 17 w(INDEXED)f FA(allo)o(ws)h(replication)h(of)d(an)i(old)f(datat)o(yp)q(e) g(in)o(to)g(a)75 1377 y(sequence)c(of)e(con)o(tiguous)h(blo)q(c)o(ks,)g (where)g(eac)o(h)g(blo)q(c)o(k)h(can)f(con)o(tain)g(di\013eren)o(t)g(n)o(um)o (b)q(er)g(of)f(copies)i(and)75 1433 y(ha)o(v)o(e)f(a)g(di\013eren)o(t)g (displacemen)o(t.)21 b(All)16 b(blo)q(c)o(k)f(displacemen)o(ts)g(are)f(m)o (ultiples)i(of)e(the)h(old)f(t)o(yp)q(e)h(exten)o(t.)75 1538 y Fv(MPI)p 160 1538 V 16 w(TYPE)p 293 1538 V 17 w(INDEXED\()d(count,)i(a)o (rra)o(y)p 762 1538 V 14 w(of)p 813 1538 V 16 w(blo)q(cklengths,)h(a)o(rra)o (y)p 1185 1538 V 15 w(of)p 1237 1538 V 16 w(displacements,)e(oldt)o(yp)q(e,)h (newt)o(yp)q(e\))117 1673 y Fu(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)34 b(of)f(blo)q(c)o(ks)h({)g(also)f(n)o(um)o(b)q(er)h(of)f(en)o(tries)i(in)905 1729 y Fo(a)o(rra)o(y)p 992 1729 13 2 v 15 w(of)p 1041 1729 V 15 w(displacements)17 b Fu(and)h Fo(a)o(rra)o(y)p 1483 1729 V 15 w(of)p 1532 1729 V 15 w(blo)q(cklengths)i Fu(\(non-)905 1786 y(negativ)o(e)14 b(in)o(teger\))117 1863 y(IN)155 b Fv(a)o(rra)o(y)p 416 1863 14 2 v 15 w(of)p 468 1863 V 16 w(blo)q(cklengths)191 b Fu(n)o(um)o(b)q(er)18 b(of)f(elemen)o(ts)h(p)q(er)i(blo)q(c)o(k)d(\(arra)o (y)i(of)e(nonnegativ)o(e)905 1919 y(in)o(tegers\))117 1997 y(IN)155 b Fv(a)o(rra)o(y)p 416 1997 V 15 w(of)p 468 1997 V 16 w(displacements)162 b Fu(displacemen)o(t)19 b(for)h(eac)o(h)g(blo)q(c)o (k,)g(in)g(m)o(ultiples)e(of)h Fo(oldt)o(yp)q(e)905 2053 y Fu(exten)o(t)c(\(arra)o(y)f(of)f(in)o(teger\))117 2131 y(IN)155 b Fv(oldt)o(yp)q(e)450 b Fu(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2208 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2334 y Ft(int)23 b(MPI)p 245 2334 15 2 v 17 w(Type)p 358 2334 V 17 w(indexed\(int)f(count,)h(int)h(*array)p 1068 2334 V 16 w(of)p 1132 2334 V 17 w(blocklengths,)393 2390 y(int)g(*array)p 636 2390 V 16 w(of)p 700 2390 V 17 w(displacements,)e(MPI)p 1147 2390 V 17 w(Datatype)g(oldtype,)393 2447 y(MPI)p 468 2447 V 17 w(Datatype)h(*newtype\))75 2534 y(MPI)p 150 2534 V 17 w(TYPE)p 263 2534 V 16 w(INDEXED\(COUNT,)f(ARRAY)p 757 2534 V 17 w(OF)p 822 2534 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1293 2534 V 16 w(OF)p 1357 2534 V 17 w(DISPLACEMENTS,)393 2591 y(OLDTYPE,)h(NEWTYPE,)g (IERROR\))170 2647 y(INTEGER)g(COUNT,)g(ARRAY)p 651 2647 V 17 w(OF)p 716 2647 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 2647 V 17 w(OF)p 1323 2647 V 17 w(DISPLACEMENTS\(*\),)170 2704 y(OLDTYPE,)i(NEWTYPE,)g(IERROR)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 49 49 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(49)166 49 y(Example:)25 b(let)19 b Fv(oldt)o(yp)q(e)f FA(ha)o(v)o(e)g(t)o(yp)q(e)g(map)f Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)p Fi(;)15 b FA(with)j(exten)o(t)g(16.)27 b(Let)18 b Fv(B)g(=)75 106 y(\(3,)13 b(1\))h FA(and)g(let)h Fv(D)f(=)g(\(4,)f(0\))p FA(.)19 b(A)14 b(call)i(to)d Fv(MPI)p 855 106 14 2 v 16 w(TYPE)p 988 106 V 17 w(INDEXED\(2,)g(B,)h(D,)f(oldt)o(yp)q(e,)j(newt)o(yp)q(e\))f FA(returns)75 162 y(a)g(datat)o(yp)q(e)g(with)g(t)o(yp)q(e)g(map)285 251 y Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(64\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(72\))p Fi(;)g FA(\()p Fv(double)n Fi(;)g FA(80\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(88\))p Fi(;)g FA(\()p Fv(double)n Fi(;)g FA(96\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(104\))p Fi(;)760 339 y FA(\()p Fv(double)q Fi(;)g FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)j FA(:)75 416 y(three)k(copies)h(of)e(the)i(old)f(t)o(yp)q(e)g(starting) f(at)h(displacemen)o(t)h(64,)e(and)h(one)h(cop)o(y)e(starting)h(at)f (displace-)75 473 y(men)o(t)h(0.)166 529 y(In)h(general,)f(assume)g(that)g Fv(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)600 618 y Fl(f)p FA(\()p Fi(ty)r(pe)725 625 y Fj(0)743 618 y Fi(;)8 b(disp)848 625 y Fj(0)865 618 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 625 y Fg(n)p Fh(\000)p Fj(1)1128 618 y Fi(;)g(disp)1233 625 y Fg(n)p Fh(\000)p Fj(1)1297 618 y FA(\))p Fl(g)p Fi(;)75 706 y FA(with)35 b(exten)o(t)g Fr(extent)p FA(.)79 b(Let)36 b Fv(B)f FA(b)q(e)h(the)f Fv(a)o(rra)o(y)p 1013 706 V 15 w(of)p 1065 706 V 16 w(blo)q(cklength)i FA(argumen)o(t)d(and)h Fv(D)g FA(b)q(e)h(the)75 763 y Fv(a)o(rra)o(y)p 173 763 V 15 w(of)p 225 763 V 16 w(displacements)15 b FA(argumen)o(t.)k(The)c(newly)h(created)f(datat)o(yp)q(e)g(has)g Fi(n)c Fl(\001)1450 731 y Ff(P)1494 741 y Fe(count)p Fh(\000)p Fj(1)1494 774 y Fg(i)p Fj(=0)1618 763 y Fv(B)p FA([)p Fv(i)p FA(])k(en)o(tries:)284 851 y Fl(f)p FA(\()p Fi(ty)r(pe)409 858 y Fj(0)428 851 y Fi(;)8 b(disp)533 858 y Fj(0)560 851 y FA(+)j Fv(D)p FA([)p Fv(0)p FA(])e Fl(\001)g Fi(extent)p FA(\))p Fi(;)f(:::;)g FA(\()p Fi(ty)r(pe)1047 858 y Fg(n)p Fh(\000)p Fj(1)1111 851 y Fi(;)g(disp)1216 858 y Fg(n)p Fh(\000)p Fj(1)1290 851 y FA(+)i Fv(D)p FA([)p Fv(0)p FA(])f Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)75 940 y FA(\()p Fi(ty)r(pe)177 947 y Fj(0)196 940 y Fi(;)g(disp)301 947 y Fj(0)325 940 y FA(+)f(\()p Fv(D)p FA([)p Fv(0)p FA(])e(+)i Fv(B)p FA([)p Fv(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1037 947 y Fg(n)p Fh(\000)p Fj(1)1099 940 y Fi(;)g(disp)1204 947 y Fg(n)p Fh(\000)p Fj(1)1275 940 y FA(+)f(\()p Fv(D)p FA([)p Fv(0)p FA(])e(+)i Fv(B)p FA([)p Fv(0)p FA(])g Fl(\000)g FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)134 1017 y FA(\()p Fi(ty)r(pe)236 1024 y Fj(0)255 1017 y Fi(;)g(disp)360 1024 y Fj(0)387 1017 y FA(+)i Fv(D)p FA([)p Fv(count)h Fl(\000)g Fv(1)p FA(])e Fl(\001)h Fi(extent)p FA(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1036 1024 y Fg(n)p Fh(\000)p Fj(1)1099 1017 y Fi(;)g(disp)1204 1024 y Fg(n)p Fh(\000)p Fj(1)1278 1017 y FA(+)j Fv(D)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)375 1093 y FA(\()p Fi(ty)r(pe)477 1100 y Fj(0)496 1093 y Fi(;)g(disp)601 1100 y Fj(0)628 1093 y FA(+)i(\()p Fv(D)p FA([)p Fv(count)h Fl(\000)f Fv(1)p FA(])g(+)g Fv(B)p FA([)p Fv(count)i Fl(\000)e Fv(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)346 1170 y FA(\()p Fi(ty)r(pe)448 1177 y Fg(n)p Fh(\000)p Fj(1)513 1170 y Fi(;)g(disp)618 1177 y Fg(n)p Fh(\000)p Fj(1)692 1170 y FA(+)j(\()p Fv(D)p FA([)p Fv(count)f Fl(\000)h Fv(1)p FA(])e(+)h Fv(B)p FA([)p Fv(count)i Fl(\000)e Fv(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fl(g)p Fi(:)166 1247 y FA(A)k(call)g(to)f Fv(MPI)p 431 1247 V 16 w(TYPE)p 564 1247 V 17 w(VECTOR\(count,)j(blo)q(cklength,)g (stride,)e(oldt)o(yp)q(e,)h(newt)o(yp)q(e\))g FA(is)f(equiv)m(alen)o(t)75 1303 y(to)i(a)f(call)j(to)d Fv(MPI)p 392 1303 V 16 w(TYPE)p 525 1303 V 17 w(INDEX\(count,)i(B,)f(D,)f(oldt)o(yp)q(e,)i(newt)o(yp)q(e\))h FA(where)623 1392 y Fv(D)p FA([)p Fv(j)p FA(])12 b(=)h Fi(j)f Fl(\001)e Fv(stride)16 b Fi(;)8 b(j)14 b FA(=)f(0)p Fi(;)8 b(:::;)g Fv(count)g Fl(\000)j FA(1)p Fi(;)75 1481 y FA(and)594 1537 y Fv(B)p FA([)p Fv(j)p FA(])h(=)h Fv(blo)q(cklength)18 b Fi(;)8 b(j)14 b FA(=)f(0)p Fi(;)8 b(:::;)g Fv(count)g Fl(\000)j FA(1)p Fi(:)75 1655 y Fv(Hindexed)47 b FA(The)11 b(function)h Fv(MPI)p 643 1655 V 15 w(TYPE)p 775 1655 V 17 w(HINDEXED)f FA(is)g(iden)o(tical)i(to)d Fv(MPI)p 1383 1655 V 15 w(TYPE)p 1515 1655 V 17 w(INDEXED)p FA(,)g(except)75 1711 y(that)19 b(blo)q(c)o(k)i(displacemen)o(ts)h(in)41 b Fv(a)o(rra)o(y)p 774 1711 V 14 w(of)p 825 1711 V 16 w(displacements)20 b FA(are)g(sp)q (eci\014ed)i(in)f(b)o(ytes,)g(rather)e(than)h(in)75 1768 y(m)o(ultiples)d(of) e(the)g Fv(oldt)o(yp)q(e)h FA(exten)o(t.)75 1872 y Fv(MPI)p 160 1872 V 16 w(TYPE)p 293 1872 V 17 w(HINDEXED\()21 b(count,)k(a)o(rra)o(y)p 814 1872 V 14 w(of)p 865 1872 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1248 1872 V 15 w(of)p 1300 1872 V 16 w(displacements,)d(oldt)o(yp)q(e,)h (new-)75 1928 y(t)o(yp)q(e\))117 2005 y Fu(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er)34 b(of)f(blo)q(c)o(ks)h({)g(also)f(n)o(um)o(b)q(er)h (of)f(en)o(tries)i(in)905 2062 y Fo(a)o(rra)o(y)p 992 2062 13 2 v 15 w(of)p 1041 2062 V 15 w(displacements)17 b Fu(and)h Fo(a)o(rra)o(y)p 1483 2062 V 15 w(of)p 1532 2062 V 14 w(blo)q(cklengths)i Fu(\(in)o(te-)905 2118 y(ger\))117 2191 y(IN)155 b Fv(a)o(rra)o(y)p 416 2191 14 2 v 15 w(of)p 468 2191 V 16 w(blo)q(cklengths)191 b Fu(n)o(um)o(b)q(er)15 b(of)g(elemen)o(ts)h(in)f(eac)o(h)h(blo)q(c)o(k)g (\(arra)o(y)f(of)g(nonnega-)905 2248 y(tiv)o(e)f(in)o(tegers\))117 2320 y(IN)155 b Fv(a)o(rra)o(y)p 416 2320 V 15 w(of)p 468 2320 V 16 w(displacements)162 b Fu(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h (blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o(teger\))117 2393 y(IN)155 b Fv(oldt)o(yp)q(e)450 b Fu(old)13 b(datat)o(yp)q(e)i(\(handle\))117 2466 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f (\(handle\))75 2591 y Ft(int)23 b(MPI)p 245 2591 15 2 v 17 w(Type)p 358 2591 V 17 w(hindexed\(int)f(count,)h(int)h(*array)p 1092 2591 V 16 w(of)p 1156 2591 V 17 w(blocklengths,)393 2647 y(MPI)p 468 2647 V 17 w(Aint)f(*array)p 748 2647 V 17 w(of)p 813 2647 V 17 w(displacements,)e(MPI)p 1259 2647 V 17 w(Datatype)i(oldtype,) 393 2704 y(MPI)p 468 2704 V 17 w(Datatype)g(*newtype\))-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 50 50 bop 75 -100 a FA(50)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(MPI)p 150 49 15 2 v 17 w(TYPE)p 263 49 V 16 w(HINDEXED\(COUNT,)22 b(ARRAY)p 781 49 V 17 w(OF)p 846 49 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1317 49 V 16 w(OF)p 1381 49 V 17 w(DISPLACEMENTS,)393 106 y(OLDTYPE,)h(NEWTYPE,)g (IERROR\))170 162 y(INTEGER)g(COUNT,)g(ARRAY)p 651 162 V 17 w(OF)p 716 162 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 162 V 17 w(OF)p 1323 162 V 17 w(DISPLACEMENTS\(*\),)170 219 y(OLDTYPE,)i (NEWTYPE,)g(IERROR)166 306 y FA(Assume)15 b(that)g Fv(oldt)o(yp)q(e)h FA(has)f(t)o(yp)q(e)g(map)600 411 y Fl(f)p FA(\()p Fi(ty)r(pe)725 418 y Fj(0)743 411 y Fi(;)8 b(disp)848 418 y Fj(0)865 411 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 418 y Fg(n)p Fh(\000)p Fj(1)1128 411 y Fi(;)g(disp)1233 418 y Fg(n)p Fh(\000)p Fj(1)1297 411 y FA(\))p Fl(g)p Fi(;)75 517 y FA(with)35 b(exten)o(t)g Fr(extent)p FA(.)79 b(Let)36 b Fv(B)f FA(b)q(e)h(the)f Fv(a)o(rra)o(y)p 1013 517 14 2 v 15 w(of)p 1065 517 V 16 w(blo)q(cklength)i FA(argumen)o(t)d(and)h Fv(D)g FA(b)q(e)h(the)75 573 y Fv(a)o(rra)o(y)p 173 573 V 15 w(of)p 225 573 V 16 w(displacements)21 b FA(argumen)o(t.)39 b(The)22 b(newly)h(created)f(datat)o(yp)q(e)f(has)h(a)f(t)o(yp)q(e)h(map)f (with)i Fi(n)14 b Fl(\001)75 598 y Ff(P)119 608 y Fe(count)p Fh(\000)p Fj(1)119 641 y Fg(i)p Fj(=0)243 630 y Fv(B)p FA([)p Fv(i)p FA(])h(en)o(tries:)446 735 y Fl(f)p FA(\()p Fi(ty)r(pe)571 742 y Fj(0)589 735 y Fi(;)8 b(disp)694 742 y Fj(0)721 735 y FA(+)j Fv(D)p FA([)p Fv(0)p FA(])o(\))p Fi(;)d(:::;)g FA(\()o Fi(ty)r(pe)1048 742 y Fg(n)p Fh(\000)p Fj(1)1111 735 y Fi(;)g(disp)1216 742 y Fg(n)p Fh(\000)p Fj(1)1290 735 y FA(+)i Fv(D)p FA([)p Fv(0)p FA(])o(\))p Fi(;)e(:::;)75 844 y FA(\()p Fi(ty)r(pe)177 851 y Fj(0)196 844 y Fi(;)g(disp)301 851 y Fj(0)325 844 y FA(+)f Fv(D)p FA([)p Fv(0)p FA(])e(+)i(\()p Fv(B)p FA([)p Fv(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1037 851 y Fg(n)p Fh(\000)p Fj(1)1099 844 y Fi(;)g(disp)1204 851 y Fg(n)p Fh(\000)p Fj(1)1275 844 y FA(+)f Fv(D)p FA([)p Fv(0)p FA(])f(+)h(\()p Fv(B)p FA([)p Fv(0)p FA(])f Fl(\000)h FA(1\))g Fl(\001)g Fi(extent)p FA(\))p Fi(;)h(:::;)295 934 y FA(\()p Fi(ty)r(pe)397 941 y Fj(0)416 934 y Fi(;)g(disp)521 941 y Fj(0)548 934 y FA(+)j Fv(D)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])o(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1037 941 y Fg(n)p Fh(\000)p Fj(1)1099 934 y Fi(;)g(disp)1204 941 y Fg(n)p Fh(\000)p Fj(1)1278 934 y FA(+)j Fv(D)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])o(\))p Fi(;)e(:::;)375 1024 y FA(\()p Fi(ty)r(pe)477 1031 y Fj(0)496 1024 y Fi(;)g(disp)601 1031 y Fj(0)628 1024 y FA(+)i Fv(D)p FA([)p Fv(count)h Fl(\000)g Fv(1)p FA(])e(+)i(\()p Fv(B)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fi(;)f(:::;)346 1115 y FA(\()p Fi(ty)r(pe)448 1122 y Fg(n)p Fh(\000)p Fj(1)513 1115 y Fi(;)g(disp)618 1122 y Fg(n)p Fh(\000)p Fj(1)692 1115 y FA(+)j Fv(D)p FA([)p Fv(count)f Fl(\000)h Fv(1)p FA(])e(+)i(\()p Fv(B)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])g Fl(\000)g FA(1\))g Fl(\001)f Fi(extent)p FA(\))p Fl(g)p Fi(:)75 1241 y Fv(Struct)47 b(MPI)p 323 1241 V 16 w(TYPE)p 456 1241 V 16 w(STRUCT)23 b FA(is)g(the)f(most)g(general)h(constructor:)33 b(It)22 b(further)g (generalizes)i(the)75 1297 y(previous)16 b(one)f(in)h(that)f(it)g(allo)o(ws)h (eac)o(h)f(blo)q(c)o(k)h(to)e(consist)i(of)f(replications)h(of)f(a)g (di\013eren)o(t)h(datat)o(yp)q(e.)75 1402 y Fv(MPI)p 160 1402 V 16 w(TYPE)p 293 1402 V 17 w(STRUCT\(count,)c(a)o(rra)o(y)p 729 1402 V 15 w(of)p 781 1402 V 16 w(blo)q(cklengths,)h(a)o(rra)o(y)p 1151 1402 V 14 w(of)p 1202 1402 V 16 w(displacements,)e(a)o(rra)o(y)p 1599 1402 V 15 w(of)p 1651 1402 V 15 w(t)o(yp)q(es,)i(new-)75 1459 y(t)o(yp)q(e\))117 1537 y Fu(IN)155 b Fv(count)482 b Fu(n)o(um)o(b)q(er) 18 b(of)h(blo)q(c)o(ks)g(\(in)o(teger\))g({)g(also)f(n)o(um)o(b)q(er)h(of)f (en)o(tries)905 1593 y(in)c(arra)o(ys)h Fo(a)o(rra)o(y)p 1167 1593 13 2 v 15 w(of)p 1216 1593 V 14 w(t)o(yp)q(es)p Fu(,)h Fo(a)o(rra)o(y)p 1432 1593 V 15 w(of)p 1481 1593 V 14 w(displacements)e Fu(and)g Fo(a)o(r-)905 1650 y(ra)o(y)p 959 1650 V 15 w(of)p 1008 1650 V 15 w(blo)q(cklengths)117 1727 y Fu(IN)155 b Fv(a)o(rra)o(y)p 416 1727 14 2 v 15 w(of)p 468 1727 V 16 w(blo)q(cklength)208 b Fu(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g (\(arra)o(y)g(of)f(in)o(teger\))117 1804 y(IN)155 b Fv(a)o(rra)o(y)p 416 1804 V 15 w(of)p 468 1804 V 16 w(displacements)162 b Fu(b)o(yte)14 b(displacemen)o(t)g(of)f(eac)o(h)h(blo)q(c)o(k)g(\(arra)o(y)g(of)f(in)o (teger\))117 1882 y(IN)155 b Fv(a)o(rra)o(y)p 416 1882 V 15 w(of)p 468 1882 V 16 w(t)o(yp)q(es)327 b Fu(t)o(yp)q(e)20 b(of)f(elemen)o(ts) g(in)g(eac)o(h)h(blo)q(c)o(k)f(\(arra)o(y)g(of)g(handles)g(to)905 1938 y(datat)o(yp)q(e)14 b(ob)r(jects\))117 2015 y(OUT)108 b Fv(newt)o(yp)q(e)433 b Fu(new)15 b(datat)o(yp)q(e)f(\(handle\))75 2141 y Ft(int)23 b(MPI)p 245 2141 15 2 v 17 w(Type)p 358 2141 V 17 w(struct\(int)f(count,)h(int)h(*array)p 1044 2141 V 16 w(of)p 1108 2141 V 17 w(blocklengths,)393 2197 y(MPI)p 468 2197 V 17 w(Aint)f(*array)p 748 2197 V 17 w(of)p 813 2197 V 17 w(displacements,)e(MPI)p 1259 2197 V 17 w(Datatype)i(*array)p 1635 2197 V 16 w(of)p 1699 2197 V 17 w(types,)393 2254 y(MPI)p 468 2254 V 17 w(Datatype)g(*newtype\))75 2341 y(MPI)p 150 2341 V 17 w(TYPE)p 263 2341 V 16 w(STRUCT\(COUNT,)f(ARRAY)p 733 2341 V 17 w(OF)p 798 2341 V 17 w(BLOCKLENGTHS,)g(ARRAY)p 1269 2341 V 16 w(OF)p 1333 2341 V 17 w(DISPLACEMENTS,)393 2398 y(ARRAY)p 516 2398 V 17 w(OF)p 581 2398 V 17 w(TYPES,)h(NEWTYPE,)f(IERROR\))170 2454 y(INTEGER)h(COUNT,)g(ARRAY)p 651 2454 V 17 w(OF)p 716 2454 V 17 w(BLOCKLENGTHS\(*\),)e(ARRAY)p 1258 2454 V 17 w(OF)p 1323 2454 V 17 w(DISPLACEMENTS\(*\),)170 2511 y(ARRAY)p 293 2511 V 17 w(OF)p 358 2511 V 17 w(TYPES\(*\),)h(NEWTYPE,)h(IERROR)166 2598 y FA(Example:)d(Let)c Fv(t)o(yp)q(e1)g FA(ha)o(v)o(e)f(t)o(yp)q(e)g(map) 755 2704 y Fl(f)p FA(\()p Fv(double)q Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(8\))p Fl(g)p Fi(;)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 51 51 bop 75 -100 a Fw(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b FA(51)75 49 y(with)12 b(exten)o(t)f(16.)18 b(Let)11 b Fv(B)h(=)g(\(2,)f(1,) g(3\))p FA(,)g Fv(D)h(=)f(\(0,)g(16,)h(26\))p FA(,)e(and)i Fv(T)f(=)h(\(MPI)p 1321 49 14 2 v 15 w(FLO)o(A)l(T,)f(t)o(yp)q(e1,)i(MPI)p 1711 49 V 16 w(CHAR\))p FA(.)75 106 y(Then)j(a)f(call)h(to)e Fv(MPI)p 455 106 V 16 w(TYPE)p 588 106 V 17 w(STRUCT\(3,)h(B,)g(D,)g(T,)f (newt)o(yp)q(e\))j FA(returns)e(datat)o(yp)q(e)g(with)g(t)o(yp)q(e)h(map,)240 212 y Fl(f)p FA(\()p Fv(\015oat)p Fi(;)8 b FA(0\))p Fi(;)g FA(\()p Fv(\015oat)m Fi(;)g FA(4\))p Fi(;)g FA(\()p Fv(double)o Fi(;)g FA(16\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(24\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(26\))p Fi(;)g FA(\()p Fv(cha)o(r)m Fi(;)g FA(27\))p Fi(;)g FA(\()p Fv(cha)o(r)n Fi(;)g FA(28\))p Fl(g)i FA(:)75 318 y(t)o(w)o(o)g(copies)j(of)e Fv(MPI)p 421 318 V 15 w(FLO)o(A)l(T)h FA(starting)f(at)g(0,)g(follo)o(w)o(ed)h(b)o(y)f (one)h(cop)o(y)f(of)g Fv(t)o(yp)q(e1)i FA(starting)e(at)f(16,)i(follo)o(w)o (ed)75 375 y(b)o(y)i(three)g(copies)h(of)e Fv(MPI)p 520 375 V 16 w(CHAR)p FA(,)i(starting)e(at)g(26.)19 b(\(W)l(e)14 b(assume)g(that)f(a) g(\015oat)h(o)q(ccupies)h(four)f(b)o(ytes.\))166 433 y(In)i(general,)f(let)h Fv(T)f FA(b)q(e)h(the)f Fv(a)o(rra)o(y)p 742 433 V 14 w(of)p 793 433 V 16 w(t)o(yp)q(es)i FA(argumen)o(t,)d(where)31 b Fv(T[i])15 b FA(is)h(a)f(handle)h(to)464 539 y Fi(ty)r(pemap)635 546 y Fg(i)661 539 y FA(=)d Fl(f)p FA(\()p Fi(ty)r(pe)834 520 y Fg(i)834 550 y Fj(0)853 539 y Fi(;)8 b(disp)958 520 y Fg(i)958 550 y Fj(0)975 539 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1176 520 y Fg(i)1176 550 y(n)1197 554 y Fd(i)1207 550 y Fh(\000)p Fj(1)1251 539 y Fi(;)g(disp)1356 520 y Fg(i)1356 550 y(n)1377 554 y Fd(i)1389 550 y Fh(\000)p Fj(1)1433 539 y FA(\))p Fl(g)p Fi(;)75 645 y FA(with)i(exten)o(t)g Fi(extent)434 652 y Fg(i)450 645 y FA(.)18 b(Let)10 b Fv(B)g FA(b)q(e)h(the)g Fv(a)o(rra)o(y)p 826 645 V 14 w(of)p 877 645 V 16 w(blo)q(cklength)h FA(argumen)o(t)d(and)i Fv(D)f FA(b)q(e)h(the)f Fv(a)o(rra)o(y)p 1671 645 V 14 w(of)p 1722 645 V 16 w(displacements)75 702 y FA(argumen)o(t.)19 b(Then)d(the)f (newly)h(created)f(datat)o(yp)q(e)g(has)g(a)g(t)o(yp)q(e)g(map)g(with)1399 670 y Ff(P)1443 680 y Fe(count)p Fh(\000)p Fj(1)1443 713 y Fg(i)p Fj(=0)1567 702 y Fv(B)p FA([)p Fv(i)p FA(])9 b Fl(\001)h Fi(n)1693 709 y Fg(i)1722 702 y FA(en)o(tries)472 821 y Fl(f)p FA(\()p Fi(ty)r(pe)597 802 y Fj(0)597 832 y(0)615 821 y Fi(;)e(disp)720 802 y Fj(0)720 832 y(0)748 821 y FA(+)i Fv(D)p FA([)p Fv(0)p FA(])o(\))p Fi(;)e(:::;)g FA(\()p Fi(ty)r(pe)1075 802 y Fj(0)1075 832 y Fg(n)1096 836 y Fc(0)1111 821 y Fi(;)g(disp)1216 802 y Fj(0)1216 832 y Fg(n)1237 836 y Fc(0)1263 821 y FA(+)j Fv(D)p FA([)p Fv(0)p FA(])o(\))p Fi(;)d(:::;)75 931 y FA(\()p Fi(ty)r(pe)177 913 y Fj(0)177 943 y(0)196 931 y Fi(;)g(disp)301 913 y Fj(0)301 943 y(0)325 931 y FA(+)f Fv(D)p FA([)p Fv(0)p FA(])g(+)g(\()p Fv(B)p FA([)p Fv(0)p FA(])g Fl(\000)g FA(1\))g Fl(\001)g Fi(extent)839 938 y Fj(0)860 931 y FA(\))p Fi(;)h(:::;)g FA(\()p Fi(ty)r(pe)1061 913 y Fj(0)1061 943 y Fg(n)1081 947 y Fc(0)1097 931 y Fi(;)g(disp)1202 913 y Fj(0)1202 943 y Fg(n)1223 947 y Fc(0)1247 931 y FA(+)f Fv(D)p FA([)p Fv(0)p FA(])g(+)g(\()p Fv(B)p FA([)p Fv(0)p FA(])i Fl(\000)i Fv(1)p FA(\))c Fl(\001)g Fi(extent)1767 938 y Fj(0)1787 931 y FA(\))p Fi(;)h(:::;)107 1024 y FA(\()p Fi(ty)r(pe)209 1005 y Fe(count)p Fh(\000)p Fj(1)209 1035 y(0)326 1024 y Fi(;)g(disp)431 1005 y Fe(count)n Fh(\000)p Fj(1)431 1035 y(0)556 1024 y FA(+)j Fv(D)p FA([)p Fv(count)f Fl(\000)h Fv(1)p FA(])o(\))p Fi(;)d(:::;)g FA(\()p Fi(ty)r(pe)1045 1005 y Fe(count)m Fh(\000)p Fj(1)1045 1035 y Fg(n)1066 1039 y Fb(count)m Fa(\000)p Fc(1)1153 1035 y Fh(\000)p Fj(1)1198 1024 y Fi(;)g(disp)1303 1005 y Fe(count)n Fh(\000)p Fj(1)1303 1035 y Fg(n)1324 1039 y Fb(count)n Fa(\000)p Fc(1)1412 1035 y Fh(\000)p Fj(1)1467 1024 y FA(+)i Fv(D)p FA([)p Fv(count)h Fl(\000)g Fv(1)p FA(])o(\))p Fi(;)d(:::;)218 1116 y FA(\()p Fi(ty)r(pe)320 1097 y Fe(count)q Fh(\000)p Fj(1)320 1127 y(0)437 1116 y Fi(;)g(disp)542 1097 y Fe(count)o Fh(\000)p Fj(1)542 1127 y(0)668 1116 y FA(+)i Fv(D)p FA([)p Fv(count)h Fl(\000)g Fv(1)p FA(])e(+)h(\()p Fv(B)p FA([)p Fv(count)h Fl(\000)g Fv(1)p FA(])e Fl(\000)i FA(1\))e Fl(\001)h Fi(extent)1525 1123 y Fe(count)r Fh(\000)p Fj(1)1643 1116 y FA(\))p Fi(;)e(:::;)204 1209 y FA(\()p Fi(ty)r(pe)306 1190 y Fe(count)p Fh(\000)p Fj(1)306 1220 y Fg(n)327 1224 y Fb(count)p Fa(\000)p Fc(1)417 1220 y Fh(\000)p Fj(1)461 1209 y Fi(;)g(disp)566 1190 y Fe(count)o Fh(\000)p Fj(1)566 1220 y Fg(n)587 1224 y Fb(count)o Fa(\000)p Fc(1)676 1220 y Fh(\000)p Fj(1)730 1209 y FA(+)j Fv(D)p FA([)p Fv(count)g Fl(\000)f Fv(1)p FA(])g(+)g(\()p Fv(B)p FA([)p Fv(count)h Fl(\000)f Fv(1)p FA(])g Fl(\000)g Fv(1)p FA(\))g Fl(\001)g Fi(extent)1588 1216 y Fe(count)q Fh(\000)p Fj(1)1706 1209 y FA(\))p Fl(g)166 1298 y FA(A)15 b(call)h(to)f Fv(MPI)p 439 1298 V 16 w(TYPE)p 572 1298 V 16 w(HINDEXED\()g(count,)h(B,)f(D,)g(oldt)o(yp) q(e,)h(newt)o(yp)q(e\))g FA(is)g(equiv)m(alen)o(t)h(to)d(a)h(call)75 1355 y(to)d Fv(MPI)p 213 1355 V 16 w(TYPE)p 346 1355 V 17 w(STRUCT\()h (count,)i(B,)e(D,)f(T,)h(newt)o(yp)q(e\))p FA(,)h(where)f(eac)o(h)g(en)o(try) g(of)g Fv(T)g FA(is)g(equal)h(to)e Fv(oldt)o(yp)q(e)p FA(.)75 1485 y Fs(1.13.2)49 b(Address)16 b(and)g(extent)f(functions)75 1573 y FA(The)j(displacemen)o(ts)i(in)f(a)f(general)g(datat)o(yp)q(e)g(are)g (relativ)o(e)h(to)e(some)h(initial)i(bu\013er)e(address.)29 b Fz(Ab-)75 1630 y(solute)20 b(addresses)15 b FA(can)i(b)q(e)g(substituted)h (for)e(these)h(displacemen)o(ts:)24 b(w)o(e)16 b(treat)g(them)h(as)f (displace-)75 1686 y(men)o(ts)e(relativ)o(e)h(to)f(\\address)g(zero",)g(the)h (start)e(of)h(the)h(address)g(space.)20 b(This)15 b(initial)h(address)f(zero) f(is)75 1742 y(indicated)i(b)o(y)d(the)h(constan)o(t)f Fo(MPI)p 668 1742 13 2 v 15 w(BOTTOM)p FA(.)g(Th)o(us,)h(a)f(datat)o(yp)q(e)g(can)i (sp)q(ecify)g(the)f(absolute)g(address)75 1799 y(of)k(the)g(en)o(tries)g(in)h (the)f(comm)o(unication)h(bu\013er,)f(in)h(whic)o(h)g(case)f(the)g Fv(buf)h FA(argumen)o(t)e(is)i(passed)f(the)75 1855 y(v)m(alue)e Fo(MPI)p 269 1855 V 15 w(BOTTOM)p FA(.)166 1913 y(The)31 b(address)g(of)f(a)h (lo)q(cation)g(in)h(memory)e(can)h(b)q(e)h(found)f(b)o(y)g(in)o(v)o(oking)g (the)g(function)75 1970 y Fv(MPI)p 160 1970 14 2 v 16 w(ADDRESS)p FA(.)75 2075 y Fv(MPI)p 160 2075 V 16 w(ADDRESS\(lo)q(cation,)16 b(address\))117 2153 y Fu(IN)155 b Fv(lo)q(cation)437 b Fu(lo)q(cation)13 b(in)h(caller)f(memory)f(\(c)o(hoice\))117 2231 y(OUT)108 b Fv(address)449 b Fu(address)15 b(of)f(lo)q(cation)f(\(in)o(teger\))75 2357 y Ft(int)23 b(MPI)p 245 2357 15 2 v 17 w(Address\(void*)f(location,)h (MPI)p 907 2357 V 17 w(Aint)g(*address\))75 2445 y(MPI)p 150 2445 V 17 w(ADDRESS\(LOCATION,)e(ADDRESS,)i(IERROR\))170 2501 y()g(LOCATION\(*\))170 2558 y(INTEGER)g(ADDRESS,)g(IERROR)166 2646 y FA(Returns)16 b(the)f(\(b)o(yte\))f(address)i(of)e Fv(lo)q(cation)p FA(.)166 2704 y(Example:)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 52 52 bop 75 -100 a FA(52)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)147 49 y Ft(REAL)23 b(A\(100,100\))147 106 y(INTEGER)f(I1,)i(I2,)f(DIFF)147 162 y(CALL)g(MPI_ADDRESS\(A\(1,1\),)e (I1,)j(IERROR\))147 219 y(CALL)f(MPI_ADDRESS\(A\(10,10\),)e(I2,)i(IERROR\)) 147 275 y(DIFF)g(=)h(I2)f(-)h(I1)75 332 y(!)g(The)f(value)g(of)h(DIFF)f(is)h (909*sizeofreal;)e(the)h(values)g(of)h(I1)f(and)h(I2)f(are)75 388 y(!)h(implementation)e(dependent.)189 502 y Fr(A)n(dvic)n(e)e(to)i (users.)73 b FA(C)20 b(users)h(ma)o(y)f(b)q(e)h(tempted)g(to)f(a)o(v)o(oid)h (the)f(usage)h(of)f Fv(MPI)p 1659 502 14 2 v 16 w(ADDRESS)189 558 y FA(and)e(rely)g(on)g(the)g(a)o(v)m(ailabilit)o(y)i(of)d(the)h(address)g (op)q(erator)f(&.)29 b(Note,)17 b(ho)o(w)o(ev)o(er,)h(that)f Fo(&)h Fr(c)n(ast-)189 614 y(expr)n(ession)c FA(is)j(a)e(p)q(oin)o(ter,)i (not)e(an)h(address.)22 b(ANSI)17 b(C)f(do)q(es)g(not)g(require)g(that)g(the) g(v)m(alue)h(of)e(a)189 671 y(p)q(oin)o(ter)g(\(or)g(the)g(p)q(oin)o(ter)h (cast)f(to)f Fo(int)p FA(\))h(b)q(e)h(the)g(absolute)f(address)h(of)e(the)i (ob)s(ject)e(p)q(oin)o(ted)j(at)d({)189 727 y(although)j(this)g(is)h(the)f (common)g(case.)25 b(F)l(urthermore,)17 b(referencing)h(ma)o(y)e(not)h(ha)o (v)o(e)g(a)f(unique)189 784 y(de\014nition)i(on)f(mac)o(hines)g(with)g (segmen)o(ted)g(address)g(space.)24 b(The)17 b(use)g(of)f Fv(MPI)p 1602 784 V 16 w(ADDRESS)i FA(to)189 840 y(\\reference")11 b(C)g(v)m(ariables) h(guaran)o(tees)e(p)q(ortabilit)o(y)i(to)f(suc)o(h)g(mac)o(hines)h(as)e(w)o (ell.)20 b(\()p Fr(End)12 b(of)g(advic)n(e)189 897 y(to)k(users.)p FA(\))166 1012 y(The)f(follo)o(wing)h(auxiliary)h(functions)f(pro)o(vide)f (useful)i(information)e(on)g(deriv)o(ed)h(datat)o(yp)q(es.)75 1118 y Fv(MPI)p 160 1118 V 16 w(TYPE)p 293 1118 V 17 w(EXTENT\(datat)o(yp)q (e,)g(extent\))117 1197 y Fu(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(\(handle\))117 1276 y(OUT)108 b Fv(extent)471 b Fu(datat)o(yp)q(e)14 b(exten)o(t)h(\(in)o(teger\))75 1402 y Ft(int)23 b(MPI)p 245 1402 15 2 v 17 w(Type)p 358 1402 V 17 w(extent\(MPI)p 615 1402 V 16 w(Datatype)f(datatype,)h(MPI)p 1156 1402 V 17 w(Aint)g(*extent\))75 1490 y(MPI)p 150 1490 V 17 w(TYPE)p 263 1490 V 16 w(EXTENT\(DATATYPE,)f(EXTENT,)h(IERROR\))170 1547 y(INTEGER)g(DATATYPE,)g(EXTENT,)g(IERROR)166 1635 y FA(Returns)16 b(the)f(exten)o(t)g(of)g(a)f(datat)o(yp)q(e)h({)g(where)g(exten)o(t)g(is)h (as)f(de\014ned)h(in)g(Eq.)f(1.1)f(on)h(page)h(45.)75 1740 y Fv(MPI)p 160 1740 14 2 v 16 w(TYPE)p 293 1740 V 17 w(SIZE\(datat)o(yp)q(e,) f(size\))117 1819 y Fu(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(\(handle\))117 1898 y(OUT)108 b Fv(size)519 b Fu(datat)o(yp)q(e)14 b(size)h(\(in)o(teger\))75 2024 y Ft(int)23 b(MPI)p 245 2024 15 2 v 17 w(Type)p 358 2024 V 17 w(size\(MPI)p 567 2024 V 16 w(Datatype)g(datatype,)f(MPI)p 1108 2024 V 17 w(Aint)h(*size\))75 2113 y(MPI)p 150 2113 V 17 w(TYPE)p 263 2113 V 16 w(SIZE\(DATATYPE,)f(SIZE,)i (IERROR\))170 2169 y(INTEGER)f(DATATYPE,)g(SIZE,)g(IERROR)166 2257 y FA(Returns)15 b(the)h(n)o(um)o(b)q(er)f(of)g(b)o(ytes)f(o)q(ccupied)j (b)o(y)e(en)o(tries)h(in)g(the)f(datat)o(yp)q(e)f({)h(i.e.)20 b(the)15 b(sum)g(of)g(the)75 2314 y(sizes)h(of)f(the)g(datat)o(yp)q(e)g (elemen)o(ts.)75 2419 y Fv(MPI)p 160 2419 14 2 v 16 w(TYPE)p 293 2419 V 17 w(COUNT\(datat)o(yp)q(e,)i(count\))117 2499 y Fu(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(\(handle\))117 2577 y(OUT)108 b Fv(count)482 b Fu(datat)o(yp)q(e)14 b(coun)o(t)g(\(in)o (teger\))75 2704 y Ft(int)23 b(MPI)p 245 2704 15 2 v 17 w(Type)p 358 2704 V 17 w(count\(MPI)p 591 2704 V 16 w(Datatype)g(datatype,)f(int)i (*count\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 53 53 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(53)75 49 y Ft(MPI)p 150 49 15 2 v 17 w(TYPE)p 263 49 V 16 w(COUNT\(DATATYPE,)22 b(COUNT,)h(IERROR\))170 106 y(INTEGER)g(DATATYPE,)g (COUNT,)g(IERROR)166 192 y FA(Returns)16 b(the)f(n)o(um)o(b)q(er)g(of)g(en)o (tries)h(in)g(the)f(datat)o(yp)q(e.)75 316 y Fs(1.13.3)49 b(Lo)o(w)o(er-b)q (ound)17 b(and)g(upp)q(er-b)q(ound)g(ma)o(rk)n(ers)75 402 y FA(It)e(is)g(often)g(con)o(v)o(enien)o(t)g(to)g(de\014ne)h(explicitly)h(the)e (lo)o(w)o(er)g(b)q(ound)h(and)f(upp)q(er)h(b)q(ound)f(of)g(a)g(t)o(yp)q(e)g (map,)75 459 y(and)20 b(o)o(v)o(erwrite)g(the)g(de\014nition)i(giv)o(en)f(b)o (y)f(Equation)g(1.1)g(on)g(page)g(45.)34 b(This)21 b(allo)o(ws)f(to)g (de\014ne)h(a)75 515 y(datat)o(yp)q(e)12 b(that)g(has)h(\\holes")g(at)f(its)h (b)q(eginning)h(or)f(its)g(end;)g(or)g(a)f(datat)o(yp)q(e)g(with)h(en)o (tries)g(that)f(extend)75 572 y(ab)q(o)o(v)o(e)k(the)g(upp)q(er)g(b)q(ound)h (or)f(b)q(elo)o(w)g(the)g(lo)o(w)o(er)g(b)q(ound.)23 b(Examples)16 b(of)g(suc)o(h)g(a)g(usage)g(are)f(pro)o(vided)75 628 y(in)h(section)g (1.13.7.)j(T)l(o)d(ac)o(hiev)o(e)g(this,)g(w)o(e)f(add)h(t)o(w)o(o)e (additional)j(\\pseudo-datat)o(yp)q(es")e Fv(MPI)p 1706 628 14 2 v 16 w(LB)p FA(,)g(and)75 685 y Fv(MPI)p 160 685 V 16 w(UB)i FA(that)e(can)i(b)q(e)g(used,)g(resp)q(ectiv)o(ely)l(,)h(to)e(mark)f (the)i(lo)o(w)o(er)f(b)q(ound)h(or)f(the)h(upp)q(er)g(b)q(ound)g(of)f(a)75 741 y(datat)o(yp)q(e.)j(These)c(pseudo-datat)o(yp)q(es)g(o)q(ccup)o(y)g(no)f (space)h(\()p Fi(extent)p FA(\()p Fv(MPI)p 1357 741 V 17 w(LB)p FA(\))d(=)h Fi(extent)p FA(\()p Fv(MPI)p 1734 741 V 17 w(UB)p FA(\))g(=)75 798 y(0\).)18 b(They)c(do)f(not)f(a\013ect)g(the)h(size)h(or)f (coun)o(t)g(of)f(a)h(datat)o(yp)q(e,)f(and)h(do)g(not)g(a\013ect)f(the)h(the) g(con)o(ten)o(t)g(of)f(a)75 854 y(message)i(created)h(with)h(this)f(datat)o (yp)q(e.)k(Ho)o(w)o(ev)o(er,)14 b(they)h(do)g(a\013ect)f(the)h(de\014nition)h (of)f(the)g(exten)o(t)f(of)75 911 y(a)e(datat)o(yp)q(e)g(and,)h(therefore,)g (a\013ect)f(the)g(outcome)h(of)f(a)g(replication)i(of)f(this)g(datat)o(yp)q (e)f(b)o(y)g(a)g(datat)o(yp)q(e)75 967 y(constructor.)166 1024 y(Example:)27 b(Let)19 b Fv(D)f(=)h(\(-3,)f(0,)h(6\))p FA(;)g Fv(T)f(=)h(\(MPI)p 990 1024 V 15 w(LB,)f(MPI)p 1173 1024 V 16 w(INT,)g(MPI)p 1378 1024 V 16 w(UB\))p FA(,)g(and)h Fv(B)f(=)h(\(1,)f(1,)h (1\))p FA(.)75 1080 y(Then)e(a)e(call)i(to)f Fv(MPI)p 459 1080 V 16 w(TYPE)p 592 1080 V 16 w(STRUCT\(3,)g(B,)g(D,)g(T,)f(t)o(yp)q(e1\))i FA(creates)e(a)h(new)g(datat)o(yp)q(e)g(that)f(has)h(an)75 1137 y(exten)o(t)f(of)g(9)g(\(from)f(-3)i(to)e(5\),)h(and)g(con)o(tains)h(an) f(in)o(teger)h(at)e(displacemen)o(t)j(0.)j(This)c(is)g(the)f(datat)o(yp)q(e) 75 1193 y(de\014ned)g(b)o(y)g(the)f(sequence)h Fl(f)p Fv(\(lb,)f(-3\),)f (\(int,)i(0\),)e(\(ub,)h(6\))p Fl(g)g FA(.)19 b(If)c(this)f(t)o(yp)q(e)g(is)h (replicated)h(t)o(wice)e(b)o(y)g(a)g(call)75 1250 y(to)f Fv(MPI)p 214 1250 V 16 w(TYPE)p 347 1250 V 16 w(CONTIGUOUS\(2,)i(t)o(yp)q(e1,)f(t)o (yp)q(e2\))g FA(then)g(the)g(newly)g(created)g(t)o(yp)q(e)f(can)h(b)q(e)g (describ)q(ed)75 1306 y(b)o(y)f(the)g(sequence)h Fl(f)p Fv(\(lb,)f(-3\),)g (\(int,)g(0\),)f(\(int,9\),)h(\(ub,)h(15\))p Fl(g)d FA(.)20 b(\(En)o(tries)12 b(of)h(t)o(yp)q(e)g Fv(lb)g FA(or)g Fv(ub)h FA(can)f(b)q(e)h(deleted)75 1363 y(if)i(they)f(are)g(not)g(at)f(the)h(end-p)q (oin)o(ts)i(of)e(the)g(datat)o(yp)q(e.\))166 1419 y(In)h(general,)f(if)476 1522 y Fi(T)6 b(y)r(pemap)12 b FA(=)h Fl(f)p FA(\()p Fi(ty)r(pe)849 1529 y Fj(0)867 1522 y Fi(;)8 b(disp)972 1529 y Fj(0)990 1522 y FA(\))p Fi(;)g(:::;)f FA(\()p Fi(ty)r(pe)1190 1529 y Fg(n)p Fh(\000)p Fj(1)1253 1522 y Fi(;)h(disp)1358 1529 y Fg(n)p Fh(\000)p Fj(1)1421 1522 y FA(\))p Fl(g)p Fi(;)75 1625 y FA(then)16 b(the)f Fz(lo)o(w)o(er)h(b)q(ound)g FA(of)f Fi(T)6 b(y)r(pemap)15 b FA(is)g(de\014ned)i(to)e(b)q(e)227 1759 y Fi(l)q(b)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))k(=)544 1687 y Ff(\()598 1730 y FA(min)674 1737 y Fg(j)699 1730 y Fi(disp)783 1737 y Fg(j)1142 1730 y FA(if)16 b(no)f(en)o(try)g(has)g(basic)h(t)o(yp)q(e)f Fv(lb)598 1787 y FA(min)q Fl(f)p Fi(disp)781 1794 y Fg(j)826 1787 y FA(:)27 b Fi(ty)r(pe)950 1794 y Fg(j)981 1787 y FA(=)13 b Fv(lb)p Fl(g)41 b FA(otherwise)166 1892 y(Similarly)l(,)17 b(the)e Fz(upp)q(er)j(b)q(ound)e FA(of)e Fi(T)6 b(y)r(pemap)15 b FA(is)h(de\014ned)h(to)d(b)q(e)191 2026 y Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))11 b(=)520 1954 y Ff(\()575 1997 y FA(max)659 2004 y Fg(j)684 1997 y Fi(disp)768 2004 y Fg(j)795 1997 y FA(+)g Fi(siz)r(eof)5 b FA(\()p Fi(ty)r(pe)1073 2004 y Fg(j)1091 1997 y FA(\))56 b(if)16 b(no)f(en)o(try)g(has)g(basic)h(t)o(yp)q (e)f Fv(ub)575 2054 y FA(max)o Fl(f)p Fi(disp)766 2061 y Fg(j)811 2054 y FA(:)27 b Fi(ty)r(pe)935 2061 y Fg(j)966 2054 y FA(=)13 b Fv(ub)q Fl(g)66 b FA(otherwise)166 2159 y(Then)441 2217 y Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))14 b(=)e Fi(ub)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))j Fl(\000)i Fi(l)q(b)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))i(+)i Fi(\017)75 2301 y FA(If)19 b Fi(ty)r(pe)208 2308 y Fg(i)242 2301 y FA(requires)h(alignmen)o(t)g(to)e(a)h (b)o(yte)g(address)g(that)g(is)h(a)f(m)o(ultiple)i(of)d Fi(k)1460 2308 y Fg(i)1474 2301 y FA(,)i(then)f Fi(\017)h FA(is)g(the)f(least)75 2358 y(nonnegativ)o(e)d(incremen)o(t)g(needed)g(to)f(round)g Fi(extent)p FA(\()p Fi(T)6 b(y)r(pemap)p FA(\))16 b(to)e(the)h(next)h(m)o (ultiple)h(of)d(max)1794 2365 y Fg(i)1815 2358 y Fi(k)1839 2365 y Fg(i)1853 2358 y FA(.)166 2414 y(The)g(formal)f(de\014nitions)j(giv)o (en)e(for)f(the)h(v)m(arious)g(datat)o(yp)q(e)f(constructors)g(apply)i(no)o (w,)e(with)h(the)75 2471 y(amended)i(de\014nition)h(of)e Fz(exten)o(t)p FA(.)166 2604 y Fq(Discussion:)52 b Fu(The)15 b(follo)o(wing)c(additional)h (functions)i(w)o(ere)h(suggested:)575 b Fn(\()166 2654 y Fo(MPI)p 243 2654 13 2 v 14 w(TYPE)p 365 2654 V 14 w(LB\()15 b(datat)o(yp)q(e,)f (displacement\))75 2704 y Fu(Returns)h(the)f(lo)o(w)o(er)g(b)q(ound)g(of)f (the)h(t)o(yp)q(e)h(map)d(of)h(datat)o(yp)q(e.)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 54 54 bop 75 -100 a FA(54)613 b Fw(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fo(MPI)p 243 49 13 2 v 14 w(TYPE)p 365 49 V 14 w(UB\()e(datat)o(yp)q(e,)g(displacement\))75 99 y Fu(Returns)h(the)f(upp)q(er)h(b)q(ound)f(of)f(the)i(t)o(yp)q(e)f(map)f (of)g(datat)o(yp)q(e.)75 306 y Fs(1.13.4)49 b(Comm)o(it)13 b(and)k(free)75 392 y FA(A)g(datat)o(yp)q(e)f(ob)s(ject)h(has)g(to)f(b)q(e)h Fz(committed)h FA(b)q(efore)f(it)g(can)g(b)q(e)h(used)g(in)f(a)g(comm)o (unication.)26 b(The)75 449 y(system)20 b(ma)o(y)g(\\compile")i(at)e(commit)g (time)h(an)g(in)o(ternal)g(represen)o(tation)g(for)f(the)h(datat)o(yp)q(e)f (that)75 505 y(facilitates)c(comm)o(unication,)g(e.g.)k(c)o(hange)15 b(from)g(a)g(compacted)g(represen)o(tation)h(to)e(a)h(\015at)g(represen-)75 561 y(tation)i(of)f(the)h(datat)o(yp)q(e,)g(and)g(select)g(the)g(most)g(con)o (v)o(enien)o(t)g(transfer)f(mec)o(hanism.)26 b(A)17 b(committed)75 618 y(datat)o(yp)q(e)e(can)g(still)i(b)q(e)e(used)h(as)f(a)g(argumen)o(t)f (in)i(datat)o(yp)q(e)f(constructors.)166 675 y(Note:)28 b(the)20 b(commit)f(op)q(eration)h(commits)f(the)h(datat)o(yp)q(e,)f(i.e.,)i(the)e (formal)g(description)i(of)e(a)75 731 y(comm)o(unication)i(bu\013er,)h(not)e (the)h(con)o(ten)o(t)f(of)g(that)g(bu\013er.)36 b(Th)o(us,)22 b(after)e(a)g(datat)o(yp)q(e)g(has)h(b)q(een)75 788 y(committed,)d(it)h(can)f (b)q(e)h(rep)q(eatedly)g(reused)g(to)e(comm)o(unicate)h(the)h(c)o(hanging)f (con)o(ten)o(t)g(of)f(a)h(bu\013er)75 844 y(or,)c(indeed,)j(the)e(con)o(ten)o (t)g(of)g(di\013eren)o(t)g(bu\013ers,)g(with)g(di\013eren)o(t)h(starting)e (addresses.)75 948 y Fv(MPI)p 160 948 14 2 v 16 w(TYPE)p 293 948 V 17 w(COMMIT\(datat)o(yp)q(e\))117 1026 y Fu(INOUT)62 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(that)g(is)g(committed)e (\(handle\))75 1151 y Ft(int)23 b(MPI)p 245 1151 15 2 v 17 w(Type)p 358 1151 V 17 w(commit\(MPI)p 615 1151 V 16 w(Datatype)f (*datatype\))75 1237 y(MPI)p 150 1237 V 17 w(TYPE)p 263 1237 V 16 w(COMMIT\(DATATYPE,)g(IERROR\))170 1294 y(INTEGER)h(DATATYPE,)g(IERROR) 166 1381 y FA(There)15 b(is)h(no)f(need)h(to)f(commit)g(basic)h(datat)o(yp)q (es;)e(they)h(are)g(\\pre-committed".)75 1485 y Fv(MPI)p 160 1485 14 2 v 16 w(TYPE)p 293 1485 V 17 w(FREE\(datat)o(yp)q(e\))117 1562 y Fu(INOUT)62 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(that)g(is)g(freed)h(\(handle\))75 1687 y Ft(int)23 b(MPI)p 245 1687 15 2 v 17 w(Type)p 358 1687 V 17 w(free\(MPI)p 567 1687 V 16 w(Datatype)g(*datatype\))75 1774 y(MPI)p 150 1774 V 17 w(TYPE)p 263 1774 V 16 w(FREE\(DATATYPE,)f(IERROR\))170 1830 y(INTEGER)h(DATATYPE,)g(IERROR)166 1917 y FA(Marks)16 b(the)h(datat)o(yp)q(e)f(ob)s(ject)g(asso)q(ciated)h(with)g Fv(datat)o(yp)q(e)i FA(for)d(deallo)q(cation.)26 b(The)17 b(ob)s(ject)f(will) 75 1974 y(b)q(e)g(deallo)q(cated)h(after)d(an)o(y)i(p)q(ending)h(comm)o (unication)f(that)e(uses)i(this)g(ob)s(ject)f(completes,)g(at)g(whic)o(h)75 2030 y(p)q(oin)o(t)21 b Fv(datat)o(yp)q(e)g FA(b)q(ecomes)g(n)o(ull.)36 b(It)20 b(is)h(erroneous)f(to)f(use)i(a)f(datat)o(yp)q(e)f(as)h(argumen)o(t)f (to)h(an)g(MPI)75 2087 y(function)c(after)e(it)i(w)o(as)e(freed.)166 2143 y(Example:)75 2308 y Ft(INTEGER)23 b(type1,)g(type2)75 2365 y(CALL)g(MPI_TYPE_CONTIGUOUS\(5,)e(MPI_REAL,)i(type1,)g(ierr\))409 2421 y(!)h(new)f(type)h(object)f(created)75 2478 y(CALL)g (MPI_TYPE_COMMIT\(type1,)e(ierr\))409 2534 y(!)j(now)f(type1)g(can)h(be)f (used)h(for)f(communication)75 2591 y(type2)g(=)h(type1)409 2647 y(!)g(type2)f(can)g(be)h(used)f(for)h(communication)409 2704 y(!)g(\(it)f(is)h(a)g(handle)f(to)g(same)h(object)f(as)g(type1\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 55 55 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(55)75 49 y Ft(CALL)23 b(MPI_TYPE_VECTOR\(3,)f(5,)h(4,)h(MPI_REAL,)f (type1,)g(ierr\))409 106 y(!)h(new)f(uncommitted)g(type)g(object)g(created)75 162 y(CALL)g(MPI_TYPE_COMMIT\(type1,)e(ierr\))409 219 y(!)j(now)f(type1)g (can)h(be)f(used)h(anew)f(for)h(communication)166 317 y FA(F)l(reeing)19 b(a)f(datat)o(yp)q(e)f(do)q(es)i(not)f(a\013ect)f(an)o(y)h(other)g(datat)o (yp)q(e)f(that)h(w)o(as)f(built)j(from)d(the)i(freed)75 373 y(datat)o(yp)q(e.)34 b(The)20 b(system)g(should)h(b)q(eha)o(v)o(e)g(as)e(if)i (a)f(separate)f(cop)o(y)h(of)g(the)g(datat)o(yp)q(e)g(w)o(as)f(created)75 430 y(whenev)o(er)d(this)f(datat)o(yp)q(e)g(w)o(as)f(incorp)q(orated)i(in)g (a)f(new)g(deriv)o(ed)i(datat)o(yp)q(e.)166 562 y Fq(Implemen)o(tati)o(on)12 b(note:)166 612 y Fu(The)e(implemen)o(tati)o(on)d(ma)o(y)g(k)o(eep)j(a)f (reference)j(coun)o(t)e(of)f(activ)o(e)g(comm)o(unications)d(that)k(use)g (the)g(datat)o(yp)q(e,)75 662 y(in)15 b(order)i(to)f(decide)h(when)f(to)g (free)h(it.)24 b(Also,)15 b(one)h(ma)o(y)e(implemen)o(t)g(constructors)k(of)d (deriv)o(ed)h(datat)o(yp)q(es)h(so)75 711 y(that)e(they)h(k)o(eep)g(p)q(oin)o (ters)g(to)f(their)h(datat)o(yp)q(e)f(argumen)o(ts,)f(rather)j(then)e(cop)o (ying)g(them.)21 b(In)16 b(suc)o(h)g(case,)g(one)75 761 y(needs)e(to)e(k)o (eep)h(trac)o(k)f(of)g(activ)o(e)g(datat)o(yp)q(e)g(de\014nition)g (references)j(in)d(order)h(to)f(kno)o(w)g(when)h(a)f(datat)o(yp)q(e)g(ob)r (ject)75 811 y(can)i(b)q(e)h(freed.)75 1014 y Fs(1.13.5)49 b(Use)16 b(of)h(general)f(datat)o(yp)q(es)g(in)h(comm)n(unication)75 1100 y FA(Handles)g(to)f(deriv)o(ed)h(datat)o(yp)q(es)f(can)h(b)q(e)f(passed) h(to)f(a)g(comm)o(unication)h(call)g(wherev)o(er)f(a)g(datat)o(yp)q(e)75 1157 y(argumen)o(t)11 b(is)h(required.)19 b(A)12 b(call)g(of)f(the)h(form)e Fv(MPI)p 947 1157 14 2 v 16 w(SEND\(buf,)j(count,)g(datat)o(yp)q(e)g(,)f (...\))p FA(,)e(where)i Fv(count)i Fi(>)75 1213 y Fv(1)p FA(,)e(is)h(in)o (terpreted)g(as)g(if)g(the)f(call)i(w)o(as)d(passed)i(a)f(new)h(datat)o(yp)q (e)f(whic)o(h)h(is)g(the)g(concatenation)g(of)f Fv(count)75 1270 y FA(copies)k(of)f Fv(datat)o(yp)q(e)p FA(.)21 b(Th)o(us,)166 1326 y Fv(MPI)p 251 1326 V 16 w(SEND\(buf,)15 b(count,)i(datat)o(yp)q(e,)f (dest,)g(tag,)f(comm)m(\))d FA(is)k(equiv)m(alen)o(t)h(to)75 1424 y Ft(MPI_TYPE_CONTIGUOUS\(count,)j(datatype,)j(newtype\))75 1480 y(MPI_TYPE_COMMIT\(newtype\))75 1537 y(MPI_SEND\(buf,)f(1,)i(newtype,)e (dest,)i(tag,)f(comm\))166 1635 y FA(The)14 b(same)g(applies)h(to)f(all)h (other)e(comm)o(unication)i(functions)g(that)e(ha)o(v)o(e)g(a)h Fv(count)h FA(and)f Fv(datat)o(yp)q(e)75 1691 y FA(argumen)o(t.)166 1748 y(Supp)q(ose)k(that)e(a)g(a)h(send)g(op)q(eration)g Fv(MPI)p 920 1748 V 16 w(SEND\(buf,)h(count,)g(datat)o(yp)q(e.)26 b(dest,)18 b(tag,)f(comm)m(\))c FA(is)75 1804 y(executed,)j(where)f Fv(datat)o(yp)q(e)i FA(ha)o(v)o(e)e(t)o(yp)q(e)g(map)600 1908 y Fl(f)p FA(\()p Fi(ty)r(pe)725 1915 y Fj(0)743 1908 y Fi(;)8 b(disp)848 1915 y Fj(0)865 1908 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 1915 y Fg(n)p Fh(\000)p Fj(1)1128 1908 y Fi(;)g(disp)1233 1915 y Fg(n)p Fh(\000)p Fj(1)1297 1908 y FA(\))p Fl(g)p Fi(;)166 2003 y FA(with)20 b(exten)o(t)f Fi(extent)i FA(\(empt)o(y)e(en)o(tries)g (deleted\).)34 b(Then)20 b(the)g(send)g(bu\013er)f(used)h(b)o(y)g(the)f(send) 75 2060 y(op)q(eration)c(consists)h(of)f Fi(n)10 b Fl(\001)g Fv(count)16 b FA(en)o(tries,)f(where)h(en)o(try)f Fi(i)9 b Fl(\001)h Fi(n)h FA(+)f Fi(j)17 b FA(is)f(at)f(lo)q(cation)653 2163 y Fi(addr)746 2170 y Fg(i;j)796 2163 y FA(=)e Fv(buf)h FA(+)d Fi(extent)g Fl(\001)f Fi(i)g FA(+)g Fi(disp)1280 2170 y Fg(j)166 2258 y FA(and)15 b(has)h(t)o(yp)q(e)f Fi(ty)r(pe)522 2265 y Fg(j)540 2258 y FA(,)g(for)f Fi(i)f FA(=)g(0)p Fi(;)8 b(:::;)g Fv(count)h Fl(\000)h FA(1)15 b(and)h Fi(j)f FA(=)e(0)p Fi(;)8 b(:::;)g(n)f Fl(\000)k FA(1.)20 b(These)15 b(en)o(tries)h(need)g(not) 75 2315 y(b)q(e)g(con)o(tiguous,)f(nor)g(distinct;)h(their)f(order)g(can)h(b) q(e)g(arbitrary)l(.)166 2371 y(The)g(v)m(ariable)h(stored)e(at)g(address)g Fi(addr)879 2378 y Fg(i;j)933 2371 y FA(in)h(the)g(calling)h(program)e (should)h(b)q(e)h(of)e(a)g(t)o(yp)q(e)h(that)75 2428 y(matc)o(hes)d Fi(ty)r(pe)333 2435 y Fg(j)351 2428 y FA(,)h(where)g(t)o(yp)q(e)g(matc)o (hing)g(is)g(de\014ned)i(as)d(in)i(section)f(1.5.)19 b(The)14 b(message)f(sen)o(t)h(con)o(tains)75 2484 y Fi(n)c Fl(\001)g Fv(count)17 b FA(en)o(tries,)e(where)g(en)o(try)g Fi(i)10 b Fl(\001)g Fi(n)g FA(+)g Fi(j)18 b FA(has)d(t)o(yp)q(e)g Fi(ty)r(pe)1100 2491 y Fg(j)1118 2484 y FA(.)166 2541 y(Similarly)l(,)g(supp)q(ose)f(that)e (a)h(receiv)o(e)h(op)q(eration)f Fv(MPI)p 1100 2541 V 16 w(RECV\(buf,)h (count,)g(datat)o(yp)q(e,)h(source,)e(tag,)75 2597 y(comm)m(,)f(status\))17 b FA(is)e(executed,)h(where)g Fv(datat)o(yp)q(e)g FA(has)g(t)o(yp)q(e)f(map) 600 2704 y Fl(f)p FA(\()p Fi(ty)r(pe)725 2711 y Fj(0)743 2704 y Fi(;)8 b(disp)848 2711 y Fj(0)865 2704 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 2711 y Fg(n)p Fh(\000)p Fj(1)1128 2704 y Fi(;)g(disp)1233 2711 y Fg(n)p Fh(\000)p Fj(1)1297 2704 y FA(\))p Fl(g)p Fi(;)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 56 56 bop 75 -100 a FA(56)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y FA(with)c(exten)o(t)f Fi(extent)i FA(\(empt)o(y)e(en)o(tries)i(deleted\).)19 b(Then)12 b(the)g(receiv)o(e)h(bu\013er)f(used)g(for)f(this)h(receiv)o(e)75 106 y(op)q(eration)j(consists)h(of)e(the)h Fi(n)c Fl(\001)e Fv(count)16 b FA(en)o(tries,)f(where)h(en)o(try)e Fi(i)c Fl(\001)f Fi(n)i FA(+)f Fi(j)17 b FA(is)f(at)e(lo)q(cation)i Fv(buf)e FA(+)c Fi(extent)h Fl(\001)75 162 y Fi(i)d FA(+)h Fi(disp)227 169 y Fg(j)259 162 y FA(and)15 b(has)f(t)o(yp)q(e)h Fi(ty)r(pe)613 169 y Fg(j)630 162 y FA(.)20 b(If)15 b(the)f(incoming)i(message)e(consists)h (of)f Fi(k)i FA(elemen)o(ts,)e(then)h(w)o(e)g(m)o(ust)75 219 y(ha)o(v)o(e)f Fi(k)g Fl(\024)f Fi(n)d Fl(\001)f Fv(count)p FA(;)15 b(the)g Fi(i)9 b Fl(\001)g Fi(n)h FA(+)f Fi(j)s FA(-th)15 b(elemen)o(t)g(of)g(the)f(message)h(should)h(ha)o(v)o(e)e(a)g(t)o(yp)q(e)h (that)f(matc)o(hes)75 275 y Fi(ty)r(pe)159 282 y Fg(j)177 275 y FA(.)166 332 y(Note)d(that)g(t)o(yp)q(e)h(matc)o(hing)f(is)h(de\014ned)h (according)f(to)f(the)h(t)o(yp)q(e)f(signature)h(of)f(the)h(corresp)q(onding) 75 389 y(datat)o(yp)q(es,)f(i.e.,)h(the)f(sequence)i(of)e(basic)h(t)o(yp)q(e) g(comp)q(onen)o(ts,)f(and)h(do)q(es)g(not)f(dep)q(end)i(on)e(the)g(datat)o (yp)q(e)75 445 y(de\014nition.)166 502 y(Example:)75 599 y Ft(...)75 655 y(CALL)23 b(MPI_TYPE_CONTIGUOUS\()e(2,)j(MPI_REAL,)f(type2,)g (...\))75 712 y(CALL)g(MPI_TYPE_CONTIGUOUS\()e(4,)j(MPI_REAL,)f(type4,)g (...\))75 768 y(CALL)g(MPI_TYPE_CONTIGUOUS\()e(2,)j(type2,)f(type22,)g(...\)) 75 825 y(...)75 881 y(CALL)g(MPI_SEND\()g(a,)h(4,)f(MPI_REAL,)g(...\))75 938 y(CALL)g(MPI_SEND\()g(a,)h(2,)f(type2,)g(...\))75 994 y(CALL)g (MPI_SEND\()g(a,)h(1,)f(type22,)g(...\))75 1051 y(CALL)g(MPI_SEND\()g(a,)h (1,)f(type4,)g(...\))75 1107 y(...)75 1163 y(CALL)g(MPI_RECV\()g(a,)h(4,)f (MPI_REAL,)g(...\))75 1220 y(CALL)g(MPI_RECV\()g(a,)h(2,)f(type2,)g(...\))75 1276 y(CALL)g(MPI_RECV\()g(a,)h(1,)f(type22,)g(...\))75 1333 y(CALL)g(MPI_RECV\()g(a,)h(1,)f(type4,)g(...\))166 1429 y FA(F)l(rom)14 b(a)h(t)o(yp)q(e)h(matc)o(hing)f(viewp)q(oin)o(t)h(an)o(y)f(of)g(the)g(sends) h(legally)g(matc)o(hes)f(an)o(y)g(of)g(the)g(receiv)o(es.)166 1486 y(A)22 b(datat)o(yp)q(e)f(ma)o(y)g(sp)q(ecify)i(o)o(v)o(erlapping)g(en)o (tries.)40 b(If)22 b(suc)o(h)h(datat)o(yp)q(e)e(is)h(used)h(in)f(a)g(receiv)o (e)75 1543 y(op)q(eration,)17 b(i.e.,)f(if)h(the)f(some)g(part)g(of)g(the)g (receiv)o(e)i(bu\013er)e(is)h(o)o(v)o(erwritten)f(more)g(than)g(once)h(b)o(y) f(the)75 1599 y(receiv)o(e)g(op)q(eration,)f(then)h(the)f(outcome)g(of)g(the) g(receiv)o(e)h(is)g(unde\014ned.)166 1656 y(Supp)q(ose,)j(again,)f(that)f Fv(MPI)p 683 1656 14 2 v 15 w(RECV\(buf,)j(count,)f(datat)o(yp)q(e,)g(dest,)g (tag,)f(comm)m(,)d(status\))k FA(is)f(exe-)75 1712 y(cuted,)d(where)h Fv(datat)o(yp)q(e)h FA(has)e(t)o(yp)q(e)g(map)600 1826 y Fl(f)p FA(\()p Fi(ty)r(pe)725 1833 y Fj(0)743 1826 y Fi(;)8 b(disp)848 1833 y Fj(0)865 1826 y FA(\))p Fi(;)g(:::;)g FA(\()p Fi(ty)r(pe)1066 1833 y Fg(n)p Fh(\000)p Fj(1)1128 1826 y Fi(;)g(disp)1233 1833 y Fg(n)p Fh(\000)p Fj(1)1297 1826 y FA(\))p Fl(g)p Fi(:)166 1930 y FA(The)k(receiv)o(ed)g(message)f(need)h(not)f(\014ll)i(all)g(the)e (receiv)o(e)h(bu\013er,)g(nor)f(do)q(es)h(it)g(need)g(to)f(\014ll)i(a)e(n)o (um)o(b)q(er)75 1987 y(of)i(lo)q(cations)h(whic)o(h)g(is)f(a)g(m)o(ultiple)i (of)e Fi(n)p FA(;)h(an)o(y)f(n)o(um)o(b)q(er)g Fi(k)i FA(of)d(basic)i(elemen) o(ts)g(can)f(b)q(e)h(receiv)o(ed,)h(where)75 2043 y(0)d Fl(\024)h Fi(k)h Fl(\024)f Fv(count)e Fl(\001)f Fi(n)p FA(.)166 2100 y(The)k(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i(receiv)o(ed)g(can)f(b)q(e)g (retriev)o(ed)g(from)f Fv(status)j FA(using)e(the)g(query)g(function)75 2157 y Fv(MPI)p 160 2157 V 16 w(GET)p 264 2157 V 17 w(ELEMENTS)p FA(:)75 2261 y Fv(MPI)p 160 2261 V 16 w(GET)p 264 2261 V 17 w(ELEMENTS\()h(status,)h(datat)o(yp)q(e,)h(count\))117 2339 y Fu(IN)155 b Fv(status)476 b Fu(return)15 b(status)g(of)e(receiv)o(e)i(op)q (eration)f(\(Status\))117 2415 y(IN)155 b Fv(datat)o(yp)q(e)424 b Fu(datat)o(yp)q(e)14 b(used)h(b)o(y)f(receiv)o(e)h(op)q(eration)f (\(handle\))117 2491 y(OUT)108 b Fv(count)482 b Fu(n)o(um)o(b)q(er)13 b(of)h(receiv)o(ed)h(basic)f(elemen)o(ts)g(\(in)o(teger\))75 2617 y Ft(int)23 b(MPI)p 245 2617 15 2 v 17 w(Get)p 334 2617 V 17 w(elements\(MPI)p 639 2617 V 16 w(Status)g(status,)f(MPI)p 1084 2617 V 17 w(Datatype)h(datatype,)g(int)g(*count\))75 2704 y(MPI)p 150 2704 V 17 w(GET)p 239 2704 V 17 w(ELEMENTS\(STATUS,)e(DATATYPE,)i (COUNT,)g(IERROR\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 57 57 bop 75 -100 a Fw(1.13.)34 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1119 b FA(57)170 49 y Ft(INTEGER)23 b(STATUS\(MPI)p 604 49 15 2 v 16 w(STATUS)p 764 49 V 17 w(SIZE\),)g(DATATYPE,)f(COUNT,)h(IERROR)166 136 y FA(The)13 b(previously)h(used)f(function)h Fv(MPI)p 834 136 14 2 v 16 w(GET)p 938 136 V 16 w(COUNT)g FA(has)f(a)f(di\013eren)o(t)h(b) q(eha)o(vior:)19 b(it)13 b(returns)g(the)75 192 y(n)o(um)o(b)q(er)g(of)f (\\top)g(elemen)o(ts")h(receiv)o(ed.)21 b(In)13 b(the)g(previous)g(example,)h Fv(MPI)p 1368 192 V 16 w(GET)p 1472 192 V 16 w(COUNT)g FA(ma)o(y)e(return)75 249 y(an)o(y)i(in)o(teger)h(v)m(alue)g Fi(k)q FA(,)g(where)f(0)f Fl(\024)g Fi(k)g Fl(\024)g Fv(count)r FA(.)19 b(If)c Fv(MPI)p 1047 249 V 16 w(GET)p 1151 249 V 16 w(COUNT)h FA(returns)e Fi(k)q FA(,)g(then)h(the)g(n)o(um)o(b)q(er)75 305 y(of)f(basic)h(elemen)o(ts) f(receiv)o(ed)i(\(and)e(the)g(v)m(alue)h(returned)g(b)o(y)f Fv(MPI)p 1226 305 V 16 w(GET)p 1330 305 V 16 w(ELEMENTS)p FA(\))g(is)h Fi(n)8 b Fl(\001)g Fi(k)q FA(.)19 b(If)c(the)75 362 y(n)o(um)o(b)q(er)j(of)f (basic)h(elemen)o(ts)g(receiv)o(ed)h(is)f(not)f(a)h(m)o(ultiple)h(of)e Fi(n)p FA(,)h(i.e.)27 b(if)18 b(the)g(receiv)o(e)g(op)q(eration)g(has)75 418 y(not)f(receiv)o(ed)i(an)f(in)o(tegral)g(n)o(um)o(b)q(er)g(of)f Fv(datat)o(yp)q(e)j FA(\\copies",)e(then)g Fv(MPI)p 1360 418 V 16 w(GET)p 1464 418 V 17 w(COUNT)g FA(returns)g(the)75 475 y(v)m(alue)e Fo(MPI)p 269 475 13 2 v 15 w(UNDEFINED)p FA(.)166 531 y(Example:)75 639 y Ft(...)75 695 y(CALL)23 b(MPI_TYPE_CONTIGUOUS\(2,)e (MPI_REAL,)i(Type2,)g(ierr\))75 752 y(CALL)g(MPI_COMMIT\(Type2,)f(ierr\))75 808 y(...)75 865 y(CALL)h(MPI_RANK\(comm,)f(rank,)h(ierr\))75 921 y(IF\(rank.EQ.0\))147 977 y(THEN)218 1034 y(CALL)g(MPI_SEND\(a,)g(2,)g (MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))218 1090 y(CALL)g(MPI_SEND\(a,)g(3,)g (MPI_REAL,)g(1,)h(0,)f(comm,)g(ierr\))147 1147 y(ELSE)218 1203 y(CALL)g(MPI_RECV\(a,)g(2,)g(Type2,)g(0,)h(0,)g(comm,)f(stat,)g(ierr\))218 1260 y(CALL)g(MPI_GET_COUNT\(stat,)f(Type2,)h(i,)g(ierr\))119 b(!)24 b(returns)f(i=1)218 1316 y(CALL)g(MPI_GET_ELEMENTS\(stat,)e(Type2,)i (i,)h(ierr\))47 b(!)24 b(returns)f(i=2)218 1373 y(CALL)g(MPI_RECV\(a,)g(2,)g (Type2,)g(0,)h(0,)g(comm,)f(stat,)g(ierr\))218 1429 y(CALL)g (MPI_GET_COUNT\(stat,)f(Type2,)h(i,)g(ierr\))119 b(!)24 b(returns)f (i=MPI_UNDEFINED)218 1486 y(CALL)g(MPI_GET_ELEMENTS\(stat,)e(Type2,)i(i,)h (ierr\))47 b(!)24 b(returns)f(i=3)75 1542 y(END)g(IF)166 1649 y FA(The)15 b(function)h Fv(MPI)p 522 1649 14 2 v 16 w(GET)p 626 1649 V 17 w(ELEMENTS)f FA(can)g(also)g(b)q(e)g(used)h(after)e(a)h(prob)q (e)g(to)g(\014nd)h(the)f(n)o(um)o(b)q(er)75 1706 y(of)20 b(elemen)o(ts)h(in)g (the)f(prob)q(ed)h(message.)34 b(Note)20 b(that)f(the)h(t)o(w)o(o)f (functions)i Fv(MPI)p 1507 1706 V 16 w(GET)p 1611 1706 V 17 w(COUNT)g FA(and)75 1762 y Fv(MPI)p 160 1762 V 16 w(GET)p 264 1762 V 17 w(ELEMENTS)15 b FA(return)g(the)g(same)g(v)m(alues)i(when)e(they)h (are)e(used)i(with)g(basic)g(datat)o(yp)q(es.)189 1869 y Fr(R)n(ationale.)38 b FA(The)14 b(extension)g(giv)o(en)g(to)f(the)h(de\014nition)h(of)e Fv(MPI)p 1303 1869 V 16 w(GET)p 1407 1869 V 16 w(COUNT)i FA(seems)e(natural:) 189 1926 y(one)j(w)o(ould)h(exp)q(ect)h(this)f(function)g(to)f(return)h(the)f (v)m(alue)i(of)e(the)h Fv(count)h FA(argumen)o(t,)e(when)h(the)189 1982 y(receiv)o(e)e(bu\013er)f(is)h(\014lled.)22 b(Often)15 b(times)f Fv(datat)o(yp)q(e)i FA(represen)o(ts)f(a)f(basic)h(unit)g(of)f (data)g(one)g(w)o(an)o(ts)189 2039 y(to)19 b(transfer;)j(e.g.,)e(a)g(record)g (in)h(an)f(arra)o(y)f(of)h(records)g(\(structures\).)34 b(One)21 b(should)g(b)q(e)g(able)189 2095 y(to)f(\014nd)i(out)f(ho)o(w)f(man)o(y)h (comp)q(onen)o(ts)g(where)g(receiv)o(ed)i(without)e(b)q(othering)h(to)e (divide)j(b)o(y)189 2152 y(the)18 b(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(in)f (eac)o(h)g(comp)q(onen)o(t.)30 b(Ho)o(w)o(ev)o(er,)18 b(on)h(other)f(o)q (ccasions,)h Fv(datat)o(yp)q(e)189 2208 y FA(is)h(used)h(to)f(de\014ned)h(a)f (complex)h(la)o(y)o(out)f(of)g(data)f(in)i(the)g(receiv)o(er)g(memory)l(,)f (and)h(do)q(es)f(not)189 2265 y(represen)o(t)i(a)g(basic)h(unit)g(of)f(data)g (for)f(transfers.)41 b(In)23 b(suc)o(h)f(cases,)i(one)e(needs)i(to)d(use)i (the)189 2321 y(function)16 b Fv(MPI)p 452 2321 V 15 w(GET)p 555 2321 V 17 w(ELEMENTS)p FA(.)f(\()p Fr(End)h(of)g(r)n(ationale.)p FA(\))166 2504 y Fq(Implemen)o(tati)o(on)i(note:)58 b Fu(The)19 b(curren)o(t)i(de\014nition)e(implies)e(that)i(a)g(receiv)o(e)h(cannot)g(c)o (hange)f(the)75 2554 y(v)n(alue)d(of)g(storage)h(b)q(et)o(w)o(een)h(the)f(en) o(tries)g(de\014ned)h(to)e(comp)q(ose)g(the)h(comm)o(unication)c(bu\013er.)27 b(In)17 b(particular,)75 2604 y(the)c(de\014nition)g(implies)e(that)i (padding)f(space)i(in)e(a)h(structure)i(should)d(not)h(b)q(e)g(mo)q(di\014ed) f(when)h(suc)o(h)h(structure)75 2654 y(is)f(copied)g(from)f(one)h(pro)q(cess) i(to)e(another.)18 b(This)13 b(w)o(ould)f(prev)o(en)o(ts)j(the)f(ob)o(vious)e (optimization)e(of)j(cop)o(ying)f(the)75 2704 y(structure,)i(together)f(with) e(the)i(padding,)e(as)h(one)g(con)o(tiguous)f(blo)q(c)o(k.)18 b(The)12 b(implemen)o(tatio)o(n)d(is)j(free)h(to)e(do)h(this)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 58 58 bop 75 -100 a FA(58)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Fu(optimization)d(when)j(it)f(do)q (es)i(not)e(impact)f(the)i(outcome)f(of)g(the)h(computation.)22 b(The)16 b(user)g(can)g(\\force")g(this)75 99 y(optimization)11 b(b)o(y)j(explicitly)f(including)g(padding)g(as)h(part)g(of)f(the)h(message.) 75 302 y Fs(1.13.6)49 b(Co)o(rrect)15 b(use)h(of)g(addresses)75 388 y FA(Successiv)o(ely)23 b(declared)f(v)m(ariables)h(in)e(C)g(or)g(F)l (ortran)e(are)i(not)g(necessarily)h(stored)f(at)f(con)o(tiguous)75 444 y(lo)q(cations.)i(Th)o(us,)16 b(care)g(m)o(ust)f(b)q(e)h(exercised)i (that)d(displacemen)o(ts)i(do)f(not)f(cross)h(from)f(one)h(v)m(ariable)75 501 y(to)e(another.)20 b(Also,)15 b(in)g(mac)o(hines)h(with)f(segmen)o(ted)g (address)g(space,)g(addresses)g(are)g(not)f(unique)j(and)75 557 y(address)11 b(arithmetic)h(has)g(some)f(p)q(eculiar)i(prop)q(erties.)19 b(Th)o(us,)12 b(use)g(of)f Fz(addresses)p FA(,)f(i.e.)20 b(displacemen)o(ts) 75 614 y(relativ)o(e)c(to)e(the)i(start)e(address)29 b Fo(MPI)p 733 614 13 2 v 14 w(BOTTOM)p FA(,)14 b(has)h(to)g(b)q(e)h(restricted.)166 670 y(V)l(ariables)g(b)q(elong)f(to)f(the)g(same)g Fz(sequen)o(tial)k (storage)c FA(if)h(they)f(b)q(elong)i(to)e(the)g(same)g(arra)o(y)l(,)g(to)75 727 y(the)k(same)g Fv(COMMON)g FA(blo)q(c)o(k)h(in)g(F)l(ortran,)e(or)h(to)f (the)h(same)g(structure)f(in)i(C.)f(V)l(alid)h(addresses)f(are)75 783 y(de\014ned)f(recursiv)o(ely)f(as)f(follo)o(ws:)131 878 y(1.)22 b(The)g(function)g Fv(MPI)p 558 878 14 2 v 16 w(ADDRESS)h FA(returns)e(a)h(v)m(alid)h(address,)h(when)e(passed)g(as)f(argumen)o(t)g(a) 189 935 y(v)m(ariable)16 b(of)f(the)g(calling)i(program.)131 1025 y(2.)22 b(The)d Fv(buf)h FA(argumen)o(t)f(of)g(a)g(comm)o(unication)h (function)g(ev)m(aluates)h(to)d(a)h(v)m(alid)i(address,)f(when)189 1082 y(passed)15 b(as)g(argumen)o(t)f(a)h(v)m(ariable)i(of)e(the)g(calling)i (program.)131 1172 y(3.)22 b(If)15 b Fv(v)g FA(is)h(a)f(v)m(alid)i(address,)d (and)i Fv(i)f FA(is)g(an)h(in)o(teger,)f(then)g Fv(v+i)g FA(is)h(a)f(v)m (alid)i(address,)e(pro)o(vided)h Fv(v)f FA(and)189 1229 y Fv(v+i)g FA(are)g(in)h(the)f(same)g(sequen)o(tial)h(storage.)131 1319 y(4.)22 b(If)15 b Fv(v)g FA(is)h(a)f(v)m(alid)i(address)e(then)29 b Fo(MPI)p 820 1319 13 2 v 15 w(BOTTOM)13 b(+)h(v)h FA(is)h(a)f(v)m(alid)i (address.)166 1414 y(A)22 b(correct)g(program)g(uses)g(only)h(v)m(alid)i (addresses)d(to)g(iden)o(tify)i(the)e(lo)q(cations)h(of)f(en)o(tries)h(in)75 1471 y(comm)o(unication)16 b(bu\013ers.)k(F)l(urthermore,)14 b(if)h Fv(u)h FA(and)f Fv(v)g FA(are)g(t)o(w)o(o)f(v)m(alid)j(addresses,)e (then)g(the)g(\(in)o(teger\))75 1527 y(di\013erence)g Fv(u)f(-)g(v)g FA(can)g(b)q(e)g(computed)g(only)h(if)f(b)q(oth)g Fv(u)g FA(and)g Fv(v)g FA(are)f(in)i(the)f(same)f(sequen)o(tial)i(storage;)e(no)75 1584 y(other)i(arithmetic)h(op)q(erations)f(can)g(b)q(e)h(meaningfully)h (executed)g(on)e(addresses.)166 1640 y(The)41 b(ab)q(o)o(v)o(e)20 b(rules)h(imp)q(ose)g(no)f(constrain)o(ts)g(in)h(the)f(use)h(of)f(deriv)o(ed) h(datat)o(yp)q(es,)g(as)f(long)g(as)-1903 b Fk(\))75 1697 y FA(they)18 b(are)f(used)h(to)f(de\014ne)h(a)g(comm)o(unication)g(bu\013er)f (that)g(is)h(wholly)h(con)o(tained)f(within)h(the)e(same)75 1753 y(sequen)o(tial)23 b(storage.)38 b(Ho)o(w)o(ev)o(er,)22 b(the)g(construction)g(of)f(a)h(comm)o(unication)g(bu\013er)g(that)f(con)o (tains)75 1810 y(v)m(ariables)f(that)f(are)g(not)g(within)h(the)f(same)g (sequen)o(tial)h(storage)e(m)o(ust)h(ob)q(ey)g(certain)h(restrictions.)75 1866 y(Basically)l(,)g(a)e(comm)o(unication)h(bu\013er)f(with)g(v)m(ariables) i(that)d(are)h(not)g(within)h(the)f(same)g(sequen)o(tial)75 1923 y(storage)g(can)h(b)q(e)h(used)g(only)g(b)o(y)f(sp)q(ecifying)i(in)f (the)g(comm)o(unication)g(call)g Fv(buf)g(=)g(MPI)p 1651 1923 14 2 v 16 w(BOTTOM)p FA(,)75 1979 y Fv(count)i(=)g(1)p FA(,)g(and)f(using)h (a)f Fv(datat)o(yp)q(e)i FA(argumen)o(t)d(where)i(all)g(displacemen)o(ts)h (are)d(v)m(alid)j(\(absolute\))75 2035 y(addresses.)d(A)15 b(datat)o(yp)q(e)f(with)h(displacemen)o(ts)h(that)e(are)h(absolute)g (addresses)g(not)f(within)i(the)f(same)75 2092 y(sequen)o(tial)h(storage)e (can)i(only)f(b)q(e)h(constructed)g(using)g Fv(MPI)p 1139 2092 V 15 w(TYPE)p 1271 2092 V 17 w(STRUCT)p FA(.)189 2187 y Fr(A)n(dvic)n(e)f(to) i(users.)39 b FA(It)16 b(is)f(not)g(exp)q(ected)i(that)d(MPI)h(implemen)o (tations)i(will)f(b)q(e)g(able)g(to)f(detect)189 2244 y(erroneous,)22 b(\\out)f(of)g(b)q(ound")h(displacemen)o(ts)h({)f(unless)g(those)f(o)o(v)o (er\015o)o(w)g(the)g(user)h(address)189 2300 y(space)15 b({)h(since)g(the)g (MPI)g(call)g(ma)o(y)f(not)g(kno)o(w)g(the)h(exten)o(t)f(of)g(the)h(arra)o (ys)e(and)i(records)f(in)i(the)189 2356 y(host)d(program.)19 b(\()p Fr(End)d(of)g(advic)n(e)g(to)h(users.)p FA(\))166 2534 y Fq(Implemen)o(tati)o(on)d(note:)41 b Fu(There)16 b(is)f(no)g(need)h(to)f (distinguish)g(\(absolute\))h(addresses)h(and)e(\(relativ)o(e\))75 2591 y(displacemen)o(ts)h(on)h(a)g(mac)o(hine)e(with)i(con)o(tiguous)f (address)i(space:)25 b Fo(MPI)p 1274 2591 13 2 v 15 w(BOTTOM)16 b Fu(is)h(zero,)h(and)e(b)q(oth)h(ad-)75 2647 y(dresses)d(and)e(displacemen)o (ts)g(are)g(in)o(tegers.)18 b(On)13 b(mac)o(hines)e(where)i(the)f (distinction)g(is)g(required,)g(addresses)j(are)75 2704 y(recognized)g(as)f (expressions)h(that)f(in)o(v)o(olv)o(e)f Fo(MPI)p 852 2704 V 14 w(BOTTOM)p Fu(.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 59 59 bop 75 -100 a Fw(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b FA(59)75 49 y Fs(1.13.7)49 b(Examples)75 137 y FA(The)15 b(follo)o(wing)h(examples)g(illustrate)h(the)e(use)g(of)g(deriv)o(ed)h(datat) o(yp)q(es.)75 263 y Fv(First)f(example:)43 b FA(Send)16 b(and)g(receiv)o(e)g (a)e(section)i(of)f(a)g(3D)g(arra)o(y)l(.)218 376 y Ft(REAL)23 b(a\(100,100,100\),)f(e\(9,9,9\))218 432 y(INTEGER)h(oneslice,)g(twoslice,)f (threeslice,)h(sizeofreal,)f(myrank,)h(ierr)218 489 y(MPI_STATUS)g(status)75 602 y(C)143 b(extract)23 b(the)g(section)g(a\(1:17:2,)g(3:11,)g(2:10\))75 658 y(C)143 b(and)23 b(store)h(it)f(in)h(e\(*,*,*\).)218 771 y(CALL)f(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))218 884 y(CALL)j(MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 997 y(C)119 b(create)23 b(datatype)g(for)g(a)h(1D)g(section)218 1053 y(CALL)f(MPI_TYPE_VECTOR\()f(9,)i(1,)f(2,)h(MPI_REAL,)f(oneslice,)f (ierr\))75 1166 y(C)119 b(create)23 b(datatype)g(for)g(a)h(2D)g(section)218 1223 y(CALL)f(MPI_TYPE_HVECTOR\(9,)f(1,)h(100*sizeofreal,)f(oneslice,)h (twoslice,ierr\))75 1336 y(C)119 b(create)23 b(datatype)g(for)g(the)h(entire) f(section)218 1392 y(CALL)g(MPI_TYPE_HVECTOR\()f(9,)i(1,)f (100*100*sizeofreal,)e(twoslice,)i(1,)767 1449 y(threeslice,)f(ierr\))218 1561 y(CALL)h(MPI_TYPE_COMMIT\()f(threeslice,)h(ierr\))218 1618 y(CALL)g(MPI_SENDRECV\(a\(1,3,2\),)e(1,)j(threeslice,)e(myrank,)h(0,)h (e,)f(9*9*9,)648 1674 y(MPI_REAL,)f(myrank,)h(0,)h(MPI_COMM_WORLD,)e(status,) h(ierr\))75 1801 y Fv(Second)17 b(example:)42 b FA(Cop)o(y)15 b(the)g(\(strictly\))g(lo)o(w)o(er)g(triangular)g(part)g(of)g(a)g(matrix.)218 1913 y Ft(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 1970 y(INTEGER)47 b(disp\(100\),)22 b(blocklen\(100\),)g(ltype,)h(myrank,)g(ierr)218 2026 y(MPI_STATUS)g(status)75 2139 y(C)119 b(copy)23 b(lower)h(triangular)e (part)h(of)h(array)f(a)75 2195 y(C)119 b(onto)23 b(lower)h(triangular)e(part) h(of)h(array)f(b)218 2308 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\)) 75 2421 y(C)119 b(compute)23 b(start)g(and)h(size)f(of)h(each)f(column)218 2478 y(DO)h(i=1,)f(100)266 2534 y(disp\(i\))g(=)h(100*\(i-1\))e(+)i(i)266 2591 y(block\(i\))f(=)g(100-i)218 2647 y(END)h(DO)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 60 60 bop 75 -100 a FA(60)613 b Fw(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(C)119 b(create)23 b(datatype)g(for)g(lower)h(triangular)e(part)218 106 y(CALL)h (MPI_TYPE_INDEX\()f(100,)i(block,)f(disp,)g(MPI_REAL,)f(ltype,)i(ierr\))218 219 y(CALL)f(MPI_TYPE_COMMIT\(ltype,)e(ierr\))218 275 y(CALL)i (MPI_SENDRECV\()g(a,)g(1,)h(ltype,)f(myrank,)g(0,)g(b,)h(1,)552 332 y(ltype,)f(myrank,)g(0,)h(MPI_COMM_WORLD,)e(status,)h(ierr\))75 458 y Fv(Third)15 b(example:)43 b FA(T)l(ransp)q(ose)15 b(a)g(matrix)218 572 y Ft(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 628 y(INTEGER)g(row,)g (xpose,)g(sizeofreal,)g(myrank,)g(ierr)218 685 y(MPI_STATUS)g(status)75 798 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 911 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))218 1024 y(CALL)j(MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 1137 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row)218 1193 y(CALL)f(MPI_TYPE_VECTOR\()f(100,)h(1,)h(100,)f(MPI_REAL,)g(row,)g(ierr\))75 1306 y(C)119 b(create)23 b(datatype)g(for)g(matrix)g(in)h(row-major)f(order) 218 1362 y(CALL)g(MPI_TYPE_HVECTOR\()f(100,)h(1,)h(sizeofreal,)e(row,)i (xpose,)f(ierr\))218 1475 y(CALL)g(MPI_TYPE_COMMIT\()f(xpose,)h(ierr\))75 1588 y(C)119 b(send)23 b(matrix)h(in)f(row-major)g(order)g(and)g(receive)g (in)h(column)f(major)g(order)218 1645 y(CALL)g(MPI_SENDRECV\()g(a,)g(1,)h (xpose,)f(myrank,)g(0,)g(b,)h(100*100,)457 1701 y(MPI_REAL,)e(myrank,)h(0,)h (MPI_COMM_WORLD,)e(status,)h(ierr\))166 1813 y FA(Another)15 b(approac)o(h)g(to)f(the)i(transp)q(ose)f(problem:)218 1913 y Ft(REAL)23 b(a\(100,100\),)g(b\(100,100\))218 1970 y(INTEGER)47 b(disp\(2\),)23 b(blocklen\(2\),)f(type\(2\),)h(row,)g(row1,)g(sizeofreal)218 2026 y(INTEGER)47 b(myrank,)23 b(ierr)218 2083 y(MPI_STATUS)g(status)218 2195 y(CALL)g(MPI_COMM_RANK\(MPI_COMM_WORLD,)d(myrank\))75 2308 y(C)119 b(transpose)23 b(matrix)g(a)h(onto)f(b)218 2421 y(CALL)g(MPI_TYPE_EXTENT\()f(MPI_REAL,)h(sizeofreal,)f(ierr\))75 2534 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row)218 2591 y(CALL)f(MPI_TYPE_VECTOR\()f(100,)h(1,)h(100,)f(MPI_REAL,)g(row,)g(ierr\))75 2704 y(C)119 b(create)23 b(datatype)g(for)g(one)h(row,)f(with)h(the)f(extent) g(of)h(one)f(real)g(number)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 61 61 bop 75 -100 a Fw(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b FA(61)218 49 y Ft(disp\(1\))23 b(=)h(0)218 106 y(disp\(2\))f(=)h (sizeofreal)218 162 y(type\(1\))47 b(=)24 b(row)218 219 y(type\(2\))47 b(=)24 b(MPI_UB)218 275 y(blocklen\(1\))46 b(=)24 b(1)218 332 y(blocklen\(2\))46 b(=)24 b(1)218 388 y(CALL)f(MPI_TYPE_STRUCT\()f(2,)i (blocklen,)e(disp,)i(type,)f(row1,)g(ierr\))218 501 y(CALL)g (MPI_TYPE_COMMIT\()f(row1,)h(ierr\))75 614 y(C)119 b(send)23 b(100)h(rows)f(and)h(receive)f(in)g(column)g(major)g(order)218 670 y(CALL)g(MPI_SENDRECV\()g(a,)g(100,)g(row1,)h(myrank,)f(0,)g(b,)h (100*100,)457 727 y(MPI_REAL,)e(myrank,)h(0,)h(MPI_COMM_WORLD,)e(status,)h (ierr\))75 847 y Fv(F)o(ourth)15 b(example:)43 b FA(manipulate)17 b(an)e(arra)o(y)f(of)g(structures.)75 953 y Ft(struct)23 b(Partstruct)147 1010 y({)147 1066 y(int)95 b(class;)47 b(/*)23 b(particle)g(class)g(*/)147 1123 y(double)g(d[6];)71 b(/*)23 b(particle)g(coordinates)f(*/)147 1179 y(char)71 b(b[7];)g(/*)23 b(some)g(additional)g(information)f(*/)147 1236 y(};)75 1349 y(struct)h(Partstruct)94 b(particle[1000];)75 1462 y(int)429 b(i,)24 b(dest,)f(rank;)75 1518 y(MPI_Comm)118 b(comm;)75 1687 y(/*)24 b(build)f(datatype)g(describing)f(structure)h(*/)75 1800 y(MPI_Datatype)f(Particletype;)75 1857 y(MPI_Datatype)g(type[3])h(=)h ({MPI_INT,)e(MPI_DOUBLE,)h(MPI_CHAR};)75 1913 y(int)238 b(blocklen[3])23 b(=)g({1,)h(6,)f(7};)75 1970 y(MPI_Aint)118 b(disp[3];)75 2139 y(/*)24 b(compute)e(displacements)h(of)g(structure)g(components)f(*/)75 2252 y(MPI_Address\()g(particle,)h(disp\);)75 2308 y(MPI_Address\()f (particle[0].d,)g(disp+1\);)75 2365 y(MPI_Address\()g(particle[0].b,)g (disp+2\);)75 2421 y(for)h(\(i=0;)h(i)f(<3;)h(i++\))f(disp[i])g(-=)h (disp[0];)75 2534 y(MPI_Type_struct\()e(3,)h(blocklen,)g(disp,)g(type,)g (&Particletype\);)147 2647 y(/*)g(If)h(compiler)f(does)g(padding)g(in)g (mysterious)g(ways,)147 2704 y(the)g(following)g(may)g(be)h(safer)f(*/)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 62 62 bop 75 -100 a FA(62)613 b Fw(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 106 y Ft(MPI_Datatype)22 b(type1[4])h(=)h({MPI_INT,)e(MPI_DOUBLE,)h(MPI_CHAR,)f(MPI_UB};)75 162 y(int)238 b(blocklen1[4])22 b(=)i({1,)g(6,)f(7,)h(1};)75 219 y(MPI_Aint)118 b(disp1[4];)75 332 y(/*)24 b(compute)e(displacements)h(of) g(structure)g(components)f(*/)75 444 y(MPI_Address\()g(particle,)h(disp1\);) 75 501 y(MPI_Address\()f(particle[0].d,)g(disp1+1\);)75 557 y(MPI_Address\()g(particle[0].b,)g(disp1+2\);)75 614 y(MPI_Address\()g (particle+1,)h(disp1+3\);)75 670 y(for)g(\(i=0;)h(i)f(<4;)h(i++\))f(disp1[i]) g(-=)g(disp1[0];)75 783 y(/*)h(build)f(datatype)g(describing)f(structure)h (*/)75 896 y(MPI_Type_struct\()f(4,)h(blocklen1,)g(disp1,)g(type1,)g (&Particletype\);)409 1065 y(/*)h(4.1:)266 1122 y(send)f(the)h(entire)f (array)g(*/)75 1235 y(MPI_Type_commit\()f(&Particletype\);)75 1291 y(MPI_Send\()h(particle,)f(1000,)h(Particletype,)g(dest,)g(tag,)g (comm\);)409 1461 y(/*)h(4.2:)266 1517 y(send)f(only)h(the)f(entries)g(of)g (class)h(zero)f(particles,)266 1574 y(preceded)g(by)g(the)h(number)f(of)g (such)h(entries)f(*/)75 1686 y(MPI_Datatype)f(Zparticles;)70 b(/*)24 b(datatype)f(describing)f(all)i(particles)791 1743 y(with)f(class)g(zero)h(\(needs)f(to)g(be)h(recomputed)791 1799 y(if)f(classes)g(change\))g(*/)75 1856 y(MPI_Datatype)f(Ztype;)75 1969 y(MPI_Aint)118 b(zdisp[1000];)75 2025 y(int)23 b(zblock[1000],)g(j,)g (k;)75 2082 y(int)g(zzblock[2])g(=)h({1,1};)75 2138 y(MPI_Aint)118 b(zzdisp[2];)75 2195 y(MPI_Datatype)22 b(zztype[2];)75 2307 y(/*)i(compute)e(displacements)h(of)g(class)g(zero)h(particles)e(*/)75 2364 y(j)i(=)f(0;)75 2420 y(for\(i=0;)g(i)g(<)h(1000;)f(i++\))123 2477 y(if)g(\(particle[i].class==0\))194 2533 y({)194 2590 y(zdisp[j])g(=)h(i;)194 2646 y(zblock[j])f(=)h(1;)194 2703 y(j++;)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 63 63 bop 75 -100 a Fw(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b FA(63)194 49 y Ft(})75 162 y(/*)24 b(create)f(datatype)f(for)i(class)f (zero)g(particles)47 b(*/)75 219 y(MPI_Type_indexed\()22 b(j,)h(zblock,)g (zdisp,)g(Particletype,)f(&Zparticles\);)75 332 y(/*)i(prepend)e(particle)h (count)g(*/)75 388 y(MPI_Address\(&j,)f(zzdisp\);)75 444 y (MPI_Address\(particle,)f(zzdisp+1\);)75 501 y(zztype[0])i(=)g(MPI_INT;)75 557 y(zztype[1])g(=)g(Zparticles;)75 614 y(MPI_Type_struct\(2,)e(zzblock,)i (zzdisp,)g(zztype,)g(&Ztype\);)75 727 y(MPI_Type_commit\()f(&Ztype\);)75 783 y(MPI_Send\()h(MPI_BOTTOM,)f(1,)i(Ztype,)f(dest,)g(tag,)g(comm\);)242 953 y(/*)h(A)f(probably)g(more)g(efficient)g(way)h(of)f(defining)g (Zparticles)f(*/)75 1065 y(/*)i(consecutive)e(particles)h(with)g(index)g (zero)g(are)h(handled)f(as)g(one)h(block)f(*/)75 1122 y(j=0;)75 1178 y(for)g(\(i=0;)h(i)f(<)h(1000;)f(i++\))123 1235 y(if)g (\(particle[i].index==0\))170 1291 y({)170 1348 y(for)h(\(k=i+1;)f(\(k)g(<)h (1000\)&&\(particle[k].index)d(==)i(0\))h(;)f(k++\);)170 1404 y(zdisp[j])g(=)h(i;)170 1461 y(zblock[j])f(=)h(k-i;)170 1517 y(j++;)170 1574 y(i)g(=)g(k;)170 1630 y(})75 1686 y(MPI_Type_indexed\()e(j,)h (zblock,)g(zdisp,)g(Particletype,)f(&Zparticles\);)457 1856 y(/*)h(4.3:)314 1912 y(send)g(the)g(first)h(two)f(coordinates)f(of)i(all)f (entries)g(*/)75 2025 y(MPI_Datatype)f(Allpairs;)142 b(/*)24 b(datatype)f(for)g(all)g(pairs)h(of)f(coordinates)g(*/)75 2138 y(MPI_Aint)g(sizeofentry;)75 2251 y(MPI_Type_extent\()f(Particletype,)g (sizeofentry\);)194 2364 y(/*)i(sizeofentry)e(can)i(also)f(be)h(computed)e (by)i(subtracting)e(the)i(address)266 2420 y(of)f(particle[0])g(from)g(the)h (address)f(of)g(particle[1])f(*/)75 2533 y(MPI_Type_hvector\()g(1000,)h(2,)g (sizeofentry,)g(MPI_REAL,)f(&Allpairs\);)75 2590 y(MPI_Type_commit\()g (&Allpairs\);)75 2646 y(MPI_Send\()h(particle.d,)f(1,)i(Allpairs,)e(dest,)h (tag,)h(comm\);)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 64 64 bop 75 -100 a FA(64)613 b Fw(CHAPTER)16 b(1.)29 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)218 49 y Ft(/*)24 b(an)f(alternative)g (solution)f(to)i(4.3)g(*/)75 162 y(MPI_Datatype)e(Onepair;)71 b(/*)23 b(datatype)g(for)h(one)f(pair)g(of)h(coordinates,)e(with)695 219 y(the)i(extent)f(of)h(one)f(particle)g(entry)g(*/)75 275 y(MPI_Aint)g(disp2[3];)75 332 y(MPI_Datatype)f(type2[3])h(=)h({MPI_LB,)e (MPI_DOUBLE,)h(MPI_UB};)75 388 y(int)g(blocklen2[3])g(=)g({1,)h(2,)f(1};)75 501 y(MPI_Address\()f(particle,)h(disp2\);)75 557 y(MPI_Address\()f (particle[0].d,)g(disp2+1\);)75 614 y(MPI_Address\()g(particle+1,)h (disp2+2\);)75 670 y(for)g(\(i=0;)h(i<2;)f(i++\))g(disp2[i])g(-=)h(disp2[0];) 75 783 y(MPI_Type_struct\()e(3,)h(blocklen2,)g(disp2,)g(type2,)g(&Onepair\);) 75 840 y(MPI_Type_commit\()f(&Onepair\);)75 896 y(MPI_Send\()h (particle[0].d,)f(1000,)h(Onepair,)g(dest,)g(tag,)g(comm\);)75 1073 y Fv(Fifth)18 b(example:)43 b FA(same)17 b(arra)o(y)g(manipulations)i (as)f(in)g(the)g(previous)g(example,)h(but)f(using)h(absolute)75 1129 y(addresses)c(in)h(datat)o(yp)q(es.)75 1236 y Ft(struct)23 b(Partstruct)147 1292 y({)147 1349 y(int)g(class;)147 1405 y(double)g(d[6];)147 1462 y(char)g(b[7];)147 1518 y(};)75 1631 y(struct)g(Partstruct)g(particle[1000];)337 1744 y(/*)h(build)f(datatype)g (describing)g(first)g(array)g(entry)g(*/)75 1857 y(MPI_Datatype)f (Particletype;)75 1913 y(MPI_Datatype)g(type[3])h(=)h({MPI_INT,)e (MPI_DOUBLE,)h(MPI_CHAR};)75 1970 y(int)238 b(block[3])23 b(=)h({1,)f(6,)h (7};)75 2026 y(MPI_Aint)118 b(disp[3];)75 2139 y(MPI_Address\()22 b(particle,)h(disp\);)75 2195 y(MPI_Address\()f(particle[0].d,)g(disp+1\);)75 2252 y(MPI_Address\()g(particle[0].b,)g(disp+2\);)75 2308 y (MPI_Type_struct\()g(3,)h(block,)g(disp,)g(type,)h(&Particletype\);)75 2421 y(/*)g(Particletype)e(describes)g(first)i(array)f(entry)g(--)h(using)f (absolute)147 2478 y(addresses)f(*/)505 2591 y(/*)h(5.1:)361 2647 y(send)h(the)f(entire)g(array)g(*/)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 65 65 bop 75 -100 a Fw(1.13.)29 b(DERIVED)15 b(D)o(A)l(T)l(A)l(TYPES)1124 b FA(65)75 49 y Ft(MPI_Type_commit\()22 b(&Particletype\);)75 106 y(MPI_Send\()h(MPI_BOTTOM,)f(1000,)h(Particletype,)f(dest,)h(tag,)h (comm\);)481 275 y(/*)f(5.2:)290 332 y(send)g(the)h(entries)e(of)i(class)f (zero,)290 388 y(preceded)g(by)g(the)h(number)f(of)g(such)h(entries)e(*/)75 501 y(MPI_Datatype)g(Zparticles,)h(Ztype;)75 614 y(MPI_Aint)g(zdisp[1000])75 670 y(int)g(zblock[1000],)g(i,)g(j,)h(k;)75 727 y(int)f(zzblock[2])g(=)h ({1,1};)75 783 y(MPI_Datatype)e(zztype[2];)75 840 y(MPI_Aint)118 b(zzdisp[2];)75 953 y(j=0;)75 1009 y(for)23 b(\(i=0;)h(i)f(<)h(1000;)f(i++\)) 123 1065 y(if)g(\(particle[i].index==0\))170 1122 y({)170 1178 y(for)h(\(k=i+1;)f(\(k)g(<)h(1000\)&&\(particle[k].index)d(=)i(0\))h(;)g (k++\);)170 1235 y(zdisp[j])f(=)h(i;)170 1291 y(zblock[j])f(=)h(k-i;)170 1348 y(j++;)170 1404 y(i)g(=)g(k;)170 1461 y(})75 1517 y(MPI_Type_indexed\()e (j,)h(zblock,)g(zdisp,)g(Particletype,)f(&Zparticles\);)75 1574 y(/*)i(Zparticles)e(describe)h(particles)f(with)i(class)f(zero,)g(using) 147 1630 y(their)g(absolute)g(addresses*/)75 1743 y(/*)h(prepend)e(particle)h (count)g(*/)75 1799 y(MPI_Address\(&j,)f(zzdisp\);)75 1856 y(zzdisp[1])h(=)g(MPI_BOTTOM;)75 1912 y(zztype[0])g(=)g(MPI_INT;)75 1969 y(zztype[1])g(=)g(Zparticles;)75 2025 y(MPI_Type_struct\(2,)e(zzblock,)i (zzdisp,)g(zztype,)g(&Ztype\);)75 2138 y(MPI_Type_commit\()f(&Ztype\);)75 2195 y(MPI_Send\()h(MPI_BOTTOM,)f(1,)i(Ztype,)f(dest,)g(tag,)g(comm\);)75 2371 y Fv(Sixth)16 b(example:)43 b FA(handling)17 b(of)e(unions.)75 2478 y Ft(union)23 b({)147 2534 y(int)118 b(ival;)147 2591 y(float)70 b(fval;)218 2647 y(})24 b(u[1000])-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 66 66 bop 75 -100 a FA(66)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(int)119 b(utype;)75 162 y(/*)24 b(All)f(entries)g(of)h(u)f(have)h(identical)e(type;)h(variable) 147 219 y(utype)g(keeps)g(track)g(of)h(their)f(current)g(type)g(*/)75 332 y(MPI_Datatype)70 b(type[2];)75 388 y(int)286 b(blocklen[2])22 b(=)i({1,1};)75 444 y(MPI_Aint)166 b(disp[2];)75 501 y(MPI_Datatype)70 b(mpi_utype[2];)75 557 y(MPI_Aint)166 b(i,j;)75 670 y(/*)24 b(compute)e(an)i(MPI)g(datatype)e(for)i(each)f(possible)g(union)g(type;)147 727 y(assume)g(values)g(are)g(left-aligned)f(in)i(union)f(storage.)g(*/)75 840 y(MPI_Address\()f(u,)i(&i\);)75 896 y(MPI_Address\()e(u+1,)i(&j\);)75 953 y(disp[0])f(=)h(0;)f(disp[1])g(=)h(j-i)75 1009 y(type[1])f(=)h(MPI_UB;)75 1122 y(type[0])f(=)h(MPI_INT;)75 1178 y(MPI_Type_struct\(2,)d(blocklen,)i (disp,)g(type,)g(&mpi_utype[0]\);)75 1291 y(type[0])g(=)h(MPI_FLOAT;)75 1348 y(MPI_Type_struct\(2,)d(blocklen,)i(disp,)g(type,)g(&mpi_utype[1]\);)75 1461 y(for\(i=0;)g(i<2;)g(i++\))g(MPI_Type_commit\(&mpi_utype[i])o(\);)75 1574 y(/*)h(actual)f(communication)f(*/)75 1686 y(MPI_Send\(u,)g(1000,)i (mpi_utype[utype],)d(dest,)i(tag,)h(comm\);)75 1901 y FB(1.14)60 b(New)19 b(p)n(rop)r(osal)h(fo)n(r)g(pack)g(and)g(unpack)-28 1969 y Fk(\))189 2026 y Fr(R)n(ationale.)37 b FA(Man)o(y)12 b(existing)h(comm)o(unication)f(libraries)i(pro)o(vide)e(pac)o(k/unpac)o(k)g (functions)h(for)189 2083 y(sending)19 b(noncon)o(tiguous)f(data:)24 b(the)18 b(user)g(explicitly)i(pac)o(ks)e(data)f(in)o(to)h(a)f(con)o(tiguous) h(bu\013er)189 2139 y(b)q(efore)c(sending)h(it,)f(and)g(unpac)o(ks)g(it)g (from)f(a)h(con)o(tiguous)g(bu\013er)f(after)g(receiv)o(eing)j(it.)j(Deriv)o (ed)189 2195 y(datat)o(yp)q(es,)g(whic)o(h)h(are)e(describ)q(ed)j(in)f (Section)g(1.13)e(b)q(elo)o(w,)i(allo)o(w)g(in)g(most)e(cases)h(to)f(a)o(v)o (oid)189 2252 y(explicit)d(pac)o(king)f(and)f(unpac)o(king)h(in)g(most)e (cases:)19 b(the)13 b(user)h(sp)q(eci\014es)h(the)e(la)o(y)o(out)g(of)f(the)i (data)189 2308 y(to)f(b)q(e)h(send)g(or)f(receiv)o(ed,)i(and)f(the)f(comm)o (unication)i(library)f(directly)h(accesses)f(a)f(noncon)o(tigu-)189 2365 y(ous)18 b(bu\013er.)30 b(The)19 b(pac)o(k/unpac)o(k)f(routines)h(are)f (pro)o(vided)h(for)f(compatibilit)o(y)i(with)f(previous)189 2421 y(libraries.)h(Also,)13 b(they)f(pro)o(vide)h(some)e(functionalit)o(y)j (that)d(is)i(not)e(otherwise)i(a)o(v)m(ailable)g(in)g(MPI:)189 2478 y(a)f(message)g(can)g(b)q(e)i(receiv)o(ed)f(in)h(sev)o(eral)e(parts,)g (where)h(the)f(receiv)o(e)i(op)q(eration)f(done)f(on)h(a)f(later)189 2534 y(part)i(ma)o(y)g(dep)q(end)i(on)f(the)g(con)o(ten)o(t)f(of)h(a)f (former)g(part;)g(and)h(outgoing)g(messages)f(ma)o(y)g(b)q(e)i(ex-)189 2591 y(plicitly)22 b(bu\013ered)e(in)h(user)f(supplied)i(space,)f(th)o(us)e (o)o(v)o(eriding)i(the)f(system)f(bu\013ering)h(p)q(olicy)l(.)189 2647 y(Finally)l(,)f(the)e(a)o(v)m(ailabilit)o(y)i(of)e(pac)o(k)f(and)i (unpac)o(k)f(op)q(erations)g(facilitate)h(the)g(dev)o(elopmen)o(t)f(of)189 2704 y(additional)f(comm)o(unication)g(libraries)h(la)o(y)o(ered)e(on)h(top)e (of)h(MPI.)g(\()p Fr(End)g(of)i(r)n(ationale.)p FA(\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 67 67 bop 75 -100 a Fw(1.14.)34 b(NEW)15 b(PR)o(OPOSAL)h(F)o(OR)g(P)l(A)o(CK)f(AND) g(UNP)l(A)o(CK)662 b FA(67)75 49 y Fv(MPI)p 160 49 14 2 v 16 w(P)l(A)o(CK\(inbuf,)15 b(incount,)i(int)o(yp)q(e,)f(outbuf,)g(outsize,)g(p)q (osition,)g(dest,)g(comm)m(\))117 126 y Fu(IN)155 b Fv(inbuf)493 b Fu(input)14 b(bu\013er)h(start)f(\(c)o(hoice\))117 201 y(IN)155 b Fv(incount)448 b Fu(n)o(um)o(b)q(er)13 b(of)h(input)f(data)h(items)f(\(in)o (teger\))117 275 y(IN)155 b Fv(int)o(yp)q(e)473 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(input)g(data)f(item)g(\(handle\))117 350 y(OUT)108 b Fv(outbuf)465 b Fu(output)14 b(bu\013er)h(start)g(\(arra)o(y)e(of)h(c)o (haracters\))117 424 y(IN)155 b Fv(outsize)457 b Fu(output)14 b(bu\013er)h(size,)f(in)g(c)o(haracter)h(units)f(\(in)o(teger\))117 498 y(INOUT)62 b Fv(outcount)420 b Fu(curren)o(t)16 b(p)q(osition)d(in)g (bu\013er)i(\(in)o(teger\))117 573 y(IN)155 b Fv(dest)511 b Fu(destination)14 b(of)f(pac)o(k)o(ed)h(message)g(\(in)o(teger\))117 647 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(for)j(pac)o(k)o(ed)g (message)g(\(handle\))75 771 y Ft(int)23 b(MPI)p 245 771 15 2 v 17 w(Pack\(void*)g(inbuf,)g(int)g(incount,)g(MPI)p 1074 771 V 17 w(Datatype)g(intype,)f(char)i(*outbuf,)393 828 y(int)g(outsize,)e (int)i(*position,)e(int)i(dest,)f(MPI)p 1375 828 V 17 w(Comm)g(comm\))75 914 y(MPI)p 150 914 V 17 w(PACK\(INBUF,)f(INCOUNT,)h(INTYPE,)g(OUTBUF,)g (OUTSIZE,)f(POSITION,)h(DEST,)g(COMM,)393 971 y(IERROR\))170 1027 y()g(INBUF\(*\))170 1084 y(CHARACTER*1)g(OUTBUF\(*\))170 1140 y(INTEGER)g(INCOUNT,)g(INTYPE,)g(OUTSIZE,)g(POSITION,)f(DEST,)i(COMM,)f (IERROR)166 1227 y FA(P)o(ac)o(ks)15 b(the)h(message)f(in)i(the)f(send)g (bu\013er)g(sp)q(eci\014ed)i(b)o(y)d Fv(inbuf,)i(incount,)g(int)o(yp)q(e)g FA(in)o(to)f(the)g(bu\013er)75 1283 y(space)i(sp)q(eci\014ed)h(b)o(y)f Fv(outbuf)h FA(and)e Fv(outsize)p FA(.)28 b(The)17 b(input)i(bu\013er)e(can)h (b)q(e)g(an)o(y)f(comm)o(unication)h(bu\013er)75 1340 y(allo)o(w)o(ed)f(in)g Fv(MPI)p 377 1340 14 2 v 15 w(SEND)p FA(.)g(The)f(output)g(bu\013er)g Fv(outbuf)i FA(is)e(an)g(arra)o(y)f(of)h(c)o(haracters)g(of)f(length)i Fv(outsize)75 1396 y FA(\(declared)j(as)e Fv(cha)o(r)g(buf[outsize])j FA(in)e(C)g(and)f Fv(CHARA)o(CTER*1)i(BUF\(OUTSIZE\))f FA(in)g(F)l(ortran\).) 29 b(The)75 1452 y(input)22 b(v)m(alue)g(of)e Fv(p)q(osition)i FA(is)f(the)g(\014rst)g(lo)q(cation)g(in)h(the)f(output)f(bu\013er)h(to)f(b)q (e)h(used)h(for)e(pac)o(king.)75 1509 y Fv(p)q(osition)d FA(is)f(incremen)o (ted)g(b)o(y)g(the)f(size)i(of)e(the)g(pac)o(k)o(ed)h(message,)f(and)g(the)h (output)f(v)m(alue)i(of)e Fv(p)q(osition)75 1565 y FA(is)21 b(the)f(\014rst)g(lo)q(cation)i(in)f(the)f(output)h(bu\013er)f(follo)o(wing)h (the)g(lo)q(cations)g(o)q(ccupied)h(b)o(y)e(the)h(pac)o(k)o(ed)75 1622 y(message.)k(The)17 b Fv(dest)i FA(and)e Fv(comm)11 b FA(argumen)o(ts)17 b(iden)o(ti\014es)h(the)g(in)o(tended)g(destination)g(of)f (the)g(pac)o(k)o(ed)75 1678 y(message)e(\(see)g(b)q(elo)o(w\).)75 1782 y Fv(MPI)p 160 1782 V 16 w(UNP)l(A)o(CK\(inbuf,)h(insize,)f(p)q (osition,)h(outbuf,)g(outcount,)h(outt)o(yp)q(e,)g(source,)e(comm)m(\))117 1859 y Fu(IN)155 b Fv(inbuf)493 b Fu(input)14 b(bu\013er)h(start)f(\(arra)o (y)g(of)f(c)o(haracters\))117 1934 y(IN)155 b Fv(insize)485 b Fu(size)15 b(of)e(input)h(bu\013er,)g(in)g(c)o(haracter)h(units)f(\(in)o (teger\))117 2008 y(INOUT)62 b Fv(p)q(osition)439 b Fu(curren)o(t)16 b(p)q(osition)d(in)g(bu\013er)i(\(in)o(teger\))117 2082 y(OUT)108 b Fv(outbuf)465 b Fu(output)14 b(bu\013er)h(start)g(\(c)o(hoice\))117 2157 y(IN)155 b Fv(outcount)420 b Fu(n)o(um)o(b)q(er)13 b(of)h(items)f(to)g (b)q(e)i(unpac)o(k)o(ed)f(\(in)o(teger\))117 2231 y(IN)155 b Fv(outt)o(yp)q(e)445 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(output)g(data) g(item)e(\(handle\))117 2305 y(IN)155 b Fv(source)468 b Fu(source)15 b(of)f(pac)o(k)o(ed)g(message)f(\(in)o(teger\))117 2380 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(for)j(pac)o(k)o(ed)g(message)g (\(handle\))75 2504 y Ft(int)23 b(MPI)p 245 2504 15 2 v 17 w(Unpack\(char*)f(inbuf,)h(int)h(insize,)f(int)g(*position,)g(void)g (*outbuf,)393 2561 y(int)h(outcount,)e(MPI)p 802 2561 V 17 w(Datatype)h(outtype,)g(int)g(source,)g(MPI)p 1607 2561 V 17 w(Comm)g(comm\))75 2647 y(MPI)p 150 2647 V 17 w(UNPACK\(INBUF,)f(INSIZE,)h (POSITION,)f(OUTBUF,)h(OUTCOUNT,)g(OUTTYPE,)g(SOURCE,)393 2704 y(COMM,)g(IERROR\))-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 68 68 bop 75 -100 a FA(68)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)170 49 y Ft(CHARACTER*1)23 b(INBUF\(*\))170 106 y()g(OUTBUF\(*\))170 162 y(INTEGER)g(INSIZE,)g (POSITION,)g(OUTCOUNT,)g(OUTTYPE,)f(SOURCE,)h(COMM,)g(IERROR)166 249 y FA(Unpac)o(ks)17 b(a)f(message)g(in)o(to)h(the)f(receiv)o(e)i(bu\013er) f(sp)q(eci\014ed)h(b)o(y)f Fv(outbuf,)h(outcount,)g(outt)o(yp)q(e)h FA(from)75 305 y(the)14 b(bu\013er)g(space)g(sp)q(eci\014ed)i(b)o(y)e Fv(inbuf)i FA(and)e Fv(insize)p FA(.)20 b(The)14 b(output)g(bu\013er)g(can)g (b)q(e)h(an)o(y)e(comm)o(unication)75 362 y(bu\013er)21 b(allo)o(w)o(ed)h(in) g Fv(MPI)p 523 362 14 2 v 16 w(RECV)p FA(.)g(The)g(input)g(bu\013er)g Fv(inbuf)g FA(is)g(an)f(arra)o(y)g(of)g(c)o(haracters)f(of)h(length)75 418 y Fv(outsize)p FA(.)f(\(declared)14 b(as)f Fv(cha)o(r)g(buf[outsize])i FA(in)f(C)f(and)g Fv(CHARA)o(CTER*1)h(BUF\(OUTSIZE\))g FA(in)g(F)l(ortran\).) 75 474 y(The)h(input)g(v)m(alue)g(of)f Fv(p)q(osition)i FA(is)f(the)f (\014rst)g(lo)q(cation)h(in)g(the)g(output)f(bu\013er)g(o)q(ccupied)i(b)o(y)e (the)h(pac)o(k)o(ed)75 531 y(message.)34 b Fv(p)q(osition)21 b FA(is)g(incremen)o(ted)g(b)o(y)f(the)g(size)h(of)f(the)g(pac)o(k)o(ed)g (message,)h(so)f(that)f(the)h(output)75 587 y(v)m(alue)g(of)e Fv(p)q(osition)i FA(is)f(the)g(\014rst)f(lo)q(cation)i(in)f(the)g(output)g (bu\013er)f(after)g(the)h(lo)q(cations)g(o)q(ccupied)i(b)o(y)75 644 y(the)14 b(message)f(that)f(w)o(as)h(unpac)o(k)o(ed.)20 b(The)13 b Fv(source)h FA(and)g Fv(comm)8 b FA(argumen)o(ts)k(iden)o(tify)j (the)e(source)h(of)f(the)75 700 y(pac)o(k)o(ed)i(message)g(\(see)g(b)q(elo)o (w\).)189 802 y Fr(A)n(dvic)n(e)21 b(to)i(users.)81 b FA(Note)21 b(the)h(di\013erence)i(b)q(et)o(w)o(een)e Fv(MPI)p 1277 802 V 16 w(RECV)g FA(and)h Fv(MPI)p 1608 802 V 15 w(UNP)l(A)o(CK)p FA(:)f(In)189 859 y Fv(MPI)p 274 859 V 15 w(RECV)p FA(,)17 b(the)f Fv(count)h FA(argumen)o(t)e(sp)q(eci\014es)j(the)e(maxim)o(um)f(n)o (um)o(b)q(er)i(of)e(items)h(that)f(can)h(b)q(e)189 915 y(receiv)o(ed.)35 b(The)21 b(actual)f(n)o(um)o(b)q(er)g(of)g(items)g(receiv)o(ed)h(is)g (determined)g(b)o(y)f(the)g(length)h(of)f(the)189 972 y(incoming)15 b(message.)20 b(In)15 b Fv(MPI)p 716 972 V 16 w(UNP)l(A)o(CK)p FA(,)f(the)h Fv(count)h FA(argumen)o(t)e(sp)q(eci\014es)i(the)f(actual)g(n)o (um)o(b)q(er)189 1028 y(of)f(items)i(that)e(are)h(unpac)o(k)o(ed;)g(the)g (\\size")h(of)e(the)i(corresp)q(onding)g(message)e(is)i(the)f(incremen)o(t) 189 1085 y(in)20 b Fv(p)q(osition)p FA(.)35 b(The)20 b(reason)f(for)g(this)i (c)o(hange)e(is)i(that)e(the)h(\\incoming)g(message)g(size")g(is)g(not)189 1141 y(predetermined)15 b(since)f(the)g(user)g(decides)h(ho)o(w)e(m)o(uc)o(h) h(to)e(unpac)o(k;)j(nor)e(is)h(it)g(easy)f(to)g(determine)189 1198 y(the)d(\\message)g(size")h(from)f(the)g(n)o(um)o(b)q(er)h(of)f(items)h (to)f(b)q(e)h(unpac)o(k)o(ed.)19 b(In)11 b(fact,)g(in)g(a)f(heterogenous)189 1254 y(system,)k(this)i(n)o(um)o(b)q(er)f(ma)o(y)g(not)f(b)q(e)i(determined)h (apriori.)j(\()p Fr(End)c(of)g(advic)n(e)g(to)h(users.)p FA(\))166 1356 y(T)l(o)f(understand)h(the)f(b)q(eha)o(vior)h(of)e(pac)o(k)h(and)h (unpac)o(k,)f(it)h(is)f(con)o(v)o(enien)o(t)h(to)f(think)h(of)e(the)i(data)75 1413 y(part)11 b(of)g(a)g(message)g(as)g(b)q(eing)i(the)e(sequence)i (obtained)f(b)o(y)g(concatenating)f(the)h(successiv)o(e)h(v)m(alues)f(sen)o (t)75 1469 y(in)k(that)f(message.)20 b(The)c(pac)o(k)f(op)q(eration)h(stores) f(this)g(sequence)i(in)f(the)g(bu\013er)f(space,)h(as)f(if)h(sending)75 1526 y(the)d(message)g(to)g(that)f(bu\013er.)19 b(The)14 b(unpac)o(k)f(op)q (eration)h(retriev)o(es)f(this)h(sequence)g(from)f(bu\013er)g(space,)75 1582 y(as)j(if)h(receiv)o(eing)i(a)d(message)g(from)g(that)g(bu\013er.)25 b(\(It)16 b(is)h(helpful)i(to)d(think)i(of)e(in)o(ternal)h(F)l(ortran)f (\014les)75 1639 y(or)f Fv(sscanf)h FA(in)g(C,)f(for)f(a)h(similar)i (function.\))166 1695 y(A)k(message)f(can)g(b)q(e)i(pac)o(k)o(ed,)f(next)g (sen)o(t)f(with)h(t)o(yp)q(e)g Fo(MPI)p 1228 1695 13 2 v 14 w(BYTE)p FA(;)f(the)g(outcome)h(is)g(as)f(if)h(the)75 1752 y(message)15 b(w)o(as)g(directly)i(sen)o(t)f(without)f(pac)o(king)h(\014rst.) 21 b(Similarly)l(,)d(a)d(message)h(can)f(b)q(e)i(receiv)o(ed)g(with)75 1808 y(t)o(yp)q(e)g Fo(MPI)p 256 1808 V 14 w(BYTE)p FA(,)f(next)h(unpac)o(k)o (ed;)h(the)f(outcome)f(is)h(as)g(if)g(it)g(w)o(as)f(receiv)o(ed)i(directly)l (,)h(without)d(\014rst)75 1864 y(unpac)o(king.)21 b(Th)o(us)75 1967 y Ft(position)i(=)g(0;)75 2023 y(MPI_PACK\(inbuf,)f(incount,)h(intype,)f (outbuf,)h(outsize,)g(position,)g(dest,)g(tag\);)75 2080 y(MPI_SEND\(outbuf,) f(position,)g(MPI_BYTE,)h(dest,)g(tag,)g(comm\))166 2182 y FA(is)16 b(equiv)m(alen)o(t)h(to)75 2284 y Ft(MPI_SEND\(inbuf,)22 b(incount,)h(intype,)f(dest,)i(tag,)f(comm\))75 2386 y(MPI_RECV\(inbuf,)f (insize,)h(MPI_BYTE,)f(source,)h(tag,)h(comm,)f(status\);)75 2443 y(position)g(=)g(0;)75 2499 y(MPI_UNPACK\(inbuf,)f(insize,)g(position,)h (outbuf,)g(outcount,)g(outtype\))166 2601 y FA(is)16 b(equiv)m(alen)o(t)h(to) 75 2704 y Ft(MPI_RECV\(outbuf,)22 b(outcount,)g(outtype,)h(source,)g(tag,)g (comm,)g(status\))1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 69 69 bop 75 -100 a Fw(1.14.)29 b(NEW)15 b(PR)o(OPOSAL)h(F)o(OR)g(P)l(A)o(CK)f(AND) g(UNP)l(A)o(CK)667 b FA(69)166 49 y(assuming)13 b(that)g(the)g(incoming)i (message)d(con)o(tains)i Fv(outcount)h FA(items)e(of)g(t)o(yp)q(e)g Fv(outt)o(yp)q(e)p FA(.)21 b(\(It)13 b(is)h(the)75 106 y(user)h(the)h (repsonsibilit)o(y)h(to)e(unpac)o(k)g(the)g(righ)o(t)g(n)o(um)o(b)q(er)h(of)f (data)f(items.\))166 162 y(The)20 b Fv(comm)14 b FA(and)20 b Fv(dest)h FA(argumen)o(ts)e(in)h(a)g(call)h(to)e Fv(MPI)p 1148 162 14 2 v 16 w(P)l(A)o(CK)g FA(should)i(matc)o(h)e(the)h(argumen)o(ts) 75 219 y(used)e(to)e(subsequen)o(tly)i(send)g(the)f(pac)o(k)o(ed)g(bu\013er;) h(the)f Fv(comm)11 b FA(and)17 b Fv(source)h FA(argumen)o(ts)e(in)i(a)e(call) j(to)75 275 y Fv(MPI)p 160 275 V 16 w(UNP)l(A)o(CK)c FA(should)h(matc)o(h)e (the)g(argumen)o(ts)g(previously)i(used)g(to)e(receiv)o(e)h(the)g(pac)o(k)o (ed)g(bu\013er.)20 b(If)75 332 y(the)14 b(bu\013er)g(is)g(pac)o(k)o(ed)g(and) g(unpac)o(k)o(ed)h(b)o(y)f(the)g(same)f(pro)q(cess,)h(then)g(the)g Fv(comm)8 b FA(and)14 b Fv(dest)i FA(argumen)o(ts)75 388 y(of)g(the)h Fv(MPI)p 293 388 V 16 w(P)l(A)o(CK)f FA(call)i(should)f(matc)o(h)f(the)h Fv(comm)11 b FA(and)16 b Fv(source)h FA(argumen)o(ts)f(of)g(the)h Fv(MPI)p 1684 388 V 16 w(UNP)l(A)o(CK)75 444 y FA(call.)189 546 y Fr(R)n(ationale.)38 b FA(The)13 b(need)h(for)e(the)h Fv(comm)7 b FA(and)13 b Fv(source/dest)i FA(argumen)o(ts)d(follo)o(ws)h(from) f(the)h(desire)189 602 y(to)g(\(i\))i(supp)q(ort)f(comm)o(unication)h(across) f(heterogenous)h(no)q(des;)f(\(ii\))h(not)f(require)i(messages)e(to)189 659 y(b)q(e)i(self-t)o(yp)q(ed;)g(and)g(\(iii\))h(allo)o(w)f(data)f(con)o(v)o (ersion)g(to)g(b)q(e)i(done)f(either)g(b)o(y)f(sender)i(or)e(receiv)o(er.)189 715 y(I.e.,)e(w)o(e)h(wish)h(to)e(con)o(tin)o(ue)i(the)f(general)h(MPI)f (design)h(where)f(b)q(oth)h(sender)f(and)h(receiv)o(er)f(ha)o(v)o(e)189 772 y(su\016cien)o(t)21 b(information)f(to)g(do)g(data)g(con)o(v)o(ersion,)h (and)g(b)q(oth)f(c)o(hoices)h(are)f(a)o(v)m(ailable)i(to)e(the)189 828 y(implemen)o(ter,)i(ev)o(en)f(if)g(messages)f(carry)g(no)g(information,)h (in)h(addition)f(to)f(data.)34 b(\()p Fr(End)21 b(of)189 885 y(r)n(ationale.)p FA(\))166 1062 y Fq(Discussion:)33 b Fu(Rather)13 b(than)g(requiring)f(that)h(the)h Fo(comm)7 b Fu(and)13 b Fo(dest)g Fu(argumen)o(ts)f(of)h Fo(MPI)p 1608 1062 13 2 v 14 w(P)m(A)o(CK)e Fu(iden)o(tify)75 1112 y(the)j(target)f(of)f(the)i(pac)o(k)o(ed)f(message,)f (one)i(could)e(merely)g(require)i(that)f(the)h(argumen)o(t)d(iden)o(ti\014es) j(a)e(target)i(\\of)75 1162 y(the)h(same)e(kind".)18 b(This)c(w)o(ould)f (allo)o(w)g(to)h(pac)o(k)g(a)g(message)f(and)h(send)h(it)f(to)g(sev)o(eral)h (destinations,)f(as)g(long)f(as)75 1212 y(all)g(these)i(destinations)f(are)g (\\of)f(the)i(same)e(kind".)k(T)m(o)c(supp)q(ort)i(this)f(extension,)g(it)g (w)o(ould)f(b)q(e)h(useful)g(to)g(ha)o(v)o(e)75 1261 y(an)e(inquiry)f (function)h(that)h(returns)g(the)g(\\kind")e(of)h(a)g(destination,)g(or)g(an) g(enquiry)g(function)g(that)g(c)o(hec)o(ks)i(that)75 1311 y(t)o(w)o(o)f (destinations)h(are)h(\\of)e(the)h(same)f(kind".)166 1368 y(A)e(further)h (extension)g(is)f(to)g(allo)o(w)e(a)i(\\generic")g(destination,)g(of)g(unkno) o(wn)f(\\kind".)17 b(This)11 b(could)f(sp)q(eci\014ed)75 1424 y(b)o(y)h(a)g(constan)o(t)h Fo(dest)g Fu(=)g Fo(MPI)p 530 1424 V 14 w(ANY)p 629 1424 V 15 w(DEST)p Fu(.)e(A)i(message)f(pac)o(k)o(ed)g(for)g (a)g(destination)h(of)e(unkno)o(wn)h(kind,)g(m)o(ust)f(b)q(e)75 1481 y(unpac)o(k)o(ed)i(with)f Fo(source)i Fu(=)f Fo(MPI)p 590 1481 V 14 w(ANY)p 689 1481 V 15 w(SOURCE)p Fu(,)d(a)i(generic)i(source)g (of)e(unkno)o(wn)g(kind.)17 b(The)11 b(implemen)o(tation)75 1537 y(w)o(ould)h(need)h(to)g(use)g(a)f(common)e(in)o(terc)o(hange)j(format)e (\(e.g.,)g(XDR\),)h(for)g(suc)o(h)h(messages.)18 b(The)13 b(last)f(extension) 75 1594 y(could)i(b)q(e)g(in)g(addition)e(to)i(or)g(in)f(place)h(of)g(the)g (previous)g(one.)166 1733 y FA(One)23 b(can)f(pac)o(k)g(sev)o(eral)h (messages,)g(in)g(succession,)i(in)e(a)f(bu\013er,)h(then)g(send)g(the)f (resulting)75 1789 y(concatenation.)d(The)12 b(outcome)f(is)h(as)g(if)g(the)g (concatenated)g(message)f(w)o(as)g(prepared)h(using)g(a)g(suitable)75 1846 y(deriv)o(ed)k(datat)o(yp)q(e.)k(Similarly)l(,)d(one)e(can)g(unpac)o(k)h (a)f(bu\013er)g(in)o(to)g(sev)o(eral)h(messages.)j(Th)o(us)75 1947 y Ft(...)75 2004 y(POSITION)k(=)g(0)75 2060 y(CALL)g(MPI_PACK\()g(LEN,)g (1,)h(MPI_INTEGER,)e(OUTBUF,)290 2116 y(1000,)h(POSITION,)g(DEST,)g(COMM,)g (IERR\))75 2173 y(CALL)g(MPI_PACK\()g(A,)h(LEN,)f(MPI_REAL,)f(OUTBUF,)314 2229 y(1000,)h(POSITION,)f(DEST,)i(COMM,)f(IERR\);)75 2286 y(CALL)g(MPI_SEND\()g(OUTBUF,)g(POSITION,)f(MPI_BYTE,)h(DEST,)314 2342 y(TAG,)g(COMM,)g(IERR\))166 2444 y FA(is)16 b(equiv)m(alen)o(t)h(to)75 2534 y Ft(...)75 2591 y(BLEN\(1\))23 b(=)h(1)75 2647 y(BLEN\(2\))f(=)h(LEN)75 2704 y(CALL)f(MPI_ADDRESS\()g(LEN,)g(DISP\(1\)\))-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 70 70 bop 75 -100 a FA(70)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)75 49 y Ft(CALL)23 b(MPI_ADDRESS\()g(A,)g (DISP\(2\)\))75 106 y(TYPE\(1\))g(=)h(MPI_INTEGER)75 162 y(TYPE\(2\))f(=)h (MPI_REAL)75 219 y(CALL)f(MPI_TYPE_STRUCT\(2,)f(BLEN,)h(DISP,)g(TYPE,)g (NEWTYPE,)g(IERR\))75 275 y(CALL)g(MPI_COMMIT\(NEWTYPE,)f(IERR\))75 332 y(CALL)h(MPI_SEND\(MPI_BOTTOM,)e(1,)j(NEWTYPE,)f(DEST,)g(TAG,)g(COMM\)) 166 425 y FA(Both)15 b(programs)f(send)i(a)f(message)f(that)h(consists)g(of)g (one)g(in)o(teger)h(\(with)f(v)m(alue)h Fv(LEN)p FA(\),)f(follo)o(w)o(ed)75 481 y(b)o(y)g Fv(LEN)h FA(reals)f(from)g(arra)o(y)f Fv(A)p FA(.)i(The)f(second)h(program)e(p)q(ossibly)j(a)o(v)o(oids)f(an)f(additional) i(memory-to-)75 537 y(memory)e(cop)o(y)g(op)q(eration,)g(at)f(the)i(exp)q (ense)g(of)f(additional)h(prepro)q(cessing.)166 594 y(If)h(the)f(v)m(alue)i (of)e Fv(LEN)g FA(is)h(not)f(kno)o(wn)g(at)g(the)g(receiving)i(side)g(ahead)e (of)g(the)h(receiv)o(e)g(op)q(eration,)75 650 y(then)f(the)f(message)g(can)g (b)q(e)h(receiv)o(ed)g(b)o(y)f(executing)i(the)e(co)q(de)h(b)q(elo)o(w)75 755 y Ft(...)75 812 y(CALL)23 b(MPI_RECV\(INBUF,)f(1000,)h(MPI_BYTE,)g (SOURCE,)g(TAG,)g(COMM,)385 868 y(STATUS,)g(IERR\))75 925 y(CALL)g (MPI_UNPACK\(INBUF,)f(1000,)h(POSITION,)g(LEN,)g(1,)385 981 y(MPI_INTEGER,)f(SOURCE,)h(COMM,)h(IERR\))75 1038 y(CALL)f (MPI_UNPACK\(INBUF,)f(1000,)h(POSITION,)g(A,)g(LEN,)385 1094 y(MPI_REAL,)g(SOURCE,)g(COMM,)g(IERR\))189 1199 y Fr(A)n(dvic)n(e)c(to)h (users.)66 b FA(The)20 b(simplest)h(w)o(a)o(y)d(of)h(receiving)j(an)d(in)o (teger)h Fv(LEN)p FA(,)f(follo)o(w)o(ed)h(b)o(y)f Fv(LEN)189 1256 y FA(reals,)14 b(is)h(to)f(send)h(t)o(w)o(o)e(messages,)g(the)i(\014rst) f(with)g(the)h(v)m(alue)g Fv(LEN)p FA(,)f(the)h(second)g(with)f Fv(LEN)h FA(real)189 1312 y(n)o(um)o(b)q(ers,)g(as)g(sho)o(wn)f(b)q(elo)o(w) 189 1436 y Ft(...)189 1492 y(!)23 b(send)h(side)189 1549 y(CALL)f (MPI_SEND\(LEN,)f(1,)i(MPI_INTEGER,)e(DEST,)h(TAG,)g(COMM,)h(IERR\))189 1605 y(CALL)f(MPI_SEND\(A,)f(LEN,)i(MPI_REAL,)e(DEST,)h(TAG,)h(COMM,)f (IERR\))189 1718 y(...)189 1775 y(!)g(receive)g(side)189 1831 y(CALL)g(MPI_RECV\(LEN,)f(1,)i(MPI_INTEGER,)e(SOURCE,)h(TAG,)g(COMM,)g (IERR\))189 1887 y(CALL)g(MPI_RECV\(A,)f(LEN,)i(MPI_REAL,)e(SOURCE,)h(TAG,)g (COMM,)h(IERR\))189 2011 y FA(The)c(\\pac)o(king")g(of)f(sev)o(eral)h (di\013eren)o(t)h(data)e(items)h(in)h(one)f(message,)g(using)h(either)g (explicit)189 2067 y(calls)c(to)e Fv(MPI)p 432 2067 14 2 v 16 w(P)l(A)o(CK)p FA(,)h(or)g(deriv)o(ed)h(datat)o(yp)q(es)e(is)i(only)g(to)e (impro)o(v)o(e)h(p)q(erformance,)g(in)h(systems)189 2124 y(where)e(comm)o (unication)h(latencies)h(are)d(signi\014can)o(t.)21 b(\()p Fr(End)16 b(of)g(advic)n(e)g(to)h(users.)p FA(\))166 2305 y Fq(Discussion:)51 b Fu(Sev)o(eral)18 b(restrictions)h(w)o(ere)g(suggested)h (ab)q(out)e(sending/receiving)g(of)g(messages)g(that)75 2355 y(where)d(pac)o(k)o(ed)f(together:)166 2405 y(\(1\))j(Messages)h(are)g(alw)o (a)o(ys)e(pac)o(k)o(ed/unpac)o(k)o(ed)h(in)g(sequence,)i(and)e(the)g (starting)g(p)q(oin)o(t)g(for)g(unpac)o(king)75 2455 y(is)e(the)g(starting)f (p)q(oin)o(t)h(for)f(pac)o(king.)20 b(E.g.,)13 b(one)i(cannot)g(pac)o(k)f (separately)i(t)o(w)o(o)e(messages,)g(then)i(concatenate)75 2504 y(them)c(and)g(send)h(the)g(result.)18 b(Or,)13 b(one)f(cannot)h(pac)o (k)f(\014v)o(e)h(messages,)f(and)g(then)h(send)g(a)g(message)f(that)g (consists)75 2554 y(of)h(the)i(last)f(three)h(ones)g(pac)o(k)o(ed.)k(With)13 b(this)h(rule,)g(the)h(only)e(constrain)o(t)i(on)f(message)f(represen)o (tation)j(is)e(that)75 2604 y(it)f(is)g(\\monotone":)j(the)e(represen)o (tation)g(for)f(a)g(pre\014x)h(of)f(the)h(full)e(message)h(is)g(a)g(pre\014x) h(of)f(the)g(represen)o(tation)75 2654 y(for)h(the)g(full)f(message.)19 b(T)m(o)13 b(pic)o(k)h(a)g(fancy)g(example:)j(supp)q(ose)e(that)f(one)h(w)o (an)o(ts)f(to)g(compress)g(messages)g(using)75 2704 y(a)h(Lemp)q(el-Ziv)f(co) q(de)j(b)q(efore)f(they)g(are)g(sen)o(t,)g(and)f(vice-v)o(ersa)i(on)e(the)h (receiv)o(e)h(side.)23 b(This)15 b(restriction)i(w)o(ould)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Page: 71 71 bop 75 -100 a Fw(1.14.)34 b(NEW)15 b(PR)o(OPOSAL)h(F)o(OR)g(P)l(A)o(CK)f(AND) g(UNP)l(A)o(CK)662 b FA(71)75 49 y Fu(allo)o(w)12 b(to)i(do)g(so;)g(the)h (pac)o(k/unpac)o(k)f(routines)g(w)o(ould)g(ha)o(v)o(e)g(to)g(store)h(in)o (ternally)e(information)e(on)j(the)g(curren)o(t)75 99 y(state)g(of)g(the)g (Ziv-Lemp)q(el)e(algorithm.)k(A)d(less)i(fancy)e(example:)k(one)d(migh)o(t)d (w)o(an)o(t)j(to)f(store)i(a)e(general)h(header)75 149 y(at)g(the)g(start)h (of)e(the)i(message\))166 199 y(The)g(formal)e(constrain)o(t)i(is)g(that,)g (on)g(the)h(send/pac)o(k)f(side,)h(the)f(initial)e(v)n(alue)i(of)f Fo(p)q(osition)i Fu(matc)o(hes)e(the)75 249 y(initial)c(bu\013er)k(address)f (used)h(in)d(the)i(send)h(op)q(eration,)e(and)g(that)g Fo(p)q(osition)h Fu(is)f(only)g(mo)q(di\014ed)f(b)o(y)h(the)h(successiv)o(e)75 298 y(calls)g(to)f(pac)o(k.)18 b(I)13 b(sligh)o(t)f(extension)i(to)f(this,)f (whic)o(h)h(is)g(harmless)f(and)h(simpli\014es)e(implemen)o(tation,)f(is)i (that)h(the)75 348 y(initial)f(v)n(alue)h(of)g(p)q(osition)h(is)f(zero)i (\(pac)o(king)e(starts)i(at)f(bu\013er)h(start\).)166 405 y(\(2\))e(Messages) h(are)f(unpac)o(k)o(ed)h(in)e(the)i(same)e(\\c)o(h)o(unks")h(that)g(they)g (where)h(pac)o(k)o(ed.)k(This)13 b(w)o(ould)f(preserv)o(e)75 461 y(ev)o(en)i(more)e(freedom)h(in)g(the)h(in)o(ternal)f(represen)o(tation)i (of)e(messages,)g(as)g(co)q(de)h(do)q(es)h(not)e(ha)o(v)o(e)g(an)o(ymore)f (to)h(b)q(e)75 517 y(self-delimiting.)i(But)f(this)g(restriction)h(is)f(more) f(signi\014can)o(t)g(from)f(the)j(user)g(viewp)q(oin)o(t.)166 657 y FA(The)e(follo)o(wing)g(calls)g(allo)o(w)g(the)f(user)h(to)f(\014nd)h (out)f(ho)o(w)g(m)o(uc)o(h)g(space)h(is)f(needed)i(to)e(pac)o(k/unpac)o(k)75 713 y(messages)j(and,)g(th)o(us,)f(manage)h(space)g(allo)q(cation)i(for)d (bu\013ers.)75 817 y Fv(MPI)p 160 817 14 2 v 16 w(P)l(A)o(CK)p 291 817 V 16 w(SIZE\(incount,)i(int)o(yp)q(e,)g(dest,)g(comm)n(,)11 b(size\))117 894 y Fu(IN)155 b Fv(incount)448 b Fu(n)o(um)o(b)q(er)13 b(of)h(input)f(data)h(items)f(\(in)o(teger\))117 967 y(IN)155 b Fv(int)o(yp)q(e)473 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(input)g(data)f (item)g(\(handle\))117 1040 y(IN)155 b Fv(dest)511 b Fu(destination)14 b(of)f(pac)o(k)o(ed)h(message)g(\(in)o(teger\))117 1112 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(for)j(pac)o(k)o(ed)g(message)g (\(handle\))117 1185 y(OUT)108 b Fv(size)519 b Fu(size)15 b(of)e(pac)o(k)o (ed)h(message)g(\(in)o(teger\))75 1310 y Ft(int)23 b(MPI)p 245 1310 15 2 v 17 w(Pack)p 358 1310 V 17 w(size\(int)g(incount,)f(MPI)p 876 1310 V 17 w(Datatype)h(intype,)g(int)g(dest,)g(MPI)p 1609 1310 V 17 w(Comm)h(comm,)393 1366 y(int*)f(size\))75 1453 y(MPI)p 150 1453 V 17 w(PACK)p 263 1453 V 16 w(SIZE\(INCOUNT,)f(INTYPE,)h(DEST,)h (COMM,)f(SIZE,)g(IERROR\))170 1509 y(INTEGER)g(INCOUNT,)g(INTYPE,)g(DEST,)g (COMM,)g(SIZE,)h(IERROR)166 1595 y FA(A)11 b(call)h(to)e Fv(MPI)p 426 1595 14 2 v 16 w(P)l(A)o(CK)p 557 1595 V 16 w(SIZE\(incount,)j(int)o(yp)q (e,)g(dest,)g(comm)m(,)8 b(size\))j FA(returns)g(in)h Fv(size)f FA(the)g(incremen)o(t)75 1652 y(in)21 b Fv(p)q(osition)h FA(that)e(is)h (e\013ected)f(b)o(y)h(a)f(call)h(to)f Fv(MPI)p 974 1652 V 16 w(P)l(A)o(CK\(inbuf,)i(incount,)h(int)o(yp)q(e,)g(outbuf,)g(outsize,)75 1708 y(p)q(osition,)16 b(dest,)g(tag\))p FA(;)75 1812 y Fv(MPI)p 160 1812 V 16 w(UNP)l(A)o(CK)p 354 1812 V 17 w(SIZE\(outcount,)g(outt)o(yp)q (e,)g(source,)g(comm)m(,)11 b(size\))117 1889 y Fu(IN)155 b Fv(outcount)420 b Fu(n)o(um)o(b)q(er)13 b(of)h(output)g(data)f(items)g(\(in)o (teger\))117 1962 y(IN)155 b Fv(outt)o(yp)q(e)445 b Fu(datat)o(yp)q(e)14 b(of)g(eac)o(h)g(output)g(data)g(item)e(\(handle\))117 2035 y(IN)155 b Fv(source)468 b Fu(source)15 b(of)f(pac)o(k)o(ed)g(message)f(\(in) o(teger\))117 2108 y(IN)155 b Fv(comm)466 b Fu(comm)o(unicator)11 b(for)j(pac)o(k)o(ed)g(message)g(\(handle\))117 2180 y(OUT)108 b Fv(size)519 b Fu(size)15 b(of)e(pac)o(k)o(ed)h(message)g(\(in)o(teger\))75 2305 y Ft(int)23 b(MPI)p 245 2305 15 2 v 17 w(Unpack)p 406 2305 V 17 w(size\(int)f(outount,)h(MPI)p 924 2305 V 17 w(Datatype)g(outtype,) f(int)i(source,)393 2361 y(MPI)p 468 2361 V 17 w(Comm)f(comm,)g(int*)h (size\))75 2448 y(MPI)p 150 2448 V 17 w(PACK)p 263 2448 V 16 w(SIZE\(OUTCOUNT,)e(OUTTYPE,)h(SOURCE,)g(COMM,)g(SIZE,)g(IERROR\))170 2504 y(INTEGER)g(OUTCOUNT,)g(OUTTYPE,)g(SOURCE,)g(COMM,)g(SIZE,)g(IERROR)166 2591 y FA(A)15 b(call)i(to)d Fv(MPI)p 439 2591 14 2 v 16 w(UNP)l(A)o(CK)p 633 2591 V 17 w(SIZE\(outcount,)i(outt)o(yp)q(e,)h(source,)f(comm)m(,)c (size\))j FA(returns)g(in)h Fv(size)g FA(the)75 2647 y(incremen)o(t)i(in)g Fv(p)q(osition)g FA(that)f(is)g(e\013ected)h(b)o(y)f(a)f(call)j(to)d Fv(MPI)p 1157 2647 V 16 w(UNP)l(A)o(CK\(outbuf,)j(outcount,)g(outt)o(yp)q(e,) 75 2704 y(inbuf,)d(insize,)f(p)q(osition,)h(dest,)h(tag\))p FA(;)-32 46 y Fy(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop %%Page: 72 72 bop 75 -100 a FA(72)608 b Fw(CHAPTER)15 b(1.)35 b(POINT)16 b(TO)f(POINT)h(COMMUNICA)l(TION)166 49 y Fq(Discussion:)37 b Fu(This)15 b(de\014nition)f(implies)f(that)h(a)h(pac)o(k)o(ed)g(message)f (tak)o(es)h(the)g(same)f(size,)h(whether)h(it's)75 99 y(pac)o(k)o(ed)g (\014rst,)f(or)h(second,)g(or)f(third.)22 b(This)15 b(is)g(not)h(true)g(in)f (our)g(h)o(yp)q(othetical)g(example)f(where)i(messages)g(are)75 149 y(compressed,)g(or)f(in)f(the)i(case)g(where)g(a)f(header)h(is)e(stored)i (when)g(the)g(\014rst)f(message)g(is)g(pac)o(k)o(ed.)22 b(Rather,)15 b(one)75 199 y(migh)o(t)d(w)o(an)o(t)h(these)j(t)o(w)o(o)d(functions)h(to)g (return)h(an)e(upp)q(er)i(b)q(ound)f(on)g(the)g(amoun)o(t)e(of)i(storage)g (required.)166 249 y(If)f(w)o(e)i(put)f(in)f(these)i(functions,)f(w)o(e)g(ma) o(y)e(w)o(an)o(t)h(to)h(delete)h(the)g(function)e Fo(MPI)p 1424 249 13 2 v 14 w(TYPE)p 1546 249 V 15 w(SIZE)p Fu(.)166 298 y(If)g(w)o(e)i(supp)q(ort)f(pac)o(king)f(and)h(unpac)o(king,)f(w)o(e)h (ma)o(y)e(w)o(an)o(t)i(to)f(delete)i(the)g(function)75 348 y Fo(MPI)p 152 348 V 14 w(USER)p 270 348 V 14 w(SPECIFIES)p 482 348 V 13 w(BUFFER)p Fu(.)10 b(If)i(the)h(user)h(w)o(an)o(ts)e(to)g (bu\013er)i(on)e(its)h(o)o(wn,)f(then)h(it)f(will)f(use)i(pac)o(k/unpac)o(k) 75 398 y(calls.)166 454 y(I)i(cannot)g(resist)h(p)q(oin)o(ting)d(out)i(that)g (at)g(least)g(three)h(di\013eren)o(t)g(groups)f(came)f(out)g(with)h (suggestions)g(for)75 511 y(pac)o(k/unpac)o(k)i(functions)g(that)g(w)o(ere)h (essen)o(tially)f(iden)o(tical.)27 b(This)17 b(suggests)h(to)f(me)f(that)h (MPI)g(has)h(a)e(go)q(o)q(d)75 567 y(in)o(ternal)d(coherence,)j(so)e(that)g (there)h(often)f(is)g(one)g(\\righ)o(t")f(w)o(a)o(y)g(of)h(pro)o(viding)e(a)i (new)g(function.)1967 46 y Fy(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Sun Feb 20 15:21:37 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id PAA04046; Sun, 20 Feb 1994 15:21:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id PAA27592; Sun, 20 Feb 1994 15:20:05 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 20 Feb 1994 15:20:04 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id PAA27564; Sun, 20 Feb 1994 15:20:01 -0500 Message-Id: <199402202020.PAA27564@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 4661; Sun, 20 Feb 94 15:20:02 EST Date: Sun, 20 Feb 94 14:54:53 EST From: "Marc Snir ((914) 945-3204 (862)" To: mpi-pt2pt@CS.UTK.EDU cc: frankeh@watson.ibm.com Subject: buffering in point to point Reply-To: SNIR@watson.ibm.com Among the comments we got there where many that insisted that buffering is very important, and some that wanted CMMD style communication. In the time honored MPI tradition, we should consider appeasing both by providing additional functions. A possible proposal is outlined below: Add a fourth mode in pt2pt communication. Thus, we shall have, for blocking sends 1. Synchronous: returns when a matching receive has started. The intended implementation is that no buffering is done. Completion of such send is nonlocal. 2. Asynchronous: returns irrespective of the state of the receiver. If no receive is posted, then the message has to be buffered. Completion of such send is local. If the operation cannot complete (no buffer, no receiver) then an error occurs. 3. Standard: I.e., implementation-dependent. The message may be buffered, thus allowing the operation to complete, or the send may block until a matching receive occurs. Completion is nonlocal. The operation never fails, but lack of buffer space may cause dealocks. 4. Ready: a variant of (1) and/or (2), where the user promises that a receive is already posted -- thus, no buffering is needed. This extends, as usual to nonblocking sends: (1) a WAIT for a synchronous immediate send blocks until a receive has celared the send buffer. (2) a WAIT for an asynchronous immediate send returns after the message is received or buffered, and causes an error if neither can be done. (3) a WAIT for a standard send can block if there is no posted receive or available buffer space, waiting for either to become available, due to other processes' acitivity. (4) a WAIT for a ready send returns after the receive has cleared the send buffer. This means 3 more functions (blocking, immediate, persistent send). Comments? From owner-mpi-pt2pt@CS.UTK.EDU Sun Feb 20 19:40:47 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id TAA05796; Sun, 20 Feb 1994 19:40:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA11915; Sun, 20 Feb 1994 19:39:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 20 Feb 1994 19:39:55 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id TAA11908; Sun, 20 Feb 1994 19:39:52 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA15251; Sun, 20 Feb 94 18:37:15 CST Date: Sun, 20 Feb 94 18:37:15 CST From: Tony Skjellum Message-Id: <9402210037.AA15251@Aurora.CS.MsState.Edu> To: mpi-pt2pt@CS.UTK.EDU, SNIR@watson.ibm.com, mpi-context@CS.UTK.EDU Subject: "Major changes" at this time Cc: frankeh@watson.ibm.com Marc, I think it is too late to make such big changes, there is not even a detailed proposal (though that you could surely change by Wednesday!). As you have taught me over the past year, by now we should only consider air-tight recommendations with air-tight proposals, whereas during the readings we started to require carefully thought out and worked out proposals. I am concerned that we will overlook something major. In analogy, within the context chapter, we will be restricting changes to fixing the intercommunication section (except for minor details elsewhere), and looking hard at some of the proposals (some of the ideas worked out in France [eg, ideas discussed between you and Lyndon on Intercommunication] have serious problems). We are sticking with well-formed proposals, and trying to augment incomplete proposals (or defeat them in advance by finding serious problems). I also think that we should not try to bring back ideas that were defeated (perhaps multiple times, or in the subcommittee process) at this time. Otherwise, we stand a good chance of not finishing. In short, I think we are just doing incremental fixes at this meeting. Can't we leave other things for MPI follow-on? -Tony PS Your idea does sound reasonable, but I think that the four modes need to be clarified better, so that the design appears more "orthogonal" to users. Can you clarify what the total send/receive functionality would look like, etc. ----- Begin Included Message ----- From owner-mpi-pt2pt@CS.UTK.EDU Sun Feb 20 14:21:03 1994 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 20 Feb 1994 15:20:04 EST Date: Sun, 20 Feb 94 14:54:53 EST From: "Marc Snir ((914) 945-3204 (862)" To: mpi-pt2pt@CS.UTK.EDU Cc: frankeh@watson.ibm.com Subject: buffering in point to point Reply-To: SNIR@watson.ibm.com Content-Length: 1802 Among the comments we got there where many that insisted that buffering is very important, and some that wanted CMMD style communication. In the time honored MPI tradition, we should consider appeasing both by providing additional functions. A possible proposal is outlined below: Add a fourth mode in pt2pt communication. Thus, we shall have, for blocking sends 1. Synchronous: returns when a matching receive has started. The intended implementation is that no buffering is done. Completion of such send is nonlocal. > [No buffering, non-local, always succeeds, can deadlock] 2. Asynchronous: returns irrespective of the state of the receiver. If no receive is posted, then the message has to be buffered. Completion of such send is local. If the operation cannot complete (no buffer, no receiver) then an error occurs. > [Buffering (if necessary), local, fails sometimes, cannot deadlock] 3. Standard: I.e., implementation-dependent. The message may be buffered, thus allowing the operation to complete, or the send may block until a matching receive occurs. Completion is nonlocal. The operation never fails, but lack of buffer space may cause dealocks. > [Buffering (if necessary), non-local, always succeeds, can deadlock] > > "3" is either... > "1" > or "2" with retry (should we discuss semantics for retrying/timeouts?) > 4. Ready: a variant of (1) and/or (2), where the user promises that a receive is already posted -- thus, no buffering is needed. > [No buffering, non-local, fails if receive not posted, cannot deadlock] > > "4" is "3" + the promise of the user to have a receive posted. > This extends, as usual to nonblocking sends: (1) a WAIT for a synchronous immediate send blocks until a receive has cleared the send buffer. (2) a WAIT for an asynchronous immediate send returns after the message is received or buffered, and causes an error if neither can be done. (3) a WAIT for a standard send can block if there is no posted receive or available buffer space, waiting for either to become available, due to other processes' acitivity. (4) a WAIT for a ready send returns after the receive has cleared the send buffer. This means 3 more functions (blocking, immediate, persistent send). > > Please explain further. > Comments? ----- End Included Message ----- From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 21 02:06:13 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id CAA06872; Mon, 21 Feb 1994 02:06:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA03678; Mon, 21 Feb 1994 02:05:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 21 Feb 1994 02:05:21 EST Errors-to: owner-mpi-pt2pt@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id CAA03663; Mon, 21 Feb 1994 02:05:17 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA04769 (5.65c8/IDA-1.4.4 for ); Mon, 21 Feb 1994 08:05:18 +0100 Received: by f1neuman.gmd.de (AIX 3.2/UCB 5.64/4.03) id AA16052; Mon, 21 Feb 1994 08:05:26 GMT Date: Mon, 21 Feb 1994 08:05:26 GMT From: Rolf.Hempel@gmd.de (Rolf Hempel) Message-Id: <9402210805.AA16052@f1neuman.gmd.de> To: mpi-pt2pt@CS.UTK.EDU Subject: asynchronous send Cc: SNIR@watson.ibm.com, gmap10@f1neuman.gmd.de I strongly support Marc's proposal for a fourth mode of send, even if it increases the number of functions in MPI. We had enough discussions on the buffering issue that I don't want to repeat the reasons for requesting it. If we are concerned about the growing number of MPI functions, I would rather vote for abandoning the so-called "Standard" send, which Marc correctly describes as "implementation-dependent". - Rolf