zyda@trouble.cs.nps.navy.mil (michael zyda) (07/06/90)
Help Needed with Using Arenas for Sharing Data Between Processes
Arenas can be used to share data between processes under IRIX.
The arena is a pool of shared memory (really a memory mapped file).
When one process allocates (through usmalloc) memory from the arena,
that piece of memory is unavailable to other processes for allocation
(through usmalloc) though that piece of memory is "shared" (meaning
accessible) by the other processes.
In order for the other processes to have access to that allocated
shared memory, the other processes must somehow obtain a pointer
to that allocated memory. How is this done safely, i.e. what
system routine do I call in process 1 to find a pointer to
the first byte allocated by usmalloc in process 2?
I have thought of a few ways of doing this, most unsatisfactory.
I am working from the SGI Manual entitled "Parallel Programming
on the IRIS-4D" (Version 1.0). This manual has a sample program for every
chapter BUT Chapter 7, "Sharing Data through Arenas".
(Way 1) Use System V shared memory routines (shmget, shmat)
to pass the pointers. If I have to use shmget/shmat, why bother
with arenas? shmat gets me the pointer I want.
(Way 2) Have process 1 usmalloc all of the bytes in the arena.
When it does this, process 1 has a ptr to these bytes. Then have
process 1 return the bytes with usfree, keeping the ptr (very
unsafe). Then process 2 does a usmalloc for all of the bytes
in the arena, gets the pointer and then returns the bytes
with usfree. Now both processes have pointers to the bytes
in the arena and can read/write data there. As long as I
never again usmalloc this arena AND no other system service
does, I am ok (maybe) but feel pretty terrible about
this solution. Perhaps I can get this pointer from the
arena structure? Please show me or tell me which routine
to call to get this.
Notes: I am concerned with independent programs that need
to share data. I am NOT talking about threads that have been
"sproc"ed or "m_fork"ed, as the sharing of the virtual address
space in such cases is the default.
Michael Zyda
zyda@trouble.cs.nps.navy.mil
The following two programs illustrate Way 2.
/*
this is file share1.c
This program is an example of shared memory for two unrelated
processes. This program sets up an arena and allocates some
bytes from that arena.
Once the arena is successfully set up and the shared memory
has been obtained, this program then looks for a message
in the shared memory. If this program sees a message, it
prints out the message and zeroes the message. The zeroing
of the message indicates to the other process that it can then
generate another message.
Either program can be spawned off first.
The programs establish communication with each other
by trying to allocate SIZEIWANT bytes from the arena.
If this is the first program to try and allocate that
size buffer, then this program will get a ptr to that
many bytes from the arena. The program will then free
the bytes BUT keep the ptr (very dangerous).
SIZEIWANT is carefully crafted to be greater than
half the size of the defined arena.
The second program to attempt to get SIZEIWANT bytes
from the arena will fail (and loop) until the first
program returns the bytes. When the second program
get the bytes, it also then returns the bytes BUT
AGAIN keeps a ptr to those bytes.
The two programs can then use the ptrs to reference
memory shared between them.
This is unsafe in that if either program does another
usmalloc() to that region, then that program will
possibly stomp all over the data being written by
the other processor.
One cure to this is to then create another arena and
properly allocate out of it, passing ptrs through maybe
the first arena.
Yes, the arena mechanism does define shared data but
no mechanism provides for passing the ptrs to the
dynamically allocated shared memory...
*/
#include <ulocks.h> /* Includes required by the arena services */
#include <malloc.h>
#include <stdio.h>
#define MAXARENASIZE 4096 /* Max arena size for this program */
#define SIZEIWANT 3000 /* Number of bytes I want for shared memory. */
main()
{
usptr_t *arenaptr; /* ptr to the arena */
volatile char *buf; /* ptr to the start of the shared bytes */
long i; /* loop temp */
/* Define the size of each arena subsequently allocated. */
if(usconfig(CONF_INITSIZE, MAXARENASIZE) == -1)
{
printf("usconfig: cannot define the size to allocate the arena!\n");
exit(1);
}
/* Create an arena */
arenaptr = usinit("myarena.file");
if(arenaptr == (usptr_t *)NULL)
{
printf("usinit: cannot create the arena!\n");
exit(1);
}
/* Allocate a piece of shared memory for communication.
We wait here until we can allocate the memory. We are
going to either be first, and get the memory right
away or we are going to wait until the other process
frees the memory back to the arena.
*/
while((buf = usmalloc(SIZEIWANT, arenaptr)) == (char *)NULL)
{
/* We do nothing in this loop but wait... */
}
/* Now free the buffer for reallocation */
/* We are going to keep using this ptr though!!! */
usfree(buf, arenaptr);
/* zero the first byte of buf as a signal to the other process */
*buf = '\0';
/* loop forever...*/
for(i=0; i < 100000000; i=i+1)
{
/* whenever the first byte is non-zero, print out a message */
while(*buf != '\0')
{
printf("Other process says:%s\n",buf);
*buf = '\0';
}
} /* end for i */
/* Leave the arena file, but here is where we could delete it */
}
/*
this is file share2.c
This program is an example of shared memory for two unrelated
processes. This program sets up an arena and allocates some
bytes from that arena.
Once the arena is successfully set up and the shared memory
has been obtained, this program then waits for the buffer
in shared memory to be clear of the message. Once the
buffer is clear, this program writes a message there
and then again waits for the buffer to be clear.
Either program can be spawned off first.
The programs establish communication with each other
by trying to allocate SIZEIWANT bytes from the arena.
If this is the first program to try and allocate that
size buffer, then this program will get a ptr to that
many bytes from the arena. The program will then free
the bytes BUT keep the ptr (very dangerous).
SIZEIWANT is carefully crafted to be greater than
half the size of the defined arena.
The second program to attempt to get SIZEIWANT bytes
from the arena will fail (and loop) until the first
program returns the bytes. When the second program
get the bytes, it also then returns the bytes BUT
AGAIN keeps a ptr to those bytes.
The two programs can then use the ptrs to reference
memory shared between them.
This is unsafe in that if either program does another
usmalloc() to that region, then that program will
possibly stomp all over the data being written by
the other processor.
One cure to this is to then create another arena and
properly allocate out of it, passing ptrs through maybe
the first arena.
Yes, the arena mechanism does define shared data but
no mechanism provides for passing the ptrs to the
dynamically allocated shared memory...
*/
#include <ulocks.h> /* Includes required by the arena services */
#include <malloc.h>
#include <stdio.h>
#define MAXARENASIZE 4096 /* Max arena size for this program */
#define SIZEIWANT 3000 /* Number of bytes I want for shared memory. */
main()
{
usptr_t *arenaptr; /* ptr to the arena */
volatile char *buf; /* ptr to the start of the shared bytes */
long i; /* loop temp */
/* Define the size of each arena subsequently allocated. */
if(usconfig(CONF_INITSIZE, MAXARENASIZE) == -1)
{
printf("usconfig: cannot define the size to allocate the arena!\n");
exit(1);
}
/* Create an arena */
arenaptr = usinit("myarena.file");
if(arenaptr == (usptr_t *)NULL)
{
printf("usinit: cannot create the arena!\n");
exit(1);
}
/* Allocate a piece of shared memory for communication.
We wait here until we can allocate the memory. We are
going to either be first, and get the memory right
away or we are going to wait until the other process
frees the memory back to the arena.
*/
while((buf = usmalloc(SIZEIWANT, arenaptr)) == (char *)NULL)
{
/* We do nothing in this loop but wait... */
}
/* Now free the buffer for reallocation */
/* We are going to keep using this ptr though!!! */
usfree(buf, arenaptr);
/* loop forever...*/
for(i=0; i < 100000000; i=i+1)
{
/* whenever the first byte is zero, put the message into the buffer */
while(*buf == '\0')
{
strcpy(buf, "Message from share2: Hello share1\n");
}
} /* end for i */
/* Leave the arena file, but here is where we could delete it */
}tristram@conan.esd.sgi.com (07/09/90)
In article <9007051819.AA09079@trouble.cs.nps.navy.mil> zyda@trouble.cs.nps.navy.mil (michael zyda) writes:
Help Needed with Using Arenas for Sharing Data Between Processes
In order for the other processes to have access to that allocated
shared memory, the other processes must somehow obtain a pointer
to that allocated memory. How is this done safely, i.e. what
system routine do I call in process 1 to find a pointer to
the first byte allocated by usmalloc in process 2?
I'm pretty sure that if you specify all the parameters for the arena
identically in the two programs, and you call usinit() in the same
order (if you have more than one arena) both arenas will be mapped at
the same address. The same is true of mapped files (since arenas are
built on mapped files, see mmap(2)). So, one solution is to:
1) map a small (1 or 2 K) file in both processes to serve as a
rendezvous area. You may specify the address at which the file is
mapped or let the system do it for you. It will be mapped at the
same address in both processes.
2) create the arena you need with usinit (in both processes)
3) allocate from the arena in one process, and store the address
somewhere in the rendezvous area. You can create a struct that has
some pointers in it to manage the rendezvous area.
4) load the address from the rendezvous area into the other
process, and access the data it points to in the arena.
You are still left with synchronization to do on your data accesses in
the arena. You can allocate semaphores from the arena in the first
process, and store pointers to them in the rendezvous area.
This still leaves you with a race on startup (if two programs both
think they are initializing the rendezvous area). I suppose the
safest way is to have a single memory manager program which must be
running before any of your clients can run. Or you can simulate
safety by hacking together something based on file locks.
--
--
David Tristram
tristram@sgi.comcdshaw@cs.UAlberta.CA (Chris Shaw) (07/10/90)
In article X zyda@trouble.cs.nps.navy.mil (michael zyda) writes: > In order for the other processes to have access to that allocated >shared memory, the other processes must somehow obtain a pointer >to that allocated memory. How is this done safely, i.e. what >system routine do I call in process 1 to find a pointer to >the first byte allocated by usmalloc in process 2? Perhaps you could try opening a socket connection such that process 1 sets up a service and the other process(es) open a simple client connection to process 1's service. Do some communication, figure who usmalloc'd what, then close the connection and await a new one. This will serialize multiple client starts into the common area. The rest of the communication could be done via the shared memory. Should work. > Michael Zyda > zyda@trouble.cs.nps.navy.mil -- Chris Shaw University of Alberta cdshaw@cs.UAlberta.ca Now with new, minty Internet flavour! CatchPhrase: Bogus as HELL !
jwag@moose.asd.sgi.com (Chris Wagner) (07/10/90)
Yes, in fact in 3.2 one has to come up with ones own solution - opening a pipe or file seems the easiest. In 3.3 we have added the usputinfo and usgetinfo functions that give you a single pointer, thus the originating process can use usputinfo to store a pointer to a data structure that further defines what addresses in the arena are being used for what Chris Wagner