a system 10 constructed in accordance with an example of the invention is illustrated generally in fig1 . a three - dimensional ( 3d ) probe 12 accumulates image data from a treatment region or organ of a patient , image data is processed using a 3d imaging card 14 . the probe 12 preferably is an ultrasound device but can be any other rapid imaging technology , such as rapid ct or mr . a conventional personal computer 16 having a monitor can be used to operate on the image data from the imaging card 14 using conventional software and hardware tools to be described in more detail hereinafter . radioactive seeds 18 are provided for insertion using any one of a variety of conventional means for inserting devices or articles into the human body , such as insertion devices 19 , which may be either needles or stiff catheters . the 3d ultrasound probe 12 , therefore , provides an image signal to the computer 16 and a virtual realty interface card 13 coupled to the imaging card 14 which enables a user to visualize a translucent image of the patient organ and real time interaction of any one of a variety of treatment devices , such as the implant needles 19 or a foley catheter 20 , and one of the seeds 18 within the organ . computer software can be utilized in a conventional manner to visualize the 3d imaging data in various formats . the formats include orthogonal two dimensional ( 2d ) images , oblique 2d images , and translucent 3d rendering . all of these reconstructions can be directly displayed on the computer monitor ; and 3d translucent , stereoscopic , rendering is also available in the vr ( virtual realty ) mode . the preferred ultrasound probe 12 is a conventional kretz ultrasound imaging system manufactured by kretz corporation , now available as medison combison 530 through medison america corporation , pleasantown , calif . this system and other such conventional systems are readily available and can provide real time ultrasound image data . the medison combison ultrasound system incorporates an endorectal probe which acquires multiple image planes in real time and the software of the present invention reconstructs the translucent 3d volume . alternate systems include bi - plane 2d imaging systems with the probe mounted in a stepper motor driven holder for rapid automatic acauisition of multiple image planes . there is nothing that is application specific about the imaging system , thus any commercially available system will suffice . for collecting ultrasound image data , the diagnostic transrectal ultrasound probe 12 ( see fig2 ) is inserted into the patient &# 39 ; s rectum to obtain real time volumetric images of the prostate for use during the implant procedure . the diagnostic probe 12 is preferably a phased array probe designed so that the array of transducers can rotate about the axis of the array sweeping out a 3d imaging volume . as the probe 12 rotates , images are captured and digitized by use of the imaging card 14 ( see fig1 ), so as to create a fixed number of images slices per rotation . an alternative method utilizes a transverse oriented phased array form of the endorectal probe 12 which is moved longitudinally in an automated rapid sequence so as to create a series of transverse image slices automatically . another embodiment of the probe 12 can incorporate multiple transverse phased arrays ( shown in phantom in fig1 b ) arranged parallel to each other orthogonal to the axis of an endorectal probe to produce multiple simultaneous image slices ( see , for example , fig5 a and 5b ). the 3d image data will be represented as a three dimensional image raster . the ultrasound probe 12 can be mounted into a probe holder 30 ( see fig3 a and 3c ) with fig3 b illustrating one example of an ultrasound image from an anatomical prostate phantom employed to carry out testing and planning . the probe holder 30 includes a digital encoder 42 for providing information regarding the position of all of the desired ultrasound image planes in the prostate relative to each other . the image plane location will be automatically sent to the system computer and “ tagged ” to the acquired ultrasound image for that position . thus , it will be possible to reproduce the longitudinal and lateral positions of the implant catheters for the ultrasound therapy applicators and for the temperature probes . there are several possible image processing cards which could be utilized ; however , using current modalities each of the processing cards is configured specifically for 3d . the 3d image raster is buffered ; and thus , for example , if the 2d images are 512 × 512 and there are sixteen image planes in the probe 12 , and each pixel is a byte ( 256 gray scales ), at least a 512 × 512 × 16 byte = 4 . 2 mbyte image buffer in the card 14 is needed . several commercial cards ( for example , made by coreco , matrox and integral technologies ) can be equipped with this amount of video ram ( vram ), but the way the card &# 39 ; s hardware interacts with the computer &# 39 ; s video and software drivers does not utilize this data in 3d . current available methodologies enable augmenting the software and some hardware of these cards so that they can act as a 3d card . the processing and memory architecture preferably is designed to allow for simultaneous image acquisition and processing . the digitizing card should also preferably have standard imaging tools , such as real time window and leveling , zoom and pan of the ultrasound images . some existing cards ( e . g ., matrox ; coreco ) do provide standard imaging tools . the 3d image data arising from the ultrasound probe 12 is preferably buffered on the imaging card 14 . the 3d image is preferably represented as a series of 2d images . this is referred to as the image stack or 3d image raster . the 3d image raster is represented in memory as a linear array of bytes of length n × m × p where n is the width of the 2d image in pixels , m is the height a 2d image in pixels , and p is the number of 2d images in the image stack . in a preferred embodiment the user can include defined formats . entire 3d image stacks at specific times during the intraoperative session can be stored in the dicom standard . the user will have the ability to select a 3d image volume for archiving as part of the system software . these image stacks can then be reviewed in any of the various visualization modes ( standard orthogonal 2d views , oblique 2d views , or 3d translucent views ) as described above . in addition , the user will have the ability to store any of the 2d views available at any time during the intraoperative session . the computational platform can , for example , be any form of computing means , such as the personal computer 16 , which incorporates a pci bus architecture . currently , pci bus is preferable over the isa or eisa bus because the pci bus is much faster . however , a generic system which will be suitable for this applicable will be described . a 200 mhz ( or greater speed ) pentium / pentium - pro computer supplied with 128 mbytes of ram and a 6 . 0 gbyte hard disk should be sufficient ram and disk memory to run the software in a real - time fashion and to archive all patient data . there should be sufficient ram to facilitate host image processing in parallel with onboard image processing for quality assurance checks . a high resolution monitor capable of displaying at least 1280 × 1024 × 64 bit resolutions is preferably used . based on currently available technology , the ultrasound images obtained from the ultrasound imaging system of the ultrasound probe 12 can be of good diagnostic quality . when transforming this input image data into a 3d representation , whether in the 3d perspective mode or the real time vr mode , the resultant volumes can , however , be noisy and hinder diagnostic and spatial accuracy . in order to improve the image quality , a number of conventional hardware and software filters can be used which will filter the incoming image data stored on the imaging card 14 . routines such as image pixel averaging , smoothing , and interpolation can improve the 3d rendering of the imaging volume . these sets of filters or routines are to be distinguished from the set of standard imaging tools running on the host cpu which are available within a conventional imaging software package . in the preferred embodiment , three of the perspective views are the standard transverse , coronal and sagittal 2d views . these three orthogonal views are taken from a user specified location within the imaging space . for example , the user can request that the three orthogonal views have their common centers at a spatial position of ( 5 . 0 cm , 15 . 0 , 25 . 0 cm ) relative to the origin of the template system . one also can select the reference point of either of the three orthogonal views independently , that is the three views do not have to have common center points . as mentioned hereinbefore , fig5 a and 5b show examples of several example 2d views from a 3d ultrasound image volume . fig6 shows a number of possible viewing directions , and fig7 gives further examples of translucent 3d viewing from different angles . the 3d ultrasound image volume was obtained from actual ultrasound images of a human prostate and of a prostate implant phantom . on each of the views , one can define , draw and edit contours using conventional computer software , such as microsoft foundation class ( mfc ) view files . each contour can be given a unique name by the user , and then drawn by the user using the mouse of the computer 16 . all attributes of the contours such as name and color can , based on conventional imaging software , be user selectable . the user can also edit the contours by selecting functions , such as adding a point to a contour , deleting a point from a contour or deleting the entire contour . once the contours are defined , the user has the option to render them in 3d or view in conventional 2d mode on the 3d perspective mode or viewed in the vr mode . again , all contour 3d attributes such as color , lighting , and shading are user controlled . the contours by default appear on the 2d images , however , the user can control the individual contour &# 39 ; s 2d and 3d visibility . in order to improve the ability to visualize the real time , 3d information , the 3d image raster can be rendered as a real time , transparent , 3d volume . this transparent volume can be viewed and displayed on the monitor of the computer 16 at any arbitrary viewing angle and is calculated using conventional 3d object reconstruction algorithms . such standard algorithms can render a large imaging volume in fractions of a second , even on present day computing platforms . the transparent nature of the reconstruction thus allows the user to “ see ” inside any objects which appear in the imaging volume . for example , if the prostate is imaged in the imaging volume , then it will be reconstructed as a transparent volume , in which other anatomical landmarks such as the urethra , tissue abnormalities or calcifications can be seen . in addition , if any other objects such as needles or catheters are inserted into the prostate , and if they are visible in the ultrasound images , they will be seen as they enter the prostate ( see fig8 showing introduction of the seed 18 with the catheter / needle 19 ). since the volumes are rendered as transparent solids , the needles 19 ( and other articles ) can thus easily be seen as they move inside the prostate volume as well . since the ultrasound images are obtained in real time , the 3d perspective reconstruction is also rendered in real time . the preferred algorithm for the perspective 3d reconstruction is the known bresenham raytrace algorithm . as described above , in the routine process of brachytherapy planning , the patient undergoes an initial volumetric ultrasound scan using the probe 12 . this scan is done before the radiation therapy planning or the actual implant . during the radiation therapy planning , the ideal positions of the radioactive seeds 18 ( see fig1 ) within the prostate are determined . this ideal seed distribution is optimized to deliver a dose distribution within the prostate that will deliver all the radiation dose to the target volume only , while sparing the surrounding healthy tissues such as the rectum and bladder . the optimal positions of the seeds 18 and the optimal position of the needles 19 are recorded for later use in the operating room when the needles 19 are loaded into the patient . the seeds 18 are then loaded into the needles 19 , and the physician then attempts to place the needles 19 inside the prostate according to the treatment dose plan positions ( again , see example in fig8 ). the dose as a function of position for a cylindrical 125 i seed of a given activity can be determined from a lookup table or calculated from a conventional analytic formula . the dose field can be visualized as a set of isodose lines in 2d or isodose surface in 3d . the dose computation routine is based upon the tg43 standard adopted by the aapm ( american association of physicists in medicine ) entitled “ dosimetry of interstitial brachytherapy sources ”: recommendations of the aapm radiation therapy committee task group no . 43 which specifies the dose model and the data used in the dose calculation . this particular implementation runs extremely fast on a conventional 233 mhz pc , computing the dose for a single seed in less than 0 . 5 seconds . the total 3d dose distribution within the prostate for a 100 seed implant requires only 50 seconds , or less than one minute total computation time . thus , this can be done “ on line ” in the operating room . in the 2d , 3d perspective , or the real time vr modes , the user has the ability to view the optimized seeds 18 and the needles 19 in the same volume as the real time ultrasound data . this allows the physician to see exactly where the needles 19 should go and hence make adjustments to position the needles 19 optimally . the pre - planned , optimal positioned needles 19 and the seeds 18 can be rendered again as a transparent solid , the color of which is user selectable . as the real needles 19 are inserted into the prostate , their positions relative to the ideal needle placements based on the dose plan can be monitored in real time . any deviation of the position of a given needles 19 can be quickly and accurately readjusted so as to follow the path of the ideal needles 19 . as the different needles 19 are placed at different positions inside the prostate , the viewing angle can be adjusted to facilitate viewing of the needle or catheter placement . fig5 a and 5b displays perspective 3d views and the three orthogonal reconstructions of the image data along with the pre - planned catheter positions . the pre - planned needles 19 can also be viewed in the vr mode as virtual objects overlaid onto the imaging volume . a flowchart description of the translucent volume visualization methodology is shown in fig1 . the input image volume is described by the vectors i , j , k of appropriate magnitude for the volume . the viewing angle parameters are the angles θ , ø described on fig6 and fig1 . the rotation matrix , r , is calculated using the formulae given in the flowchart of fig1 . the entire imaging volume is calculated by multiplying the rotation matrices in the x , y , z directions by the respective vectors i , j and k describing the incremental portions along the x , y , z directions . thus , the multiplying vector is ( i - i o , j - j o , k - k o ) where i o , j o , k o are the starting points along x , y and z axes and the volume is determined by summing the component contributions shown in fig1 . the 3d translucent image is then created by computing the translucent 2d image over the entire image volume and summing the z - pixels . a virtual reality interface system can be composed of a conventional head mounted display ( hmd ) 50 shown in fig1 and a 6d ( x , y , z , roll , pitch , yaw ) tracking system . the hmd 50 consists of two color monitors which mount to a head set in the position directly in front of the eyes . the hmd 50 is based on the principal that whatever is displayed on each monitor is directly incident on the retina for each eye , and hence true 3d images can be created by rendering objects as 3d perspective images for each eye . given the distance between the eyes ( the interoccular distance which is approximately 80 mm ) and the distance and spherical angles of the distance of the center line between the eyes from the coordinate origin , the 2d images which appear in each of the two monitors can be determined exactly as described above . this results in a true 3d image as perceived by the user . therefore , as the user moves his or her head or moves around the room , the distance from the origin and the spherical angles also change . this motion of the user or user &# 39 ; s head can be obtained from the vr tracking system . given these spatial parameters , the images which are reconstructed in the two eye monitors can be updated in real time , giving the user the illusion of the object really existing in 3d space . the user literally has the ability to walk around the object , viewing it in 3d space . instead of reconstructing computer generated geometric objects as is usually the case in vr , the transparent , 3d reconstruction of the real time imaging data will preferably be reconstructed . hence as the physician walks around the patient undergoing the implant , the physician will see the 3d ultrasound volume mapped inside the patient &# 39 ; s pelvis , spatially correlated to the position of the patient &# 39 ; s real prostate ( or other organ ) and anatomy . the physician can “ see ” inside the patient to the extent of what is visible in the ultrasound imaging volume . since the ultrasound probe 12 is locked down to the template , which is then secured to the floor , the exact positions of all voxels in the ultrasound imaging volume are known exactly relative to the template , and hence relative to the room . as the needles 19 are inserted into the patient , they will appear in the image volume and hence are reconstructed in the vr reconstruction . all of this occurs in real time so that the physician also can see the needles 19 enter the prostate in real time . as mentioned above , if the pre - planned , optimized needles 19 are displayed , the physician can then see the position of the actual needles 19 as they are being inserted relative to the optimal placement . hence , the physician has the ability to adjust the needles 19 to correspond to their optimal positions . in addition , since the needles 19 are automatically extracted , the computer software has the ability to calculate and render the 3d dose distribution in real time as the needles 19 are being inserted . as an example , a currently available , a fast and inexpensive hmd is made by virtual - io corporation ( mountain view , calif .). the hmd is full color with two 0 . 70 lcd displays with a resolution of 180 , 000 pixels per lcd panel . the video input is ntsc with field sequential format . the lcd panels are semitransparent , allowing the real outside world to be included in the virtual reconstruction . the field of view is 30 ° for each eye . a six degree of freedom ( 6 dof ) tracking system can also be attached to the hmd . the 6 dof tracking system allows for the determination of the spatial position of the user &# 39 ; s head and the yaw , pitch , and roll of the head . the conventional head set weighs only 8 ounces and comes with stereo sound . stereo sound is an extremely valuable technology in the operating room . with this capability , the physician has the ability to monitor the patient &# 39 ; s heart rate and respiration rate while performing the implant . hence any fluctuation in the patient &# 39 ; s vital signs can be instantly accessed and acted thereon if necessary . the radioactive seeds 18 are made of high density material such as stainless steel , and hence have a very bright response in the ultrasound images . therefore , automatic seed detection in the ultrasound images can readily be accomplished , for example , by a simple thresholding algorithm along with the requirement that the resultant objects which are removed by threshold have a certain maximum size determined by the actual size of the seeds . near - real - time visualization will provide immediate feedback to the physician during the implant process itself . there is a clear need for the visualization being available during the implant process . the nearly real time visualization is of great importance to the effective use of a translucent overlay of the ideal seed pre - plan ( from the therapy planning process ) in the three - dimensional volume . the physician can “ see ” in nearly real time the relationship of the needles and seeds being implanted to the ideal pre - plan locations and quickly accommodate redirection required prior to leaving the radiation seeds . further , the need for this in three - dimensional representation is very important to overcome the greatest fundamental limitation in brachytherapy , which is knowing at the same time both the lateral placement and longitudinal placement of needles and seeds relative to the target volume and pre - plan . this is a 3d problem which has up until now been addressed in 2d in a stepwise fashion without the ability to “ see ” the exact location of where you are in the target . this real time 3d visualization also would speed the implant process in the case of brachytherapy as well as make it more accurate . it would also speed other minimally invasive surgical procedures and localized tissue ablation procedures ( for example , cryosurgery or localized selected ablation of diseased liver tissue or local removal of breast tissue ). these procedures could be accomplished with real time visualization inside the tissue being treated with greater accuracy in shorter time . this aspect would reduce operating room time and costs to the patient and health care system . while preferred embodiments of the inventions have been shown and described , it will be clear to those skilled in the art that various changes and modifications can be made without departing from the invention in its broader aspects as set forth in the claims provided hereinafter .
a method and apparatus for three - dimensional imaging and treatment of a patient &# 39 ; s body . the method and apparatus utilize a system for developing a therapy plan for treatment of an organ of the patient , a device for generating ultrasound image data from a treatment region and a device for providing a translucent volume image of a portion of a patient &# 39 ; s body and a separate translucent image of the patient organ and a three dimensional viewing device to superimpose a translucent article image to enable viewing of the article image simultaneously with the patient organ and a portion of the patient &# 39 ; s body .