On credibility of simulation studies of telecommunication networks
In telecommunicationnetworks,asin many otherareasof scienceandengineering,proliferation of computersasresearchtools hasresultedin the adoptionof computersimulationasthe most commonlyusedparadigmof scientific investigations.This, togetherwith a plethoraof existing simulationlanguagesandpackages,hascreateda popularopinion that simulationis mainly an exercisein computerprogramming.In new computingenvironments,programmingcanbeminimised,or evenfully replaced,by themanipulationof icons(representingpre-built programming objectscontainingbasicfunctionalblocksof simulatedsystems)on a computermonitor. Onecan saythatwe have witnessedanothersuccessof modernscienceandtechnology:theemergenceof wonderfulandpowerful toolsfor exploringandpredictingthebehaviour of suchcomplex, stochastic dynamicsystemsastelecommunicationetworks. But this enthusiasmis not sharedby all researchersin this area. An opinion is spreading thatonecannotrely on themajority of thepublishedresultson performanceevaluationstudiesof telecommunicationnetworks basedon stochasticsimulation,sincethey lack credibility. Indeed, thespreadof this phenomenonis sowide thatonecanspeakabouta deepcrisisof credibility. In this paper , this claim is supportedby theresultsof asurvey of over2200publicationson telecommunicationnetworksin recentproceedingsof theIEEEINFOCOMandsuchjournalsastheIEEE Transactionson Communications,the IEEE/ACM Transactionson Networking, andthe PerformanceEvaluationJournal. Thiswork waspartiallysupportedby theUniversityof Canterbury’sGrantno. U6301 Status:a revisionof thepapersubmittedto theIEEECommunicationsMagazinein April 2000 Credibility of simulationstudies 2 Our discussionfocuseson two importantnecessaryconditionsof a crediblesimulationstudy: useof appropriatepseudo-randomgeneratorsof independent uniformly distributednumbers,and appropriateanalysisof simulationoutput data. Having consideredtheir perils and pitfalls, we formulateguidelinesthat,if observed,couldhelpto assureabasiclevel of credibility of simulation studiesof telecommunicationetworks.