import * as $ from "jquery" ; import * as vis from "vis" ; import { int , double } from "../main" ; import { Visualization } from "./Visualization" ; import Net from "../Net" ; import Simulation from "../Simulation" ; import NetworkVisualization from "./NetworkVisualization" ; import { TrainingData , Configuration , TrainingDataEx } from "../Configuration" ; interface Point3d { x : double ; y : double ; z : double ; style ? : double ; } /** step for [[Simulation#forwardPass]] */ interface TDNNUpdateLayer { boxSize : number ; maxX : number ; numberofNeuron : number ; timeDelayed : number ; } export interface TDNNGraphUpdate { nodes : any [ ] ; layerNumber ? : number ; currentTime ? : number ; timeDelayed ? : number ; numberofneuron ? : number ; previousIndex ? : number ; layersUpdate : { [ layer_id ] : TDNNUpdateLayer ; } ; } /** show the network as a ordered left-to-right graph using arrow color, width and label to show weights */ export default class TDNNGraph implements Visualization { actions = [ "TDNN Graph" ] ; graph : any ; nodes : any ; edges : any ; net ! : Net . NeuralNet ; container = document . createElement ( "div" ) ; showbias ! : boolean ; currentlyDisplayingForwardPass = false ; biasBeforeForwardPass = false ; alreadySetNet = false ; offsetBetweenLayers = 2 ; xyToConnectionTDNN : { [ xcommay ] : [ int , int , double ] ; } = { } ; calculatedNetwork = false ; currentLayer = 1 ; currentNeuron = 0 ; updates : TDNNGraphUpdate [ ] = [ { nodes : [ ] , layersUpdate : { } } ] ; minYofLayer : { [ layer_id ] : number ; } = { } ; constructor ( public sim ) { this . instantiateGraph ( ) ; } instantiateGraph ( ) { this . nodes = new vis . DataSet ( [ ] , { queue : true } ) ; this . edges = new vis . DataSet ( [ ] , { queue : true } ) ; const graphData = { nodes : this . nodes , edges : this . edges } ; const options = { nodes : { shape : "box" , physics : false } , physics : { stabilization : true } , layout : { improvedLayout : false } , interaction : { dragNodes : false , hover : true } } ; if ( this . graph ) this . graph . destroy ( ) ; this . graph = new vis . Network ( this . container , graphData , options ) ; let thisVar = this ; } private edgeId ( conn ) { return conn . inp . id * this . net . connections . length + conn . out . id ; } onNetworkLoaded1 ( net , forceRedraw = false ) { if ( ! forceRedraw && this . net && this . net . layers . length == net . layers . length && this . net . layers . every ( ( layer , index ) => layer . length == net . layers [ index ] . length ) && this . showbias === this . sim . state . bias ) { this . net = net ; this . onFrame ( ) ; return ; } this . showbias = this . sim . state . bias ; this . net = net ; this . drawGraph ( ) ; } /** initialize graph nodes and edges */ drawGraph ( ) { this . nodes . clear ( ) ; this . edges . clear ( ) ; const net = this . net ; for ( let lid = 0 ; lid < net . layers . length ; lid ++ ) { const layer = net . layers [ lid ] ; let nid = 1 ; let layerWithBias = layer ; if ( this . showbias && net . biases [ lid ] ) layerWithBias = layer . concat ( net . biases [ lid ] ) ; for ( const neuron of layerWithBias ) { let type = "Hidden Neuron " + nid ++ ; let color = "#000" ; if ( neuron instanceof Net . InputNeuron ) { type = "Input: " + neuron . name ; if ( neuron . constant ) { color = NetworkVisualization . colors . autoencoder . bias ; } else color = NetworkVisualization . colors . autoencoder . input ; } if ( neuron instanceof Net . OutputNeuron ) { type = "Output: " + neuron . name ; color = NetworkVisualization . colors . autoencoder . output ; } if ( this . sim . state . type == "nn" && this . sim . currentTrainingDataPoint >= 0 ) { let v = 1 - Math . min ( Math . max ( neuron . output , 0 ) , 1 ) ; v = ( v * 250 ) | 0 ; color = "rgb(" + [ v , v , v ] + ")" ; } this . nodes . add ( { id : neuron . id , label : `${ type }` , level : lid , color : color } ) ; } } for ( const conn of net . connections ) { this . edges . add ( { id : this . edgeId ( conn ) , from : conn . inp . id , to : conn . out . id , arrows : "to" , label : conn . weight . toFixed ( 2 ) } ) ; } this . nodes . flush ( ) ; this . edges . flush ( ) ; this . graph . stabilize ( ) ; this . graph . fit ( ) ; } next = 0 ; showOutput = false ; /** calculate the visualization of the individual calculation steps for a single forward pass */ forwardPass ( data ) { if ( ! this . calculatedNetwork || this . updates == undefined || this . next == this . updates . length - 1 ) { this . onFrame ( ) ; this . next = 1 ; this . calculatedNetwork = true ; this . minYofLayer = { } ; this . net . setInputVectorsAndCalculate ( data . inputVector ! ) ; this . currentLayer = 1 ; this . currentNeuron = 0 ; this . parseData ( this . net ) ; this . currentlyDisplayingForwardPass = true ; } if ( this . showOutput ) { var tmp = this . next ; this . next = this . updates . length - 2 ; for ( var i = tmp ; i <= this . next ; i ++ ) this . applyUpdate ( this . updates [ i ] ) ; } if ( this . next >= this . updates . length - 1 ) { } else { if ( this . next == this . updates . length - 2 ) { this . alreadySetNet = false ; } this . applyUpdate ( this . updates [ this . next ++ ] ) ; } this . showOutput = false ; } applyUpdate ( update ) { this . graph . off ( "afterDrawing" ) ; this . graph . redraw ( ) ; if ( this . next != 0 ) { let x1 , y2 , h1 , px1 = 0 , py2 = 0 , timeDelayed = 0 , width = 0 , height = 0 ; var thisVar = this ; if ( update . layerNumber != 0 && ! this . showOutput ) this . graph . on ( "afterDrawing" , function ( ctx ) { ctx . lineWidth = "6" ; ctx . strokeStyle = "red" ; for ( var layer in update . layersUpdate ) { x1 = update . layersUpdate [ layer ] . maxX - 30 ; y2 = thisVar . minYofLayer [ layer ] - 30 ; h1 = 50 * update . layersUpdate [ layer ] . numberofNeuron + 5 ; if ( ! ( thisVar . net . layers [ layer ] [ 0 ] instanceof Net . OutputNeuron ) ) { px1 = x1 ; py2 = thisVar . minYofLayer [ Number ( layer ) - 1 ] - 30 ; timeDelayed = update . layersUpdate [ layer ] . timeDelayed ; height = 50 * thisVar . net . layers [ Number ( layer ) - 1 ] . length + 5 ; } else { px1 = - 30 ; py2 = thisVar . minYofLayer [ Number ( layer ) - 1 ] + x1 ; timeDelayed = thisVar . net . layers [ Number ( layer ) - 1 ] [ 0 ] . outputVector . length ; height = 55 ; } width = 50 * timeDelayed + 5 ; ctx . rect ( x1 , y2 , 55 , h1 ) ; ctx . stroke ( ) ; ctx . rect ( px1 , py2 , width ! , height ! ) ; ctx . stroke ( ) ; ctx . beginPath ( ) ; ctx . moveTo ( x1 , y2 + h1 ) ; ctx . lineTo ( px1 , py2 ) ; ctx . moveTo ( x1 + 55 , y2 + h1 ) ; ctx . lineTo ( px1 + width ! , py2 ) ; ctx . stroke ( ) ; } } ) ; } this . nodes . update ( update . nodes ) ; this . nodes . flush ( ) ; this . graph . stabilize ( ) ; } onView ( previouslyHidden , action ) { this . graph . fit ( ) ; } onHide ( ) { } /** parse network layout into weights graph ordering */ maxHeight = 0 ; parseData ( net ) { this . xyToConnectionTDNN = { } ; const data = [ ] ; let maxy = 0 ; let boxSize = 50 ; let lastUpdate = 1 ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass ) { this . updates = [ { nodes : [ ] , currentTime : 0 , layerNumber : 0 , layersUpdate : { } } ] ; this . createForwardPassStep ( ) ; } lastUpdate = 1 ; this . maxHeight = Math . max . apply ( null , net . layers . map ( layer => layer . length * boxSize ) ) ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass ) this . nodes . clear ( ) ; try { for ( let outputLayer = 0 ; outputLayer < net . layers . length ; outputLayer ++ ) { const layer = net . layers [ outputLayer ] ; const layerY = maxy + this . offsetBetweenLayers * boxSize ; let maxValue = 0 ; let minValue = 9999999999999 ; for ( let i = 0 ; i < layer . length ; i ++ ) { for ( let j = 0 ; j < layer [ i ] . outputVector . length ; j ++ ) { if ( maxValue < layer [ i ] . outputVector [ j ] ) maxValue = layer [ i ] . outputVector [ j ] ; if ( minValue > layer [ i ] . outputVector [ j ] ) minValue = layer [ i ] . outputVector [ j ] ; } } for ( let outputNeuron = 0 ; outputNeuron < layer . length ; outputNeuron ++ ) { const outN = layer [ outputNeuron ] ; maxy = Math . max ( maxy , layerY + layer . length * boxSize ) ; if ( outN instanceof Net . OutputNeuron ) { let p = { x : outputNeuron * boxSize , y : layerY + boxSize , z : outN . output } ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass ) p . z = 0 ; if ( this . maxHeight != layer . length * boxSize ) { p . y += ( this . maxHeight - layer . length ) / 2 ; } let node = { id : outN . id * 1000 + outputNeuron , size : boxSize / 2 , shape : "square" , color : { background : "rgba(0,0,0," + p . z + ")" } , title : "" + p . z . toFixed ( 2 ) , x : p . x , y : - p . y } ; var updateIndex = net . inputs [ 0 ] . outputVector . length - net . layers [ 1 ] [ 0 ] . timeDelayed + outputNeuron + 1 ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass ) { this . nodes . add ( node ) ; this . updates [ 0 ] . nodes . push ( node ) ; } else { this . updates [ updateIndex ] . nodes . push ( node ) ; var tmpMinX , tmpMinY ; if ( this . updates [ updateIndex ] . layersUpdate [ outputLayer ] == undefined ) { tmpMinX = 9999999999 ; } else { tmpMinX = this . updates [ updateIndex ] . layersUpdate [ outputLayer ] . maxX ; } if ( this . minYofLayer [ outputLayer ] == undefined ) tmpMinY = 99999999 ; else tmpMinY = this . minYofLayer [ outputLayer ] ; if ( node . y < tmpMinY ) this . minYofLayer [ outputLayer ] = node . y ; this . updates [ updateIndex ] . layersUpdate [ outputLayer ] = { boxSize : boxSize , maxX : node . x < tmpMinX ? node . x : tmpMinX , numberofNeuron : layer . length , timeDelayed : outN . timeDelayed } ; } data . push ( p ) ; this . xyToConnectionTDNN [ p . x + "," + p . y ] = [ outN . id , 0 , outN . output ] ; } else { for ( let timeStep = 0 ; timeStep < outN . outputVector . length ; timeStep ++ ) { const output = outN . outputVector [ timeStep ] == null ? 0 : outN . outputVector [ timeStep ] ; let p = { x : timeStep * boxSize , y : layerY + outputNeuron * boxSize , z : output } ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass && ! ( outN instanceof Net . InputNeuron ) ) p . z = 0 ; if ( this . maxHeight != layer . length ) { p . y += ( this . maxHeight - layer . length ) / 2 ; } let node = { id : outN . id * 1000 + timeStep , size : boxSize / 2 , shape : "square" , color : { background : "rgba(0,0,0," + ( ( p . z - minValue ) / ( maxValue - minValue ) ) . toFixed ( 1 ) + ")" } , title : "" + p . z . toFixed ( 2 ) , x : p . x , y : - p . y } ; data . push ( p ) ; if ( ! this . calculatedNetwork && ! this . currentlyDisplayingForwardPass ) { this . nodes . add ( node ) ; this . updates [ 0 ] . nodes . push ( node ) ; } else { var tmpMinY ; if ( outN instanceof Net . InputNeuron ) { this . updates [ 0 ] . nodes . push ( node ) ; } else { this . updates [ lastUpdate + timeStep ] . nodes . push ( node ) ; var tmpMinX ; if ( this . updates [ lastUpdate + timeStep ] . layersUpdate [ outputLayer ] == undefined ) { tmpMinX = 9999999999 ; } else { tmpMinX = this . updates [ lastUpdate + timeStep ] . layersUpdate [ outputLayer ] . maxX ; } this . updates [ lastUpdate + timeStep ] . layersUpdate [ outputLayer ] = { boxSize : boxSize , maxX : node . x < tmpMinX ? node . x : tmpMinX , numberofNeuron : layer . length , timeDelayed : outN . timeDelayed } ; } if ( this . minYofLayer [ outputLayer ] == undefined ) tmpMinY = 99999999 ; else tmpMinY = this . minYofLayer [ outputLayer ] ; if ( node . y < tmpMinY ) this . minYofLayer [ outputLayer ] = node . y ; } this . xyToConnectionTDNN [ p . x + "," + p . y ] = [ outN . id , timeStep , output ] ; } } } if ( layer [ 0 ] instanceof Net . OutputNeuron ) lastUpdate += layer . length ; else { if ( layer [ 0 ] instanceof Net . TimeDelayedNeuron ) lastUpdate += layer [ 0 ] . timeDelayed + 1 ; } } } catch ( e ) { console . log ( e ) ; } let thisVar = this ; this . updates [ 0 ] . numberofneuron = this . net . layers [ 0 ] . length ; this . nodes . flush ( ) ; this . graph . fit ( ) ; return data ; } createForwardPassStep ( ) { for ( let neuron = 0 ; neuron < this . net . outputs . length ; neuron ++ ) { this . updates . push ( { nodes : [ ] , layersUpdate : { } } ) ; } for ( let outputTime = 0 ; outputTime <= this . net . inputs [ 0 ] . outputVector . length - this . net . inputs [ 0 ] . outputs [ 0 ] . out . timeDelayed ; outputTime ++ ) { this . updates . push ( { nodes : [ ] , layersUpdate : { } } ) ; } } onNetworkLoaded ( net ) { if ( ! net . isTDNN ) { this . actions = [ ] ; return ; } this . actions = [ "TDNN Graph" ] ; this . net = net ; this . parseData ( net ) ; this . next = 0 ; this . applyUpdate ( this . updates [ this . next ++ ] ) ; } onFrame ( ) { if ( ! this . net . isTDNN ) return ; this . calculatedNetwork = false ; this . graph . off ( "afterDrawing" ) ; this . graph . redraw ( ) ; } }