import { merge , chunk , flatten , omit } from 'lodash' ; import { argv } from 'yargs' ; import { Logger } from 'kibana/server' ; import { CollectTelemetryParams } from '../../server/lib/apm_telemetry/collect_data_telemetry' ; import { downloadTelemetryTemplate } from '../shared/download_telemetry_template' ; import { mergeApmTelemetryMapping } from '../../common/apm_telemetry' ; import { generateSampleDocuments } from './generate_sample_documents' ; import { readKibanaConfig } from '../shared/read_kibana_config' ; import { getHttpAuth } from '../shared/get_http_auth' ; import { createOrUpdateIndex } from '../shared/create_or_update_index' ; import { getEsClient } from '../shared/get_es_client' ; async function uploadData ( ) { const githubToken = process . env . GITHUB_TOKEN ; if ( ! githubToken ) { throw new Error ( 'GITHUB_TOKEN was not provided.' ) ; } const xpackTelemetryIndexName = 'xpack-phone-home' ; const telemetryTemplate = await downloadTelemetryTemplate ( { githubToken , } ) ; const config = readKibanaConfig ( ) ; const httpAuth = getHttpAuth ( config ) ; const client = getEsClient ( { node : config [ 'elasticsearch.hosts' ] , ... ( httpAuth ? { auth : { ... httpAuth , username : 'elastic' } , } : { } ) , } ) ; const newTemplate = omit ( mergeApmTelemetryMapping ( merge ( telemetryTemplate , { index_patterns : undefined , settings : { index : { mapping : { total_fields : { limit : 10000 } } } , } , } ) ) , 'index_patterns' ) ; await createOrUpdateIndex ( { indexName : xpackTelemetryIndexName , client , template : newTemplate , clear : ! ! argv . clear , } ) ; const sampleDocuments = await generateSampleDocuments ( { collectTelemetryParams : { logger : console as unknown as Logger , indices : { transaction : config [ 'xpack.apm.indices.transaction' ] , metric : config [ 'xpack.apm.indices.metric' ] , error : config [ 'xpack.apm.indices.error' ] , span : config [ 'xpack.apm.indices.span' ] , onboarding : config [ 'xpack.apm.indices.onboarding' ] , sourcemap : config [ 'xpack.apm.indices.sourcemap' ] , apmCustomLinkIndex : '.apm-custom-links' , apmAgentConfigurationIndex : '.apm-agent-configuration' , } , search : ( body ) => { return client . search ( body ) as Promise < any > ; } , indicesStats : ( body ) => { return client . indices . stats ( body ) ; } , transportRequest : ( ( params ) => { return ; } ) as CollectTelemetryParams [ 'transportRequest' ] , } , } ) ; const chunks = chunk ( sampleDocuments , 250 ) ; await chunks . reduce < Promise < any > > ( ( prev , documents ) => { return prev . then ( async ( ) => { const body = flatten ( documents . map ( ( doc ) => [ { index : { _index : xpackTelemetryIndexName } } , doc , ] ) ) ; return client . bulk ( { body , refresh : 'wait_for' , } ) . then ( ( response ) => { if ( response . errors ) { const firstError = response . items . filter ( ( item ) => item . index . status >= 400 ) [ 0 ] . index . error ; throw new Error ( `Failed to upload documents: ${ firstError . reason } ` ) ; } } ) ; } ) ; } , Promise . resolve ( ) ) ; } uploadData ( ) . catch ( ( e ) => { if ( 'response' in e ) { if ( typeof e . response === 'string' ) { console . log ( e . response ) ; } else { console . log ( JSON . stringify ( e . response , [ 'status' , 'statusText' , 'headers' , 'data' ] , 2 ) ) ; } } else { console . log ( e ) ; } process . exit ( 1 ) ; } ) . then ( ( ) => { console . log ( 'Finished uploading generated telemetry data' ) ; } ) ;