import { sleep } from '@/infrastructure/Threading/AsyncSleep' ; import { IUrlStatus } from './IUrlStatus' ; import { getUrlStatus , IRequestOptions } from './Requestor' ; import { groupUrlsByDomain } from './UrlPerDomainGrouper' ; export async function getUrlStatusesInParallel ( urls , options ? ) { const uniqueUrls = Array . from ( new Set ( urls ) ) ; options = { ... DefaultOptions , ... options } ; console . log ( 'Options: ' , options ) ; const results = await request ( uniqueUrls , options ) ; return results ; } export interface IBatchRequestOptions { domainOptions ? : IDomainOptions ; requestOptions ? : IRequestOptions ; } interface IDomainOptions { sameDomainParallelize ? : boolean ; sameDomainDelayInMs ? : number ; } const DefaultOptions = { domainOptions : { sameDomainParallelize : false , sameDomainDelayInMs : 3 * 1000 , } , requestOptions : { retryExponentialBaseInMs : 5 * 1000 , additionalHeaders : { } , } , } ; function request ( urls , options ) { if ( ! options . domainOptions . sameDomainParallelize ) { return runOnEachDomainWithDelay ( urls , ( url ) => getUrlStatus ( url , options . requestOptions ) , options . domainOptions . sameDomainDelayInMs ) ; } return Promise . all ( urls . map ( ( url ) => getUrlStatus ( url , options . requestOptions ) ) ) ; } async function runOnEachDomainWithDelay ( urls , action , delayInMs ) { const grouped = groupUrlsByDomain ( urls ) ; const tasks = grouped . map ( async ( group ) => { const results = new Array < IUrlStatus > ( ) ; for ( const url of group ) { const status = await action ( url ) ; results . push ( status ) ; if ( results . length !== group . length ) { await sleep ( delayInMs ) ; } } return results ; } ) ; const r = await Promise . all ( tasks ) ; return r . flat ( ) ; }