kenken999 commited on
Commit
62cf1c8
·
1 Parent(s): 7ad9796

youtub html update

Browse files
Files changed (4) hide show
  1. chat_history.db +2 -2
  2. output.txt +458 -0
  3. staticfiles/index.html +520 -520
  4. workspace/sample_html.html +1 -0
chat_history.db CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f0a9c767642ae220658b59d42af845649ed60a466c52d70d147006d0ac9e3b3a
3
- size 2084864
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7dce877f743bc84a2525c452bc62fe884ad45a1bae5b1050d788f02e85c3543a
3
+ size 2113536
output.txt CHANGED
@@ -2078,3 +2078,461 @@ No output available on console.
2078
  ```
2079
 
2080
  新しい行を追記します。
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2078
  ```
2079
 
2080
  新しい行を追記します。
2081
+ **Plan Recap:** We are creating a sample HTML code.
2082
+
2083
+ **Step 1: Create a sample HTML code**
2084
+
2085
+ I will create a sample HTML file in the './workspace' directory.
2086
+
2087
+
2088
+
2089
+
2090
+ ```python
2091
+ <!DOCTYPE >
2092
+ <>
2093
+ <head>
2094
+ <title>Sample HTML Page</title>
2095
+ </head>
2096
+ <body>
2097
+ <h1>Welcome to my sample HTML page!</h1>
2098
+ <p>This is a sample HTML page.</p>
2099
+ </body>
2100
+ </>
2101
+ ```
2102
+ ```code
2103
+ HTML being displayed on the user's machine...
2104
+
2105
+ ```
2106
+
2107
+ ```python
2108
+ <!DOCTYPE >
2109
+ <>
2110
+ <head>
2111
+ <title>Sample HTML Page</title>
2112
+ </head>
2113
+ <body>
2114
+ <h1>Welcome to my sample HTML page!</h1>
2115
+ <p>This is a sample HTML page.</p>
2116
+ </body>
2117
+ </>
2118
+ ```
2119
+ ```code
2120
+ Traceback (most recent call last):
2121
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/respond.py", line 220, in respond
2122
+ for line in interpreter.computer.run(language, code, stream=True):
2123
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/terminal/terminal.py", line 82, in _streaming_run
2124
+ for chunk in self._active_languages[language].run(code):
2125
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/terminal/languages/html.py", line 25, in run
2126
+ base64 = html_to_png_base64(code)
2127
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/utils/html_to_png_base64.py", line 13, in html_to_png_base64
2128
+ hti = Html2Image()
2129
+ File "/usr/local/lib/python3.10/site-packages/html2image/html2image.py", line 107, in __init__
2130
+ self.browser = browser_class(
2131
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 189, in __init__
2132
+ super().__init__(executable=executable, flags=flags, print_command=print_command, disable_logging=disable_logging, use_new_headless=use_new_headless)
2133
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chromium.py", line 30, in __init__
2134
+ self.executable = executable
2135
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 197, in executable
2136
+ self._executable = _find_chrome(value)
2137
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 159, in _find_chrome
2138
+ raise FileNotFoundError(
2139
+ FileNotFoundError: Could not find a Chrome executable on this machine, please specify it yourself.
2140
+
2141
+
2142
+ ```
2143
+
2144
+ 新しい行を追記します。
2145
+ **Plan Recap:** We are creating a sample HTML code.
2146
+
2147
+ **Step 1: Create a sample HTML file**
2148
+
2149
+ I will create a sample HTML file in the './workspace' directory.
2150
+
2151
+
2152
+ ```python
2153
+ <!DOCTYPE >
2154
+ <>
2155
+ <head>
2156
+ <title>Sample HTML Page</title>
2157
+ </head>
2158
+ <body>
2159
+ <h1>Welcome to my sample HTML page</h1>
2160
+ </body>
2161
+ </>
2162
+ ```
2163
+ ```code
2164
+ HTML being displayed on the user's machine...
2165
+
2166
+ ```
2167
+
2168
+ ```python
2169
+ <!DOCTYPE >
2170
+ <>
2171
+ <head>
2172
+ <title>Sample HTML Page</title>
2173
+ </head>
2174
+ <body>
2175
+ <h1>Welcome to my sample HTML page</h1>
2176
+ </body>
2177
+ </>
2178
+ ```
2179
+ ```code
2180
+ Traceback (most recent call last):
2181
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/respond.py", line 220, in respond
2182
+ for line in interpreter.computer.run(language, code, stream=True):
2183
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/terminal/terminal.py", line 82, in _streaming_run
2184
+ for chunk in self._active_languages[language].run(code):
2185
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/terminal/languages/html.py", line 25, in run
2186
+ base64 = html_to_png_base64(code)
2187
+ File "/usr/local/lib/python3.10/site-packages/interpreter/core/computer/utils/html_to_png_base64.py", line 13, in html_to_png_base64
2188
+ hti = Html2Image()
2189
+ File "/usr/local/lib/python3.10/site-packages/html2image/html2image.py", line 107, in __init__
2190
+ self.browser = browser_class(
2191
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 189, in __init__
2192
+ super().__init__(executable=executable, flags=flags, print_command=print_command, disable_logging=disable_logging, use_new_headless=use_new_headless)
2193
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chromium.py", line 30, in __init__
2194
+ self.executable = executable
2195
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 197, in executable
2196
+ self._executable = _find_chrome(value)
2197
+ File "/usr/local/lib/python3.10/site-packages/html2image/browsers/chrome.py", line 159, in _find_chrome
2198
+ raise FileNotFoundError(
2199
+ FileNotFoundError: Could not find a Chrome executable on this machine, please specify it yourself.
2200
+
2201
+
2202
+ ```
2203
+
2204
+ 新しい行を追記します。
2205
+ **Plan Recap:** We are creating a sample HTML file.
2206
+
2207
+ **Step 1: Create a sample HTML file**
2208
+
2209
+ I will create a sample HTML file in the './workspace' directory.
2210
+
2211
+
2212
+
2213
+ ```python
2214
+ HTML
2215
+ <!DOCTYPE html>
2216
+ <html>
2217
+ <head>
2218
+ <title>Sample HTML Page</title>
2219
+ </head>
2220
+ <body>
2221
+ <h1>Welcome to my sample HTML page</h1>
2222
+ </body>
2223
+ </html>
2224
+ ```
2225
+ ```code
2226
+ Cell In[15], line 1
2227
+
2228
+ ^
2229
+ SyntaxError: invalid syntax
2230
+
2231
+ No output available on console.
2232
+ ```
2233
+
2234
+
2235
+ ```
2236
+
2237
+ 新しい行を追記します。
2238
+ **Plan Recap:** We are creating a sample HTML file.
2239
+
2240
+ **Step 1: Create a sample HTML file**
2241
+
2242
+ I will create a sample HTML file in the './workspace' directory.
2243
+
2244
+
2245
+
2246
+
2247
+ ```python
2248
+ mkdir ./workspace
2249
+ echo "<!DOCTYPE html><html><head><title>Sample HTML Page</title></head><body><h1>Welcome to my sample HTML page</h1></body></html>" > ./workspace/sample_html.html
2250
+ ```
2251
+ ```code
2252
+
2253
+
2254
+
2255
+
2256
+ mkdir: cannot create directory ‘./workspace’: File exists
2257
+
2258
+
2259
+
2260
+
2261
+
2262
+ No output available on console.
2263
+ ```
2264
+
2265
+
2266
+ ```
2267
+
2268
+ 新しい行を追記します。
2269
+ **Plan Recap:** We are creating a sample HTML file and displaying it using Python's print function.
2270
+
2271
+ **Step 1: Create a sample HTML file**
2272
+
2273
+ I will create a sample HTML file in the './workspace' directory.
2274
+
2275
+
2276
+
2277
+
2278
+
2279
+ ```python
2280
+ html_content = """
2281
+ <!DOCTYPE html>
2282
+ <html>
2283
+ <head>
2284
+ <title>Sample HTML Page</title>
2285
+ </head>
2286
+ <body>
2287
+ <h1>Welcome to my sample HTML page</h1>
2288
+ </body>
2289
+ </html>
2290
+ """
2291
+ print(html_content)
2292
+ ```
2293
+ ```code
2294
+
2295
+ <!DOCTYPE html>
2296
+ <html>
2297
+ <head>
2298
+ <title>Sample HTML Page</title>
2299
+ </head>
2300
+ <body>
2301
+ <h1>Welcome to my sample HTML page</h1>
2302
+ </body>
2303
+ </html>
2304
+
2305
+
2306
+ No output available on console.
2307
+ ```
2308
+
2309
+
2310
+ ```
2311
+
2312
+ 新しい行を追記します。
2313
+ It seems like you want me to execute a curl command based on the provided JSON data.
2314
+
2315
+ **Plan:**
2316
+
2317
+ 1. Extract the necessary information from the JSON data to construct the curl command.
2318
+ 2. Execute the curl command using the extracted information.
2319
+
2320
+ **Let's start by extracting the necessary information:**
2321
+
2322
+ From the provided JSON data, I can see that we need to make a POST request tohttps://api.dify.ai/v1/chat-messages with a JSON body and an Authorization header.
2323
+
2324
+ Here's the extracted information:
2325
+
2326
+ * Method: POST
2327
+ * URL: https://api.dify.ai/v1/chat-messages
2328
+ * Header: Authorization: Bearer app-WM0Jmn7H2pG0gVyDEfvKpVTi
2329
+ * JSON Body:
2330
+
2331
+ ```python
2332
+ {
2333
+ "inputs": {
2334
+ "n8n": "{{ $.body.events[0].toJsonString().urlEncode() }}",
2335
+ "type": "{{ $.body.events[0].message.type }}",
2336
+ "lineid": "{{ $.body.events[0].source.userId }}"
2337
+ },
2338
+ "query": "{{ $.body.events[0].message.text }}",
2339
+ "response_mode": "streaming",
2340
+ "conversation_id": "",
2341
+ "user": "{{ $.body.events[0].source.userId }}",
2342
+ "files": [
2343
+ {
2344
+ "type": "image",
2345
+ "transfer_method": "remote_url",
2346
+ "url": "https://cloud.dify.ai/logo/logo-site.png"
2347
+ }
2348
+ ]
2349
+ }
2350
+ ```
2351
+ ```code
2352
+ json disabled or not supported.
2353
+
2354
+ ```
2355
+
2356
+ 新しい行を追記します。
2357
+ Here is the plan:
2358
+
2359
+ **Plan:**
2360
+ 1. Usecurl to send a POST request to the specified URL with the provided JSON body.
2361
+ 2. Print the response.
2362
+
2363
+ Let's execute the code:
2364
+
2365
+ ```python
2366
+ bash
2367
+ curl -X POST \
2368
+ https://api.dify.ai/v1/chat-messages \
2369
+ -H 'Authorization: Bearer app-WM0Jmn7H2pG0gVyDEfvKpVTi' \
2370
+ -H 'Content-Type: application/json' \
2371
+ -d '{
2372
+ "inputs": {"n8n":"{{$json.body.events[0].toJsonString().urlEncode()}}","type":"{{ $json.body.events[0].message.type }}","lineid":"{{ $json.body.events[0].source.userId }}"},
2373
+ "query": "{{ $json.body.events[0].message.text }}",
2374
+ "response_mode": "streaming",
2375
+ "conversation_id": "",
2376
+ "user": "{{ $json.body.events[0].source.userId }}",
2377
+ "files": [
2378
+ {"type": "image","transfer_method": "remote_url","url": "https://cloud.dify.ai/logo/logo-site.png"}
2379
+ ]
2380
+ }'
2381
+ ```
2382
+ ```code
2383
+ Cell In[37], line 7
2384
+ -d '{
2385
+ ^
2386
+ SyntaxError: unterminated string literal (detected at line 7)
2387
+
2388
+ No output available on console.
2389
+ ```
2390
+
2391
+
2392
+ ```
2393
+
2394
+ 新しい行を追記します。
2395
+ Here is the plan:
2396
+ **Plan:**
2397
+ 1. Convert the given JSON data to a CURL command.
2398
+ 2. Execute the CURL command on the user's machine.
2399
+ 3. Print the output.
2400
+
2401
+ Let's start by converting the JSON data to a CURL command.
2402
+
2403
+
2404
+ ```python
2405
+ curl
2406
+ -X POST
2407
+ -H "Authorization: Bearer vzn2zssSEtHb/IVgMlgY1KxLQUfmUXRuiQiQkZLRVsHOeQBp9KsU5/M0i/2XKtw1K+eXN4PyjHQKcG5Vj5l+4e5CGAOQa/veKWdn83UPJQJU17FC9ONucjc84gvNFcRAy4IZcFcMky2PTzazf0KGiFGUYhWQfeY8sLGRXgo3xvw="
2408
+ -H "Content-Type: application/json"
2409
+ -d '{"to": "{{ $('Webhook3').item.json.body.events[0].source.userId }}","messages": [{"type": "text","text": "(AIが査定データを分析しています…しばらくお待ちください。)"}]}'
2410
+ https://api.line.me/v2/bot/message/push
2411
+ ```
2412
+ ```code
2413
+ Cell In[44], line 1
2414
+
2415
+ ^
2416
+ SyntaxError: invalid syntax
2417
+
2418
+ No output available on console.
2419
+ ```
2420
+
2421
+
2422
+ ```
2423
+
2424
+ 新しい行を追記します。
2425
+ Here is the plan:
2426
+ **Plan:** Parse the given JSON string into a Python object.
2427
+
2428
+ Here is the code to execute:
2429
+
2430
+ ```python
2431
+ import json
2432
+
2433
+ data = '''
2434
+ {
2435
+ "result": "{\"type\":\"message\",\"message\":{\"type\":\"text\",\"id\":\"543520108811911313\",\"quoteToken\":\"-1orMoUAL1hldGPFm_7amYkbUr1gO96UdQx0UEBcROTHcgbp3bJClGtlvaYD8bi8OFVqh8ou-lJlBXgzhQNpraqP8ra5N4Snj8GEZS8lRf64JpqT9TgjbDyoxrlOlEbZ7KwHDvvKeyyLaiO8L4izUw\",\"text\":\"テスト\"},\"webhookEventId\":\"01JHGGAWR4ZS8ADWMYBWNSTA3H\",\"deliveryContext\":{\"isRedelivery\":false},\"timestamp\":1736794534153,\"source\":{\"type\":\"user\",\"userId\":\"U7a972c5cc46e966de21ff1cd3adc6d52\"},\"replyToken\":\"90d05bcce0f04c908ccb0120872c53f5\",\"mode\":\"active\"}"
2436
+ }
2437
+ '''
2438
+
2439
+ data_dict = json.loads(data)
2440
+ result_dict = json.loads(data_dict['result'])
2441
+
2442
+ print(result_dict)
2443
+ ```
2444
+ ```code
2445
+ ---------------------------------------------------------------------------
2446
+ JSONDecodeError Traceback (most recent call last)
2447
+ Cell In[51], line 10
2448
+ 8 pass
2449
+ 9 print('##active_line9##')
2450
+ ---> 10 data_dict = json.loads(data)
2451
+ 11 print('##active_line10##')
2452
+ 12 result_dict = json.loads(data_dict['result'])
2453
+
2454
+ File /usr/local/lib/python3.10/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
2455
+ 341 s = s.decode(detect_encoding(s), 'surrogatepass')
2456
+ 343 if (cls is None and object_hook is None and
2457
+ 344 parse_int is None and parse_float is None and
2458
+ 345 parse_constant is None and object_pairs_hook is None and not kw):
2459
+ --> 346 return _default_decoder.decode(s)
2460
+ 347 if cls is None:
2461
+ 348 cls = JSONDecoder
2462
+
2463
+ File /usr/local/lib/python3.10/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
2464
+ 332 def decode(self, s, _w=WHITESPACE.match):
2465
+ 333 """Return the Python representation of s (a str instance
2466
+ 334 containing a JSON document).
2467
+ 335
2468
+ 336 """
2469
+ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2470
+ 338 end = _w(s, end).end()
2471
+ 339 if end != len(s):
2472
+
2473
+ File /usr/local/lib/python3.10/json/decoder.py:353, in JSONDecoder.raw_decode(self, s, idx)
2474
+ 344 """Decode a JSON document from s (a str beginning with
2475
+ 345 a JSON document) and return a 2-tuple of the Python
2476
+ 346 representation and the index in s where the document ended.
2477
+ (...)
2478
+ 350
2479
+ 351 """
2480
+ 352 try:
2481
+ --> 353 obj, end = self.scan_once(s, idx)
2482
+ 354 except StopIteration as err:
2483
+ 355 raise JSONDecodeError("Expecting value", s, err.value) from None
2484
+
2485
+ JSONDecodeError: Expecting ',' delimiter: line 3 column 16 (char 18)
2486
+ No output available on console.
2487
+ ```
2488
+
2489
+
2490
+ ```
2491
+
2492
+ 新しい行を追記します。
2493
+ Here is the plan:
2494
+ **Plan:** Convert the obtained JSON to an object using the provided source code.
2495
+
2496
+ Here is the JSON string to convert:
2497
+
2498
+ **No JSON string provided**
2499
+
2500
+ Please provide the JSON string to convert.
2501
+
2502
+ Once the JSON string is provided, I can assist with converting it to a Python object using the given source code.
2503
+
2504
+ Here is the code to execute:
2505
+
2506
+
2507
+ ```python
2508
+ import json
2509
+
2510
+ def main(encoded_url: str) -> dict:
2511
+ import urllib.parse
2512
+ decoded_url = urllib.parse.unquote(encoded_url)
2513
+ return {'result': decoded_url}
2514
+
2515
+ # Provide the JSON string here
2516
+ json_string = "" # Replace with the actual JSON string
2517
+
2518
+ # Convert JSON string to a Python object
2519
+ _object = main(json_string)
2520
+
2521
+ print(python_object)
2522
+ ```
2523
+ ```code
2524
+ ---------------------------------------------------------------------------
2525
+ NameError Traceback (most recent call last)
2526
+ Cell In[58], line 28
2527
+ 26 pass
2528
+ 27 print('##active_line14##')
2529
+ ---> 28 print(python_object)
2530
+
2531
+ NameError: name 'python_object' is not defined
2532
+ No output available on console.
2533
+ ```
2534
+
2535
+
2536
+ ```
2537
+
2538
+ 新しい行を追記します。
staticfiles/index.html CHANGED
@@ -9,545 +9,545 @@
9
  <script src="./aivtuber.js"></script>
10
  <title>Live2Dサンプル</title>
11
  <!-- Live2D -->
12
- <script src="./live2dcubismcore.js"></script>
13
- <script src="//cdn.jsdelivr.net/gh/dylanNew/live2d/webgl/Live2D/lib/live2d.min.js"></script>
14
- <!-- PixiJS -->
15
- <script src="//cdnjs.cloudflare.com/ajax/libs/pixi.js/5.1.3/pixi.min.js"></script>
16
- <script src="//cdn.jsdelivr.net/npm/pixi-live2d-display/dist/index.min.js"></script>
17
- <!-- Kalidokit -->
18
- <script src="//cdn.jsdelivr.net/npm/[email protected]/dist/kalidokit.umd.js"></script>
19
  </head>
20
 
21
  <body>
22
 
23
- <video id="my-video" autoplay></video>
24
-
25
- <script>
26
-
27
- 'use strict';
28
- class MiiboAvatar {
29
- constructor(config) {
30
- this.container = config.container;
31
- this.speechToTextOptions = config.option.speech_to_text;
32
- this.miiboOptions = config.option.miibo;
33
- this.didOptions = config.option.d_id;
34
- this.initialize();
35
- }
36
-
37
- initialize() {
38
- //alert("38");
39
- const RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection;
40
-
41
- this.videoElement = document.getElementById(this.container);
42
- this.videoElement.setAttribute('playsinline', '');
43
- this.playIdleVideo();
44
-
45
- this.createNewStream();
46
- this.rec = new webkitSpeechRecognition()
47
-
48
- this.speaching = false;
49
- this.processing = false;
50
- this.streams = [];
51
- }
52
-
53
- async createNewStream() {
54
- //alert("53")
55
- try {
56
- this.stopAllStreams();
57
- this.closePC();
58
-
59
- let presenter = {"source_url": this.didOptions.presenter.image_url}
60
- const sessionResponse = await this.fetchWithRetries(`https://api.d-id.com/${this.didOptions.service}/streams`, {
61
- method: 'POST',
62
- headers: {
63
- Authorization: `Basic ${this.didOptions.key}`,
64
- 'Content-Type': 'application/json',
65
- },
66
- body: JSON.stringify(presenter),
67
- });
68
-
69
- const { id: newStreamId, offer, ice_servers: iceServers, session_id: newSessionId } = await sessionResponse.json();
70
- this.streamId = newStreamId;
71
- this.sessionId = newSessionId;
72
-
73
- try {
74
- this.sessionClientAnswer = await this.createPeerConnection(offer, iceServers);
75
- } catch (e) {
76
- console.log('Error creating peer connection:', e);
77
- this.stopAllStreams();
78
- this.closePC();
79
- return;
80
- }
81
 
82
- const sdpResponse = await fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}/sdp`, {
83
- method: 'POST',
84
- headers: {
85
- Authorization: `Basic ${this.didOptions.key}`,
86
- 'Content-Type': 'application/json',
87
- },
88
- body: JSON.stringify({
89
- answer: this.sessionClientAnswer,
90
- session_id: this.sessionId,
91
- }),
92
- });
93
-
94
- // Handle sdpResponse if needed
95
- } catch (error) {
96
- console.log('Error creating new stream:', error);
97
- // Handle error
98
- }
99
- }
100
-
101
- speechRecogInit() {
102
- this.audioContext = new (window.AudioContext || window.webkitAudioContext)()
103
-
104
- if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
105
- navigator.mediaDevices.getUserMedia({audio: true}).then((stream) => {
106
- var input = this.audioContext.createMediaStreamSource(stream)
107
- this.audioContext.resume()
108
- this.recorder = new Recorder(input)
109
- })
110
- }
111
- }
112
-
113
-
114
- startRecording() {
115
- this.recorder && this.recorder.record();
116
- }
117
-
118
- stopRecording() {
119
- this.playLoadingVideo();
120
- this.recorder && this.recorder.stop();
121
- this.audioRecognize();
122
- this.recorder.clear();
123
- }
124
-
125
- audioRecognize() {
126
- this.recorder && this.recorder.exportWAV((blob) => {
127
- let reader = new FileReader()
128
- reader.onload = () => {
129
- let result = new Uint8Array(reader.result)
130
-
131
- let data = {
132
- "config": {
133
- "encoding": "LINEAR16",
134
- "languageCode": "ja-JP",
135
- "alternativeLanguageCodes": ["en-US"],//,"cmn-CN","ko-KR"],
136
- "audio_channel_count": 2
137
- },
138
- "audio": {
139
- "content": this.arrayBufferToBase64(result)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
140
  }
141
  }
142
- fetch('https://speech.googleapis.com/v1/speech:recognize?key=' + this.speechToTextOptions.api_key, {
143
- method: 'POST',
144
- headers: {
145
- 'Content-Type': 'application/json; charset=utf-8'
146
- },
147
- body: JSON.stringify(data)
148
- }).then((response) => {
149
- return response.text()
150
- }).then((text) => {
151
- let result_json = JSON.parse(text)
152
- text = result_json.results[0].alternatives[0].transcript;
153
- this.languageCode = result_json.results[0].languageCode;
154
- this.ask(text)
155
- })
156
- }
157
- reader.readAsArrayBuffer(blob)
158
- })
159
- }
160
- // Chrome Only
161
- autoRecognizeMessage(message) {
162
- handleLiveComment("test",message)
163
- //alert("start word test")
164
- this.rec.continuous = false
165
- this.rec.interimResults = false
166
- this.rec.lang = 'ja-JP'
167
- this.processing = true
168
- this.playLoadingVideo();
169
-
170
- this.rec.stop()
171
- alert("startssssssssssssssssssssssssss")
172
- const { transcript } = message
173
- this.ask(transcript);
174
- //for (var i = e.resultIndex; i < e.results.length; i++) {
175
- // if (!e.results[i].isFinal) continue
176
- //
177
- // const { transcript } = e.results[i][0]
178
- // this.ask(transcript);
179
- //}
180
- //this.rec.onresult = (e) => {
181
-
182
- //}
183
-
184
- this.rec.onend = () => { this.autoRecognizeRestart() }
185
- this.rec.start()
186
- }
187
- // Chrome Only
188
- autoRecognize() {
189
- this.rec.continuous = false
190
- this.rec.interimResults = false
191
- this.rec.lang = 'ja-JP'
192
-
193
- this.rec.onresult = (e) => {
194
- this.processing = true
195
- this.playLoadingVideo();
196
-
197
- this.rec.stop()
198
- //alert("start")
199
-
200
- for (var i = e.resultIndex; i < e.results.length; i++) {
201
- if (!e.results[i].isFinal) continue
202
-
203
- const { transcript } = e.results[i][0]
204
- this.ask(transcript);
205
- }
206
- }
207
-
208
- this.rec.onend = () => { this.autoRecognizeRestart() }
209
- this.rec.start()
210
- }
211
-
212
- autoRecognizeRestart() {
213
- if (this.processing) {
214
- setTimeout(() => {this.autoRecognizeRestart()}, 1000)
215
- } else {
216
- this.rec.start()
217
- }
218
- }
219
-
220
- arrayBufferToBase64(buffer) {
221
- let binary = ''
222
- let bytes = new Uint8Array(buffer);
223
- let len = bytes.byteLength
224
- for (let i = 0; i < len; i++) {
225
- binary += String.fromCharCode(bytes[i])
226
- }
227
- return window.btoa(binary)
228
- }
229
-
230
- ask(message) {
231
- //youtubeからの質問
232
- alert("203 ask"+message);
233
- //ask(start);
234
- this.getMiiboResponse(message);
235
- }
236
-
237
- async getMiiboResponse(utterance) {
238
- alert("209 getMiiboResponse"+utterance)
239
- const params = {
240
- api_key: this.miiboOptions.api_key,
241
- agent_id: this.miiboOptions.agent_id,
242
- uid: this.miiboOptions.user_id,
243
- stream: true,
244
- utterance: utterance
245
- };
246
-
247
- try {
248
- const res = await fetch("https://api-mebo.dev/api", {
249
- method: "POST",
250
- headers: { "Content-Type": "application/json" },
251
- body: JSON.stringify(params),
252
- });
253
-
254
- const reader = res.body.getReader();
255
- const decoder = new TextDecoder();
256
- let output = "";
257
- let sentences = [];
258
- let current_index = 0;
259
-
260
- const read = async () => {
261
- const { done, value } = await reader.read();
262
- if (done) return;
263
-
264
- let dataString = decoder.decode(value).split("\n").filter(x => x != "");
265
-
266
- try {
267
- let responseData = JSON.parse(dataString[dataString.length - 1]);
268
-
269
- output = responseData.bestResponse.utterance.split("\n").filter(x => x.trim() != "").join("\n");
270
- sentences = output.replace(/[。、\.\,\!\?!?]/,".").split(".")
271
- if (this.didOptions.priority == "speed" && current_index == 0 && current_index + 1 < sentences.length) {
272
- this.startTalk(sentences[current_index++])
273
  }
274
- } catch(e) {
275
- console.log(e);
276
  }
277
 
278
- return read();
279
- };
280
-
281
- await read();
282
- reader.releaseLock();
283
-
284
- this.startTalk(sentences.slice(current_index).join("。"));
285
- } catch(error) {
286
- console.log("Error fetching AI response: ", error);
287
- }
288
- }
289
-
290
- async startTalk(input) {
291
- if (this.peerConnection?.signalingState === 'stable' || this.peerConnection?.iceConnectionState === 'connected') {
292
-
293
- const gender = this.didOptions.presenter.gender;
294
- let voice_id = this.didOptions.presenter.voice_id || "";
295
-
296
- if (voice_id == "") {
297
- switch (this.languageCode) {
298
- case "en-us":
299
- voice_id = gender == "male" ? "en-US-GuyNeural" : "en-US-AriaNeural"
300
- break;
301
- case "ko-kr":
302
- voice_id = gender == "male" ? "ko-KR-InJoonNeural" : "ko-KR-YuJinNeural"
303
- break;
304
- case "cmn-CN":
305
- voice_id = gender == "male" ? "zh-CN-YunjianNeural" : "zh-CN-XiaohanNeural"
306
- break;
307
- default:
308
- voice_id = gender == "male" ? "ja-JP-KeitaNeural" : "ja-JP-NanamiNeural"
309
  }
310
- }
311
 
312
- const requestOptions = {
313
- method: 'POST',
314
- headers: {
315
- Authorization: `Basic ${this.didOptions.key}`,
316
- 'Content-Type': 'application/json',
317
- },
318
- body: JSON.stringify({
319
- script: {
320
- type: "text",
321
- subtitles: false,
322
- provider: {
323
- type: "microsoft",
324
- voice_id: voice_id
325
- },
326
- ssml: false,
327
- input: input
328
- },
329
- config: {
330
- fluent: false,
331
- pad_audio: 0,
332
- align_driver: false,
333
- stitch: false,
334
- auto_match: false,
335
- sharpen: false,
336
- normalization_factor: 0
337
- },
338
- session_id: this.sessionId,
339
- }),
340
- };
341
 
342
- if (this.didOptions.service === 'clips') {
343
- requestOptions.body.background = { color: '#FFFFFF' };
344
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
345
 
346
- try {
347
- const playResponse = await this.fetchWithRetries(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}`, requestOptions);
348
- // Handle response if needed
349
- } catch (error) {
350
- console.error('Error starting talk:', error);
351
- // Handle error
352
- }
353
- }
354
- }
355
-
356
- async destoryTalk(input) {
357
- try {
358
- await fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}`, {
359
- method: 'DELETE',
360
- headers: {
361
- Authorization: `Basic ${this.didOptions.key}`,
362
- 'Content-Type': 'application/json',
363
- },
364
- body: JSON.stringify({ session_id: this.sessionId }),
365
- });
366
-
367
- await this.stopAllStreams();
368
- await this.closePC();
369
- } catch (error) {
370
- console.error('Error destroying talk:', error);
371
- // Handle error
372
- }
373
- }
374
-
375
- onIceCandidate(event) {
376
- if (event.candidate) {
377
- const { candidate, sdpMid, sdpMLineIndex } = event.candidate;
378
-
379
- fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}/ice`, {
380
- method: 'POST',
381
- headers: {
382
- Authorization: `Basic ${this.didOptions.key}`,
383
- 'Content-Type': 'application/json',
384
- },
385
- body: JSON.stringify({
386
- candidate,
387
- sdpMid,
388
- sdpMLineIndex,
389
- session_id: this.sessionId,
390
- }),
391
- }).catch((error) => {
392
- console.error('Error sending ICE candidate:', error);
393
- // Handle error
394
- });
395
- }
396
- }
397
-
398
- onIceConnectionStateChange() {
399
- if (this.peerConnection.iceConnectionState === 'failed' || this.peerConnection.iceConnectionState === 'closed') {
400
- this.stopAllStreams();
401
- this.closePC();
402
- }
403
- }
404
-
405
- onTrack(event) {
406
- if (!event.track || !event.streams || event.streams.length === 0) return;
407
-
408
- this.statsIntervalId = setInterval(async () => {
409
- const stats = await this.peerConnection.getStats(event.track);
410
- stats.forEach((report) => {
411
- if (report.type === 'inbound-rtp' && report.mediaType === 'video') {
412
- const videoStatusChanged = this.videoIsPlaying !== report.bytesReceived > this.lastBytesReceived;
413
-
414
- if (videoStatusChanged) {
415
- this.videoIsPlaying = report.bytesReceived > this.lastBytesReceived;
416
- this.onVideoStatusChange(this.videoIsPlaying, event.streams[0]);
417
  }
418
- this.lastBytesReceived = report.bytesReceived;
 
 
419
  }
420
- });
421
- }, 100);
422
- }
423
-
424
- onVideoStatusChange(videoIsPlaying, stream) {
425
- let status;
426
- if (videoIsPlaying) {
427
- status = 'streaming';
428
- const remoteStream = stream;
429
- this.streams.push(remoteStream);
430
- this.checkSpeaching();
431
- } else {
432
- status = 'empty';
433
- this.speaching = false;
434
- this.processing = false;
435
- this.playIdleVideo();
436
- }
437
- }
438
-
439
- checkSpeaching() {
440
- if (this.speaching) {
441
- setTimeout(() => {this.checkSpeaching()}, 20)
442
- } else {
443
- this.setVideoElement(this.streams.shift());
444
- }
445
- }
446
-
447
- async createPeerConnection(offer, iceServers) {
448
- if (!this.peerConnection) {
449
- this.peerConnection = new RTCPeerConnection({ iceServers });
450
- this.peerConnection.addEventListener('icecandidate', this.onIceCandidate.bind(this), true);
451
- this.peerConnection.addEventListener('iceconnectionstatechange', this.onIceConnectionStateChange.bind(this), true);
452
- this.peerConnection.addEventListener('track', this.onTrack.bind(this), true);
453
- }
454
-
455
- await this.peerConnection.setRemoteDescription(offer);
456
- const sessionClientAnswer = await this.peerConnection.createAnswer();
457
- await this.peerConnection.setLocalDescription(sessionClientAnswer);
458
- return sessionClientAnswer;
459
- }
460
-
461
-
462
- setVideoElement(stream) {
463
- if (!stream) return;
464
- this.videoElement.srcObject = stream;
465
- this.videoElement.loop = false;
466
-
467
- // safari hotfix
468
- if (this.videoElement.paused) {
469
- this.videoElement
470
- .play()
471
- .then((_) => {})
472
- .catch((e) => {});
473
- }
474
- }
475
-
476
- playIdleVideo() {
477
- this.videoElement.srcObject = undefined;
478
- this.videoElement.src = this.didOptions.presenter.idle_movie;
479
- this.videoElement.loop = true;
480
- }
481
-
482
- playLoadingVideo() {
483
- this.videoElement.srcObject = undefined;
484
- this.videoElement.src = this.didOptions.presenter.loading_movie;
485
- this.videoElement.loop = false;
486
- }
487
-
488
- stopAllStreams() {
489
- if (this.videoElement.srcObject) {
490
- this.videoElement.srcObject.getTracks().forEach((track) => track.stop());
491
- this.videoElement.srcObject = null;
492
- }
493
- }
494
-
495
- closePC(pc = this.peerConnection) {
496
- if (!pc) return;
497
- pc.close();
498
- pc.removeEventListener('icecandidate', this.onIceCandidate.bind(this), true);
499
- pc.removeEventListener('iceconnectionstatechange', this.onIceConnectionStateChange.bind(this), true);
500
- pc.removeEventListener('track', this.onTrack.bind(this), true);
501
- clearInterval(this.statsIntervalId);
502
- if (pc === this.peerConnection) {
503
- this.peerConnection = null;
504
- }
505
- }
506
-
507
- async fetchWithRetries(url, options, retries = 1) {
508
- const maxRetryCount = 3;
509
- const maxDelaySec = 4;
510
- try {
511
- return await fetch(url, options);
512
- } catch (err) {
513
- if (retries <= maxRetryCount) {
514
- const delay = Math.min(Math.pow(2, retries) / 4 + Math.random(), maxDelaySec) * 1000;
515
- await new Promise((resolve) => setTimeout(resolve, delay));
516
- return this.fetchWithRetries(url, options, retries + 1);
517
- } else {
518
- throw new Error(`Max retries exceeded. error: ${err}`);
519
- }
520
- }
521
- }
522
- }
523
-
524
- const miiboAvatar = new MiiboAvatar({
525
- container: "my-video",
526
- option: {
527
- miibo: {
528
- api_key: "ed212f05-a0f2-4838-9bb3-26d318504a76190c9e5bc19f0",
529
- agent_id: "3e83a6b2-3c03-42b1-abc7-8a2dd97a8999190c7d1806120c",
530
- user_id: "user",
531
- },
532
- d_id: {
533
- key: "bWl5YXRha2VuOTk3QGdtYWlsLmNvbQ:9kIeSKJzFOc382oj4IHTl",
534
- service: "talks",
535
- priority: "cost",
536
- presenter:{
537
- gender: "male",
538
- image_url: "https://kenken999-fastapi-django-main-live.hf.space/static/sugi.jpg",
539
- idle_movie: "https://github.com/miibo-takumori/resorces/raw/main/portorate-idle.mp4",
540
- loading_movie: "https://github.com/miibo-takumori/resorces/raw/main/portorate-loading.mp4",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
541
  }
542
  }
543
  }
544
- })
545
-
546
- miiboAvatar.autoRecognize();
547
- </script>
548
- <canvas id="my-live2d"></canvas>
549
- <script src="live2d.js"></script>
550
- <script src="index.js"></script>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
551
  <div id="vtubers">
552
  <!--
553
  <iframe width="auto" height="315" src="https://www.youtube.com/embed/9BSgttQDyOw?si=ax2F4oAi2h9f9eGT" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
 
9
  <script src="./aivtuber.js"></script>
10
  <title>Live2Dサンプル</title>
11
  <!-- Live2D -->
12
+ <script src="./live2dcubismcore.js"></script>
13
+ <script src="//cdn.jsdelivr.net/gh/dylanNew/live2d/webgl/Live2D/lib/live2d.min.js"></script>
14
+ <!-- PixiJS -->
15
+ <script src="//cdnjs.cloudflare.com/ajax/libs/pixi.js/5.1.3/pixi.min.js"></script>
16
+ <script src="//cdn.jsdelivr.net/npm/pixi-live2d-display/dist/index.min.js"></script>
17
+ <!-- Kalidokit -->
18
+ <script src="//cdn.jsdelivr.net/npm/[email protected]/dist/kalidokit.umd.js"></script>
19
  </head>
20
 
21
  <body>
22
 
23
+ <video id="my-video" autoplay></video>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
+ <script>
26
+
27
+ 'use strict';
28
+ class MiiboAvatar {
29
+ constructor(config) {
30
+ this.container = config.container;
31
+ this.speechToTextOptions = config.option.speech_to_text;
32
+ this.miiboOptions = config.option.miibo;
33
+ this.didOptions = config.option.d_id;
34
+ this.initialize();
35
+ }
36
+
37
+ initialize() {
38
+ //alert("38");
39
+ const RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConnection || window.mozRTCPeerConnection;
40
+
41
+ this.videoElement = document.getElementById(this.container);
42
+ this.videoElement.setAttribute('playsinline', '');
43
+ this.playIdleVideo();
44
+
45
+ this.createNewStream();
46
+ this.rec = new webkitSpeechRecognition()
47
+
48
+ this.speaching = false;
49
+ this.processing = false;
50
+ this.streams = [];
51
+ }
52
+
53
+ async createNewStream() {
54
+ //alert("53")
55
+ try {
56
+ this.stopAllStreams();
57
+ this.closePC();
58
+
59
+ let presenter = { "source_url": this.didOptions.presenter.image_url }
60
+ const sessionResponse = await this.fetchWithRetries(`https://api.d-id.com/${this.didOptions.service}/streams`, {
61
+ method: 'POST',
62
+ headers: {
63
+ Authorization: `Basic ${this.didOptions.key}`,
64
+ 'Content-Type': 'application/json',
65
+ },
66
+ body: JSON.stringify(presenter),
67
+ });
68
+
69
+ const { id: newStreamId, offer, ice_servers: iceServers, session_id: newSessionId } = await sessionResponse.json();
70
+ this.streamId = newStreamId;
71
+ this.sessionId = newSessionId;
72
+
73
+ try {
74
+ this.sessionClientAnswer = await this.createPeerConnection(offer, iceServers);
75
+ } catch (e) {
76
+ console.log('Error creating peer connection:', e);
77
+ this.stopAllStreams();
78
+ this.closePC();
79
+ return;
80
+ }
81
+
82
+ const sdpResponse = await fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}/sdp`, {
83
+ method: 'POST',
84
+ headers: {
85
+ Authorization: `Basic ${this.didOptions.key}`,
86
+ 'Content-Type': 'application/json',
87
+ },
88
+ body: JSON.stringify({
89
+ answer: this.sessionClientAnswer,
90
+ session_id: this.sessionId,
91
+ }),
92
+ });
93
+
94
+ // Handle sdpResponse if needed
95
+ } catch (error) {
96
+ console.log('Error creating new stream:', error);
97
+ // Handle error
98
  }
99
  }
100
+
101
+ speechRecogInit() {
102
+ this.audioContext = new (window.AudioContext || window.webkitAudioContext)()
103
+
104
+ if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
105
+ navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
106
+ var input = this.audioContext.createMediaStreamSource(stream)
107
+ this.audioContext.resume()
108
+ this.recorder = new Recorder(input)
109
+ })
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
  }
 
 
111
  }
112
 
113
+
114
+ startRecording() {
115
+ this.recorder && this.recorder.record();
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  }
 
117
 
118
+ stopRecording() {
119
+ this.playLoadingVideo();
120
+ this.recorder && this.recorder.stop();
121
+ this.audioRecognize();
122
+ this.recorder.clear();
123
+ }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
+ audioRecognize() {
126
+ this.recorder && this.recorder.exportWAV((blob) => {
127
+ let reader = new FileReader()
128
+ reader.onload = () => {
129
+ let result = new Uint8Array(reader.result)
130
+
131
+ let data = {
132
+ "config": {
133
+ "encoding": "LINEAR16",
134
+ "languageCode": "ja-JP",
135
+ "alternativeLanguageCodes": ["en-US"],//,"cmn-CN","ko-KR"],
136
+ "audio_channel_count": 2
137
+ },
138
+ "audio": {
139
+ "content": this.arrayBufferToBase64(result)
140
+ }
141
+ }
142
+ fetch('https://speech.googleapis.com/v1/speech:recognize?key=' + this.speechToTextOptions.api_key, {
143
+ method: 'POST',
144
+ headers: {
145
+ 'Content-Type': 'application/json; charset=utf-8'
146
+ },
147
+ body: JSON.stringify(data)
148
+ }).then((response) => {
149
+ return response.text()
150
+ }).then((text) => {
151
+ let result_json = JSON.parse(text)
152
+ text = result_json.results[0].alternatives[0].transcript;
153
+ this.languageCode = result_json.results[0].languageCode;
154
+ this.ask(text)
155
+ })
156
+ }
157
+ reader.readAsArrayBuffer(blob)
158
+ })
159
+ }
160
+ // Chrome Only
161
+ autoRecognizeMessage(message) {
162
+ handleLiveComment("test", message)
163
+ //alert("start word test")
164
+ this.rec.continuous = false
165
+ this.rec.interimResults = false
166
+ this.rec.lang = 'ja-JP'
167
+ this.processing = true
168
+ this.playLoadingVideo();
169
+
170
+ this.rec.stop()
171
+ alert("startssssssssssssssssssssssssss")
172
+ const { transcript } = message
173
+ this.ask(transcript);
174
+ //for (var i = e.resultIndex; i < e.results.length; i++) {
175
+ // if (!e.results[i].isFinal) continue
176
+ //
177
+ // const { transcript } = e.results[i][0]
178
+ // this.ask(transcript);
179
+ //}
180
+ //this.rec.onresult = (e) => {
181
+
182
+ //}
183
+
184
+ this.rec.onend = () => { this.autoRecognizeRestart() }
185
+ this.rec.start()
186
+ }
187
+ // Chrome Only
188
+ autoRecognize() {
189
+ this.rec.continuous = false
190
+ this.rec.interimResults = false
191
+ this.rec.lang = 'ja-JP'
192
 
193
+ this.rec.onresult = (e) => {
194
+ this.processing = true
195
+ this.playLoadingVideo();
196
+
197
+ this.rec.stop()
198
+ //alert("start")
199
+
200
+ for (var i = e.resultIndex; i < e.results.length; i++) {
201
+ if (!e.results[i].isFinal) continue
202
+
203
+ const { transcript } = e.results[i][0]
204
+ this.ask(transcript);
205
+ }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
206
  }
207
+
208
+ this.rec.onend = () => { this.autoRecognizeRestart() }
209
+ this.rec.start()
210
  }
211
+
212
+ autoRecognizeRestart() {
213
+ if (this.processing) {
214
+ setTimeout(() => { this.autoRecognizeRestart() }, 1000)
215
+ } else {
216
+ this.rec.start()
217
+ }
218
+ }
219
+
220
+ arrayBufferToBase64(buffer) {
221
+ let binary = ''
222
+ let bytes = new Uint8Array(buffer);
223
+ let len = bytes.byteLength
224
+ for (let i = 0; i < len; i++) {
225
+ binary += String.fromCharCode(bytes[i])
226
+ }
227
+ return window.btoa(binary)
228
+ }
229
+
230
+ ask(message) {
231
+ //youtubeからの質問
232
+ alert("203 ask" + message);
233
+ //ask(start);
234
+ this.getMiiboResponse(message);
235
+ }
236
+
237
+ async getMiiboResponse(utterance) {
238
+ alert("209 getMiiboResponse" + utterance)
239
+ const params = {
240
+ api_key: this.miiboOptions.api_key,
241
+ agent_id: this.miiboOptions.agent_id,
242
+ uid: this.miiboOptions.user_id,
243
+ stream: true,
244
+ utterance: utterance
245
+ };
246
+
247
+ try {
248
+ const res = await fetch("https://api-mebo.dev/api", {
249
+ method: "POST",
250
+ headers: { "Content-Type": "application/json" },
251
+ body: JSON.stringify(params),
252
+ });
253
+
254
+ const reader = res.body.getReader();
255
+ const decoder = new TextDecoder();
256
+ let output = "";
257
+ let sentences = [];
258
+ let current_index = 0;
259
+
260
+ const read = async () => {
261
+ const { done, value } = await reader.read();
262
+ if (done) return;
263
+
264
+ let dataString = decoder.decode(value).split("\n").filter(x => x != "");
265
+
266
+ try {
267
+ let responseData = JSON.parse(dataString[dataString.length - 1]);
268
+
269
+ output = responseData.bestResponse.utterance.split("\n").filter(x => x.trim() != "").join("\n");
270
+ sentences = output.replace(/[。、\.\,\!\?!?]/, ".").split(".")
271
+ if (this.didOptions.priority == "speed" && current_index == 0 && current_index + 1 < sentences.length) {
272
+ this.startTalk(sentences[current_index++])
273
+ }
274
+ } catch (e) {
275
+ console.log(e);
276
+ }
277
+
278
+ return read();
279
+ };
280
+
281
+ await read();
282
+ reader.releaseLock();
283
+
284
+ this.startTalk(sentences.slice(current_index).join("。"));
285
+ } catch (error) {
286
+ console.log("Error fetching AI response: ", error);
287
+ }
288
+ }
289
+
290
+ async startTalk(input) {
291
+ if (this.peerConnection?.signalingState === 'stable' || this.peerConnection?.iceConnectionState === 'connected') {
292
+
293
+ const gender = this.didOptions.presenter.gender;
294
+ let voice_id = this.didOptions.presenter.voice_id || "";
295
+
296
+ if (voice_id == "") {
297
+ switch (this.languageCode) {
298
+ case "en-us":
299
+ voice_id = gender == "male" ? "en-US-GuyNeural" : "en-US-AriaNeural"
300
+ break;
301
+ case "ko-kr":
302
+ voice_id = gender == "male" ? "ko-KR-InJoonNeural" : "ko-KR-YuJinNeural"
303
+ break;
304
+ case "cmn-CN":
305
+ voice_id = gender == "male" ? "zh-CN-YunjianNeural" : "zh-CN-XiaohanNeural"
306
+ break;
307
+ default:
308
+ voice_id = gender == "male" ? "ja-JP-KeitaNeural" : "ja-JP-NanamiNeural"
309
+ }
310
+ }
311
+
312
+ const requestOptions = {
313
+ method: 'POST',
314
+ headers: {
315
+ Authorization: `Basic ${this.didOptions.key}`,
316
+ 'Content-Type': 'application/json',
317
+ },
318
+ body: JSON.stringify({
319
+ script: {
320
+ type: "text",
321
+ subtitles: false,
322
+ provider: {
323
+ type: "microsoft",
324
+ voice_id: voice_id
325
+ },
326
+ ssml: false,
327
+ input: input
328
+ },
329
+ config: {
330
+ fluent: false,
331
+ pad_audio: 0,
332
+ align_driver: false,
333
+ stitch: false,
334
+ auto_match: false,
335
+ sharpen: false,
336
+ normalization_factor: 0
337
+ },
338
+ session_id: this.sessionId,
339
+ }),
340
+ };
341
+
342
+ if (this.didOptions.service === 'clips') {
343
+ requestOptions.body.background = { color: '#FFFFFF' };
344
+ }
345
+
346
+ try {
347
+ const playResponse = await this.fetchWithRetries(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}`, requestOptions);
348
+ // Handle response if needed
349
+ } catch (error) {
350
+ console.error('Error starting talk:', error);
351
+ // Handle error
352
+ }
353
+ }
354
+ }
355
+
356
+ async destoryTalk(input) {
357
+ try {
358
+ await fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}`, {
359
+ method: 'DELETE',
360
+ headers: {
361
+ Authorization: `Basic ${this.didOptions.key}`,
362
+ 'Content-Type': 'application/json',
363
+ },
364
+ body: JSON.stringify({ session_id: this.sessionId }),
365
+ });
366
+
367
+ await this.stopAllStreams();
368
+ await this.closePC();
369
+ } catch (error) {
370
+ console.error('Error destroying talk:', error);
371
+ // Handle error
372
+ }
373
+ }
374
+
375
+ onIceCandidate(event) {
376
+ if (event.candidate) {
377
+ const { candidate, sdpMid, sdpMLineIndex } = event.candidate;
378
+
379
+ fetch(`https://api.d-id.com/${this.didOptions.service}/streams/${this.streamId}/ice`, {
380
+ method: 'POST',
381
+ headers: {
382
+ Authorization: `Basic ${this.didOptions.key}`,
383
+ 'Content-Type': 'application/json',
384
+ },
385
+ body: JSON.stringify({
386
+ candidate,
387
+ sdpMid,
388
+ sdpMLineIndex,
389
+ session_id: this.sessionId,
390
+ }),
391
+ }).catch((error) => {
392
+ console.error('Error sending ICE candidate:', error);
393
+ // Handle error
394
+ });
395
+ }
396
+ }
397
+
398
+ onIceConnectionStateChange() {
399
+ if (this.peerConnection.iceConnectionState === 'failed' || this.peerConnection.iceConnectionState === 'closed') {
400
+ this.stopAllStreams();
401
+ this.closePC();
402
+ }
403
+ }
404
+
405
+ onTrack(event) {
406
+ if (!event.track || !event.streams || event.streams.length === 0) return;
407
+
408
+ this.statsIntervalId = setInterval(async () => {
409
+ const stats = await this.peerConnection.getStats(event.track);
410
+ stats.forEach((report) => {
411
+ if (report.type === 'inbound-rtp' && report.mediaType === 'video') {
412
+ const videoStatusChanged = this.videoIsPlaying !== report.bytesReceived > this.lastBytesReceived;
413
+
414
+ if (videoStatusChanged) {
415
+ this.videoIsPlaying = report.bytesReceived > this.lastBytesReceived;
416
+ this.onVideoStatusChange(this.videoIsPlaying, event.streams[0]);
417
+ }
418
+ this.lastBytesReceived = report.bytesReceived;
419
+ }
420
+ });
421
+ }, 100);
422
+ }
423
+
424
+ onVideoStatusChange(videoIsPlaying, stream) {
425
+ let status;
426
+ if (videoIsPlaying) {
427
+ status = 'streaming';
428
+ const remoteStream = stream;
429
+ this.streams.push(remoteStream);
430
+ this.checkSpeaching();
431
+ } else {
432
+ status = 'empty';
433
+ this.speaching = false;
434
+ this.processing = false;
435
+ this.playIdleVideo();
436
+ }
437
+ }
438
+
439
+ checkSpeaching() {
440
+ if (this.speaching) {
441
+ setTimeout(() => { this.checkSpeaching() }, 20)
442
+ } else {
443
+ this.setVideoElement(this.streams.shift());
444
+ }
445
+ }
446
+
447
+ async createPeerConnection(offer, iceServers) {
448
+ if (!this.peerConnection) {
449
+ this.peerConnection = new RTCPeerConnection({ iceServers });
450
+ this.peerConnection.addEventListener('icecandidate', this.onIceCandidate.bind(this), true);
451
+ this.peerConnection.addEventListener('iceconnectionstatechange', this.onIceConnectionStateChange.bind(this), true);
452
+ this.peerConnection.addEventListener('track', this.onTrack.bind(this), true);
453
+ }
454
+
455
+ await this.peerConnection.setRemoteDescription(offer);
456
+ const sessionClientAnswer = await this.peerConnection.createAnswer();
457
+ await this.peerConnection.setLocalDescription(sessionClientAnswer);
458
+ return sessionClientAnswer;
459
+ }
460
+
461
+
462
+ setVideoElement(stream) {
463
+ if (!stream) return;
464
+ this.videoElement.srcObject = stream;
465
+ this.videoElement.loop = false;
466
+
467
+ // safari hotfix
468
+ if (this.videoElement.paused) {
469
+ this.videoElement
470
+ .play()
471
+ .then((_) => { })
472
+ .catch((e) => { });
473
+ }
474
+ }
475
+
476
+ playIdleVideo() {
477
+ this.videoElement.srcObject = undefined;
478
+ this.videoElement.src = this.didOptions.presenter.idle_movie;
479
+ this.videoElement.loop = true;
480
+ }
481
+
482
+ playLoadingVideo() {
483
+ this.videoElement.srcObject = undefined;
484
+ this.videoElement.src = this.didOptions.presenter.loading_movie;
485
+ this.videoElement.loop = false;
486
+ }
487
+
488
+ stopAllStreams() {
489
+ if (this.videoElement.srcObject) {
490
+ this.videoElement.srcObject.getTracks().forEach((track) => track.stop());
491
+ this.videoElement.srcObject = null;
492
+ }
493
+ }
494
+
495
+ closePC(pc = this.peerConnection) {
496
+ if (!pc) return;
497
+ pc.close();
498
+ pc.removeEventListener('icecandidate', this.onIceCandidate.bind(this), true);
499
+ pc.removeEventListener('iceconnectionstatechange', this.onIceConnectionStateChange.bind(this), true);
500
+ pc.removeEventListener('track', this.onTrack.bind(this), true);
501
+ clearInterval(this.statsIntervalId);
502
+ if (pc === this.peerConnection) {
503
+ this.peerConnection = null;
504
+ }
505
+ }
506
+
507
+ async fetchWithRetries(url, options, retries = 1) {
508
+ const maxRetryCount = 3;
509
+ const maxDelaySec = 4;
510
+ try {
511
+ return await fetch(url, options);
512
+ } catch (err) {
513
+ if (retries <= maxRetryCount) {
514
+ const delay = Math.min(Math.pow(2, retries) / 4 + Math.random(), maxDelaySec) * 1000;
515
+ await new Promise((resolve) => setTimeout(resolve, delay));
516
+ return this.fetchWithRetries(url, options, retries + 1);
517
+ } else {
518
+ throw new Error(`Max retries exceeded. error: ${err}`);
519
+ }
520
  }
521
  }
522
  }
523
+
524
+ const miiboAvatar = new MiiboAvatar({
525
+ container: "my-video",
526
+ option: {
527
+ miibo: {
528
+ api_key: "ed212f05-a0f2-4838-9bb3-26d318504a76190c9e5bc19f0",
529
+ agent_id: "3e83a6b2-3c03-42b1-abc7-8a2dd97a8999190c7d1806120c",
530
+ user_id: "user",
531
+ },
532
+ d_id: {
533
+ key: "bWl5YXRha2VuOTk3QGdtYWlsLmNvbQ:9kIeSKJzFOc382oj4IHTl",
534
+ service: "talks",
535
+ priority: "cost",
536
+ presenter: {
537
+ gender: "male",
538
+ image_url: "https://kenken999-fastapi-django-main-live.hf.space/static/sugi.jpg",
539
+ idle_movie: "https://github.com/miibo-takumori/resorces/raw/main/portorate-idle.mp4",
540
+ loading_movie: "https://github.com/miibo-takumori/resorces/raw/main/portorate-loading.mp4",
541
+ }
542
+ }
543
+ }
544
+ })
545
+
546
+ miiboAvatar.autoRecognize();
547
+ </script>
548
+ <canvas id="my-live2d"></canvas>
549
+ <script src="live2d.js"></script>
550
+ <script src="index.js"></script>
551
  <div id="vtubers">
552
  <!--
553
  <iframe width="auto" height="315" src="https://www.youtube.com/embed/9BSgttQDyOw?si=ax2F4oAi2h9f9eGT" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
workspace/sample_html.html ADDED
@@ -0,0 +1 @@
 
 
1
+ <!DOCTYPE html><html><head><title>Sample HTML Page</title></head><body><h1>Welcome to my sample HTML page</h1></body></html>