text:
Article | Open | Published:

# A PDM-based bi-directional fibre-FSO integration with two RSOAs scheme

## Abstract

A polarization-division-multiplexing (PDM)-based bi-directional fibre-free-space optical (FSO) integration with two reflective semiconductor optical amplifiers (RSOAs) scheme to efficiently wipe off the modulated data for upstream modulation is proposed and successfully demonstrated. For downstream modulation, a high-speed 128 Gb/s vestigial sideband (VSB)-four-level pulse amplitude modulation (PAM4) fibre-FSO integration is feasibly established. The transmission capacity is increased up to four times through PDM operation and VSB-PAM4 modulation. For uplink transmission, a 10 Gb/s non-return-to-zero fibre-FSO integration with two RSOAs scheme to effectually erase the downstream modulated data is practically constructed. The upstream performance exhibits noticeable enhancement by using of two RSOAs scheme to wipe off the modulated data clearly. Such illustrated PDM-based bi-directional 128 Gb/s (downstream)/10 Gb/s (upstream) fibre-FSO integration is shown to be prominent not only due to its enhancement in the convergence of fibre backhaul and optical wireless reach extender but also because of its benefit in bi-directional transmission for affording high transmission capacity with long-reach optical wireless link and improved upstream performance.

## Introduction

Broadband access network is considered as an attractive frontier for providing present and emerging technologies, such as fifth-generation mobile communications, immersive virtual reality and augmented reality, 4K video streaming, and high-speed Internet1,2,3,4,5. The increasing demands raise the requirement for high transmission capacity, not only for the fibre backhaul but also for the last-mile in-building network. It can be achieved by taking advantages of both optical fibre and optical wireless technologies via the implementation of fibre-free-space optical (FSO) integration6,7,8. In comparison with other systems, fibre-FSO integration can provide not only high transmission capacity but also sufficient flexibility. In this study, a polarization-division-multiplexing (PDM)-based bi-directional fibre-FSO integration with two reflective semiconductor optical amplifiers (RSOAs) scheme to effectually erase the modulated data for uplink transmission is proposed and feasibly demonstrated. For downlink transmission, a high-speed 128 Gb/s vestigial sideband (VSB)-four-level pulse amplitude modulation (PAM4) injection-locked vertical-cavity surface-emitting laser (VCSEL)-based fibre-FSO integration is practically constructed. For an actual realization of fibre-FSO integration, a high-speed operation with improved transmission performance is the critical concern of system operators. The 1.55-μm VCSEL with a single transverse mode has been developed to a few tens of GHz due to its narrow coherent laser beam. Injection locking technique is recognized as one of the most effective techniques for enhancing the modulation bandwidth of VCSEL9,10. It integrates the optical properties and advantages of VCSEL and thus opens an access for high-speed operation. As for PAM4 modulation, PAM4 reduces the bandwidth requirement compared with non-return-to-zero (NRZ). Consequently, PAM4 modulation is an efficient approach to meet the high capacity demand. Nevertheless, PAM4 signal possesses a broad linewidth and brings on large fibre dispersion in fibre-FSO integration11,12,13,14. By contrast, VSB-PAM4 signal possesses a narrow linewidth and brings on small fibre dispersion in fibre-FSO integration15,16. The VSB-PAM4 modulation is therefore adopted to mitigate the fibre dispersion impairment, which helps to improve the transmission performance of fibre-FSO integration. Moreover, the polarisation domain has been investigated substantially for fibre-FSO integration. The use of PDM operation is a prominent one to double the channel capacity17,18,19,20. To inherit the advantages of above-mentioned VSB-PAM4 modulation and PDM operation, the channel capacity of the VSB-PAM4 fibre-FSO integration can be enhanced up to four times.

For uplink transmission, a 10 Gb/s NRZ fibre-FSO integration with two RSOAs scheme for efficiently wiping off the downstream modulated data is practically established. For a real deploying of bi-directional fibre-FSO integration, the receiver side requires an optical carrier for uplink transmission. Wavelength reuse is mostly adopted in bi-directional lightwave transmission systems for upstream. The feasibilities of employing RSOA for wavelength reuse in bi-directional fibre-FSO integration were presented previously21,22,23. However, the upstream performance can be further improved by adopting two RSOAs scheme to effectively erase the modulated data24,25. RSOA1 is adopted at the receiver side to erase the downstream data and remodulate the upstream data, whereas RSOA2 is adopted at the transmitter side to erase the modulated data (residual downstream and remodulated upstream data) and reproduce a pure optical carrier to seed into the RSOA1. Two RSOAs scheme enhances the RSOA characteristics and brings on the further erasure of the modulated data in upstream transmission. Given that both downstream and upstream data signals are delivered by the same single-mode fibre (SMF), the Rayleigh backscattering noise limits the upstream transmission performance. Since the further erasure of modulated data, yet the Rayleigh backscattering noise caused by the remodulation reaches the minimum value. Consequently, the upstream performance presents evident improvement. Noticeably, two RSOAs scheme is a promising solution for bi-directional fibre-FSO integration because it can almost erase the modulated data.

A previous study demonstrated a two-way phase-modulated fibre-FSO convergence system with VCSEL-based tunable optical band-pass filter6. Another work demonstrated the feasibility of building a bidirectional fibre-wireless and fibre-FSO transmission system with optoelectronic oscillator-based broadband light source and RSOA22. Moreover, establishing a bidirectional fibre-wireless and fibre-FSO integrated system with polarization-orthogonal modulation scheme was shown feasibly26. Furthermore, a bidirectional fibre-FSO and fibre-wireless convergence system with two orthogonally polarized optical sidebands had been illustrated7. However, the downstream data rate/FSO link of 20 Gbps/100 m6, 30 Gbps/10 m22, 30 Gbps/50 m26, and 20 Gbps/100 m7 are significantly less than the corresponding values of 128-Gb/s/200-m adopted in this proposed PDM-based bi-directional fibre-FSO integration. In addition, a real-time PAM4 fibre-FSO and fibre-wireless hybrid system with parallel/orthogonally polarized dual-wavelength scheme was presented27. Yet, it is difficult for a practical implementation in real networks due to uni-directional transmission. Conclusively, it is necessary to develop a bi-directional fibre-FSO integration with high transmission capacity, sufficient flexibility, and improved upstream performance. A PDM-based bi-directional fibre-FSO integration with two RSOAs scheme for affording high channel capacity (128-Gb/s (downstream)/10-Gb/s (upstream)) with long-reach free-space transmission (200 m) and enhanced upstream performance is thereby demonstrated to meet the targets. For downlink modulation, the channel capacity is enhanced to four times through the PDM operation and VSB-PAM4 modulation. For uplink modulation, the upstream performance shows considerable improvement by using the two RSOAs scheme to erase the modulated data clearly. Bit error rate (BER) (downstream/upstream) and VSB-PAM4 (downstream)/NRZ (upstream) eye diagrams perform well over 25 km SMF transport with 200 m free-space transmission. For in-building network application, a 200-m free-space transmission can greatly satisfy the requirement for in-building transmission distance. Accordingly, this demonstrated PDM-based bi-directional fibre-FSO integration is designed as a promising framework not only due to its enhancement in the integration of fibre backhaul and in-building network but also because of its advantage in two-way communication for providing high transmission capacity with long-range optical wireless link and enhanced upstream performance.

## Results

### Optical power-driving current (P–I) curve and the modulation response of the 1.55-μm VCSEL

Figure 1(a) shows the optical power-driving current (P–I) curve of the 1.55 μm VCSEL with a threshold current of 2.3 mA and a slope of around 26 mW/mA (1.6/(8.4-2.3)). With a driving current of 8.4 mA, a maximum optical output power of 1.6 mW is obtained. Since that the P–I curve is very close to the linear distribution for direct signal modulation, a linear driver with high-linearity is not required to boost the PAM4 electrical signal. Figure 1(b) shows the modulation responses of the free-running VCSEL under different driving currents. It is to be observed that the resonance peak decreases with the increase in driving current. Given that the driving current is 8.4 mA, the 3-dB modulation bandwidth reaches 10.3 GHz. This finding reveals that this 1.55-μm VCSEL is powerful for an effective usage in high-speed fibre-FSO integration. Thus, the driving current of free-running VCSEL is driven at 8.4 mA to acquire a maximum optical output power and a high 3-dB modulation bandwidth. Moreover, the modulation response of the VCSEL with injection locking is presented in Fig. 1(b) as well. Obviously, an injection-locked VCSEL attains a substantial enhancement in 3-dB modulation bandwidth. Under the state of injection locking, a 3-dB modulation bandwidth of 24.5 GHz is achieved. Injection locking technique substantially increases the photon density, by which leading to a considerable 3-dB bandwidth enhancement28. The frequency response is featured by a drop in the middle frequencies and a resonance peak in the high frequencies to achieve a considerable 3-dB bandwidth enhancement. Given that the 24.5 GHz is located at the resonance peak, the drop in the middle frequencies will not affect the transmission performances. Since the transmission rate is $$\sqrt{2}$$ times the bandwidth29,30, yet a 128 Gb/s fibre-FSO integration [24.5 ×$$\sqrt{2}$$× 2 (VSB-PAM4 modulation) × 2 (PDM operation) > 128] can be satisfactorily implemented.

### BER performances and eye diagrams of the PDM-based 128 Gb/s VSB-PAM4 fibre-FSO integration under different scenarios

Figure 2(a) presents the BER performances of the PDM-based 128 Gb/s VSB-PAM4/PAM4 fibre-FSO integration for the scenarios of back-to-back (BTB), and over 25-km SMF transport with 200-m free-space link (x- and y- polarisations). By adjusting the attenuation of variable optical attenuator (VOA) at the transmitter side, different received optical powers can be acquired. The figure shows the BER performances of x-polarisation and y-polarisation are nearly the same. Therefore, the BER performance’s impact in x- and y- polarisations is nearly identical. At a BER value of 10−9, a power penalty of 5.3 dB exists between the scenarios of BTB and over 25-km SMF transport with 200-m free-space link (x- or y-polarisation). Such 5.3 dB power penalty results from the use of optical VSB filter to reduce the fibre dispersion by suppressing the linewidth of optical signal. As for signal-to-signal beating interference (SSBI) and cross-beating, no SSBI and cross-beating exist. For two optical sidebands with parallel polarizations, SSBI emerges and degrades the performance of the PDM-based fiber-FSO integration due to the natural feature of two sidebands with parallel polarizations. Given that each polarisation only has one sideband, the SSBI will not emerge. Regarding cross-beating, considering the orthogonality of x-polarisation and y-polarisation, the cross-beating will be trivially small27. Thus, the BER performance will not be degraded by SSBI and cross-beating terms. Analytic result reveals that this proposed PDM-based VSB-PAM4 fibre-FSO integration meets the high accessibility target of fibre backhaul with optical wireless reach extender. Moreover, with PAM4 modulation (x-polarisation), the BER value noticeably degrades to 10−6. A BER value of 10−6 is obtained as a result of the fibre dispersion induced by 25 km SMF transport. Over 25-km SMF transport, distortions caused by fibre dispersion worsens the BER performance because of the natural feature of optical PAM4 signal with broadened linewidth.

$$r=2.3\times \frac{1}{SFC\times 2\pi }=3.6\,({\rm{\mu }}{\rm{m}})$$
(1)

Moreover, the laser beam divergent angle (θ) can be calculated as:

$${\theta }=\frac{3.6(\mu m)}{50(mm)}=72\times {10}^{-6}$$
(2)

Over a L-m FSO link, the laser beam diameter (dL) should be smaller than that of doublet lens 2 (dL < 30 mm):

$$\begin{array}{lllll}{{d}}_{{L}} & = & \sqrt{{{d}}^{2}+{(2{\theta }L)}^{2}} & = & \sqrt{{2.2}^{2}+{(0.144{L})}^{2}} < 30\end{array}$$
(3)

### BER values and eye diagrams of 10 Gb/s NRZ data signal for uplink transmission by varying the RSOA1 seeding power level

With the adoption of equaliser at the receiver side, the BER values of 10 Gb/s NRZ data signal for uplink transmission by varying the RSOA1 seeding power level from -24 dBm to −12 dBm are shown in Fig. 3(a). To adjust the RSOA1 seeding power level into −21, −18 and −15 dBm, at a BER value of 109, the corresponding received optical power levels are −2.8, −12.1 and −15.4 dBm, respectively. It is to be observed that upstream BER performance improves with the increase in RSOA1 seeding power level. The higher the optical power seeding into the RSOA1, the more suppression the downstream data signal in the upstream transmission attains, thereby improving the upstream BER performance. Whereas if RSOA1 seeding power level is too high, then the RSOA1 will be operated in the gain saturation region. With a seeding power level of −12 dBm, the BER performance can be found to degrade to 10−7. A high BER value of 10−7 is acquired because of gain saturation-induced nonlinear distortion. In addition, at a BER value of 10−9, power penalty improvements of 5.7 dB (RSOA1 with a seeding power level of −21 dBm) and 4.3 dB (RSOA1 with a seeding power level of −15 dBm) exist as two RSOAs scheme is deployed. Two RSOAs scheme brings on further suppression the modulated data in upstream transmission. Consequently, the upstream BER performance is further improved by deploying two RSOAs scheme to wipe off the modulated data effectually.

For uplink transmission, the eye diagrams of 10 Gb/s NRZ data signal corresponding to the RSOA1 seeding power level of −24, −15 and −15 (two RSOAs scheme) dBm, respectively, with the adoption of equaliser at the receiver side are displayed in Fig. 3(b–d). For the state of RSOA1 with a seeding power level of −24 dBm, amplitude and phase fluctuations in eye diagram are obviously observed in Fig. 3(b). Initially, the downstream data signal should be suppressed by RSOA1. Nevertheless, the downstream data signal is ineffectively suppressed because of low optical power seeding the RSOA1, leading to considerable fluctuations in eye diagram. In Fig. 3(c), for the state of RSOA1 with a seeding power level of −15 dBm, amplitude and phase fluctuations are reduced to some extent due to the suppression of the downstream data signal. However, in Fig. 3(d), for the state of RSOA1 with a seeding power level of −15 dBm (two RSOAs scheme), amplitude and phase fluctuations are remarkably reduced due to the effective suppression of the downstream data signal by using two RSOAs scheme.

To clarify the enhancement achieved using the two RSOAs scheme, a comparison with one RSOA scheme is presented. The eye diagrams of 10 Gb/s NRZ data signal measured at the outputs of RSOA1 and RSOA2 for the state of RSOA1 with a seeding power level of −15 dBm are shown in Fig. 4(a,b). At the output of RSOA1 (Fig. 4(a)), as shown in the eye diagram, large amplitude and phase fluctuations are observed due to the insufficient modulation bandwidth of RSOA1. Since the modulation bandwidth of RSOA1 is only 3.4 GHz, yet an upstream 10 Gb/s fibre-FSO integration [3.4 ×$$\sqrt{2}$$ < 10] cannot be feasibly realized without the adoption of equaliser at the receiver side. Whereas at the output of RSOA2 (Fig. 4(b)), experimental result shows that the data signal is almost erased, indicating that further erasure of the modulated data in upstream transmission is attained at a data rate of 10 Gb/s.

## Discussion

Directly modulated 1.55-μm VCSEL is attractive for the implementation of fibre-FSO integration. However, the transmission performance of the directly modulated VCSEL transmitter-based fibre-FSO integration will be degraded by fibre dispersion-induced distortions due to a long-haul fibre transmission. Therefore, adopting a dispersion compensation device (DCD) within the fibre transmission is necessary to compensate for the fibre chromatic dispersion and reduce the fibre dispersion-induced distortions accordingly32,33. Nevertheless, DCD within the fibre transmission will impose the framework complexity on fibre-FSO integration. Developing fibre-FSO integration with simple framework and qualified transmission performance to ensure a successful deployment is required. Optical VSB filter, which can eliminate the redundant linewidth of optical signal, is therefore adopted to effectively reduce the fibre dispersion induced by a 25-km SMF transport. Such proposed optical VSB filter is attractive because it avoids the need of sophisticated DCD within the fibre transmission. The optical spectra of the 64 Gb/s optical PAM4 and VSB-PAM4 signals are displayed in Fig. 5(a). The optical PAM4 signal possesses a broad linewidth, whereas the optical VSB-PAM4 signal possesses a narrow linewidth. The broadened optical spectrum through the interaction of fibre dispersion with the chirp of the directly modulated VCSEL transmitter is compressed by optical VSB filter to enhance the spectral efficiency. Optical VSB filter not only presents high spectral efficiency but also reduces the fibre dispersion by suppressing the linewidth of optical signal. Over 25-km SMF transport, distortions induced by fibre dispersion degrades the transmission performance of fibre-FSO integration due to broadened linewidth of optical PAM4 signal. However, fibre dispersion due to 25 km SMF transport can be mitigated by adopting optical VSB-PAM4 modulation.

To establish a direct connection with the central wavelength of the optical VSB filter and the BER performance, the measured BER values of the 128 Gb/s fibre-FSO integration over 25-km SMF transport with 200-m free-space link (x-polarisation) with different central wavelengths of optical VSB filter are displayed in Fig. 5(b). When the central wavelength of the optical VSB filter moves towards 1545.39 or 1545.73 nm, the fibre-FSO integration has the best transmission performance in terms of the lowest BER value and the clearest eye diagrams. As the central wavelength moves towards 1545.39 nm, the optical spectrum of the optical PAM4 signal on the right is partially reduced to form an optical VSB-PAM4 signal. As the central wavelength moves towards 1545.73 nm, the optical spectrum of the optical PAM4 signal on the left is partially reduced to form an optical VSB-PAM4 signal as well. Whereas as the central wavelength moves towards 1545.56 nm, the optical spectrum of the optical PAM4 signal will not be reduced. The optical signal format remains optical PAM4 format. A poor BER performance of 10−6 is attained as a result of fibre dispersion caused by 25 km SMF transport. For the state of the central wavelength moving towards a short wavelength (<1545.39 nm) or long wavelength (>1545.73 nm), however, a large part of optical spectrum of the optical PAM4 signal is filtered out. Under such state, the fibre-FSO integration has the worst transmission performance in terms of the highest BER value and the most turbid eye diagrams. A trade-off exists between the linewidth of optical signal and BER performance. Optical signal with narrow linewidth brings on high spectral efficiency but leads to low optical power and high BER value. Optical signal with wide linewidth (PAM4 format) brings on high optical power but results in low spectral efficiency and poor BER performance. Conclusively, optical signal with optimum linewidth (VSB-PAM4 format) brings on optimum optical power and spectral efficiency, and contributes to optimum BER performance. In this work, the central wavelength of the VSB filter is selected at 1545.39 nm so as to gain an optimum trade-off between the linewidth of optical signal and BER performance.

## Methods

### Framework of proposed PDM-based bi-directional fibre-FSO integration with two RSOAs scheme

The framework of the proposed PDM-based bi-directional fibre-FSO integration with two RSOAs scheme is presented in Fig. 6. Two binary pseudorandom bit sequence (PRBS) data streams with a length of 2151 at 32 Gb/s are generated from a two-channel PRBS generator. These two 32 Gb/s NRZ signals are inputted into a PAM4 converter to generate a 64 Gb/s PAM4 signal. The generated 64 Gb/s PAM4 signal is directly sent to a VCSEL. The light from a tunable laser (master laser, 1545.56 nm) is injected into a VCSEL (slave laser, 1545.53 nm) via a combination of a three-port optical circulator (OC1) and a polarisation controller (PC). The PC is adopted to match the master polarisation to the VCSEL preferred polarisation to verify the stability of injection locking. The optical PAM4 signal [insert (a)] is filtered by a VSB filter to create an optical VSB-PAM4 signal [insert (b)]. The VSB filter is an optical band-pass filter with a 3-dB bandwidth of 0.4 nm and a sharp filter slope of 500 dB/nm. To optimize the optical VSB-PAM4 signal and suppress the fibre dispersion, the central wavelength of the VSB filter is set at 1545.39 nm. In here, the operation of VSB-PAM4 modulation is to enhance the spectrum efficiency and improve the dispersion tolerance. Subsequently, the 64 Gb/s optical VSB-PAM4 signal is separated into two divisions along the two orthogonally polarised sidebands using a PC with a polarisation beam splitter (PBS). The PC is used to adjust the x-polarised [insert (c)] and y-polarised [insert (d)] sidebands. The x-polarised and y-polarised sidebands are then recombined by a polarisation beam combiner [insert (e)]. An optical delay line is deployed in the left path to make up for the phase mismatch between two paths. The optical VSB-PAM4 signal with parallel and orthogonally polarised dual sidebands is enhanced by an erbium-doped fibre amplifier (EDFA), controlled by a VOA, guided by an optical isolator, and transported through a 25-km SMF transport with a 200-m free-space transmission via two OCs (OC2 and OC3). The function of the optical isolator is to prevent the optical carrier reflected from the RSOA2. Over a span of 25-km SMF, the optical VSB-PAM4 signal is sent to a 200-m FSO link. The FSO link implements the free-space transmission by using a set of doublet lenses. A set of doublet lenses (doublet lens 1 and doublet lens 2) is utilized to emit laser beam from the ferrule of SMF to the free-space and to conduct laser beam from the free-space to the ferrule of SMF34. Thereafter, the 64 Gb/s optical VSB-PAM4 signal is split into two branches along the two orthogonally polarised sidebands utilizing a PC with a PBS. One part of the x-polarised sideband with 64 Gb/s optical VSB-PAM4 signal [insert (f)] is received by an x-polarised receiver and electrically equalised by a linear equaliser. The operation of linear equalizer is to make amends for the frequency response and increase the transmission rate of VSB-PAM4 fibre-FSO integration35. Since that PAM4 linearity is important in the transmission performance, a linear equaliser is adopted to linearize and equalise the electrical VSB-PAM4 signal. After equalisation, real-time BER measurement is executed by using a one-channel error detector (ED). Furthermore, a digital storage oscilloscope is utilized to catch the VSB-PAM4 eye diagrams. Another part of the x-polarised sideband is circulated using an OC (OC4), reused and remodulated using a RSOA (RSOA1) for uplink transmission. Similarly, the y-polarised sideband with 64 Gb/s optical VSB-PAM4 signal [insert (g)] is received by a y-polarised receiver, electrically equalised by a linear equaliser, and detected by a one-channel ED.

For uplink transmission, one part of the x-polarised sideband circulated by OC4 is reused and remodulated by RSOA1. A 10 Gb/s NRZ data stream is directly supplied to the RSOA1. Given that the bandwidth of RSOA1 is approximately 3.4 GHz, remodulating 10 Gb/s NRZ data stream using RSOA1 is relatively challenging. With electrical equalisation technique to overcome the bandwidth limitation, however, the RSOA1 can be remodulated at 10 Gb/s NRZ data stream36,37. The operation of RSOA1 is to wipe off the downstream modulated data and remodulate the upstream data. The remodulated upstream signal is circulated by OC4, promoted by an EDFA, attenuated by a VOA, circulated by OC3, and launched into a 200-m FSO link with a 25-km SMF transport. Subsequently, the upstream signal is circulated by OC2 and split by a 1 × 2 optical splitter. One part of the remodulated upstream signal with 10 Gb/s NRZ data stream is detected by a 10-GHz photodiode (PD), boosted by a 10-GHz low noise amplifier, equalised by a 10-GHz equaliser, and inputted into a BER tester for BER performance analysis. Another part of the remodulated upstream signal is injected into the RSOA2. The output of RSOA2 is circulated by OC2, blocked by optical isolator, re-circulated by OC2, and seeded into the RSOA1 through 25-km SMF transport with 200-m free-space link. The operation of RSOA2 is to erase the remodulated upstream data and reflect a pure optical carrier back to seed into the RSOA1. Consequently, two RSOAs scheme clearly erase the modulated data to attain remarkably improved upstream performance.

### Measurement setup of the frequency response of PDM-based bi-directional fibre-FSO integration

As to the frequency response, the measurement setup of the frequency response of PDM-based bi-directional fibre-FSO integration with two RSOAs scheme is presented in Fig. 6 as well. RF sweep signal (DC – 26 GHz) generated from a network analyzer is fed into the VCSEL. After electrical equalisation by an equaliser, the RF sweep signal is sent back to the network analyzer. Therefore, the frequency response of the system is measured under the states of free-running VCSEL and injection-locked VCSEL.

## References

1. 1.

Yang, Y. Multi-tier computing networks for intelligent IoT. Nature Electron. 2, 4–5 (2019).

2. 2.

Kim, J., Sung, M., Kim, E. S., Cho, S. H. & Lee, J. H. 4 × 4 MIMO architecture supporting IFoF-based analog indoor distributed antenna system for 5G mobile communications. Opt. Exp. 26, 28216–28227 (2018).

3. 3.

Yang, H., Zhang, J., Ji, Y., He, Y. & Lee, Y. Experimental demonstration of multi-dimensional resources integration for service provisioning in cloud radio over fibre network. Sci Rep. 6, 30678 (2016).

4. 4.

Wang, H. Y., Chi, Y. C. & Lin, G. R. Remote beating of parallel or orthogonally polarised dual-wavelength optical carriers for 5G millimeter-wave radio-over-fibre link. Opt. Exp. 24, 17654–17669 (2016).

5. 5.

Wu, S., Wang, H. & Youn, C. H. Visible light communications for 5G wireless networking systems: from fixed to mobile communications. IEEE Network 28, 41–45 (2014).

6. 6.

Li, C. Y. et al. A flexible two-way PM-based fibre-FSO convergence system. IEEE Photonics J. 10, 7901509 (2018).

7. 7.

Lu, H. H. et al. Bidirectional fibre-FSO and fibre-wireless convergence system with two orthogonally polarised optical sidebands. Opt. Exp. 25, 9743–9754 (2017).

8. 8.

Jurado-Navas, A. et al. 850-nm hybrid fibre/free-space optical communications using orbital angular momentum modes. Opt. Exp. 23, 33721–33732 (2015).

9. 9.

Mu, P., Pan, W. & Li, N. Analysis and characterization of chaos generated by free-running and optically injected VCSELs. Opt. Exp. 26, 15642–15655 (2018).

10. 10.

Averlant, E., Tlidi, M., Thienpont, H., Ackemann, T. & Panajotov, K. Vector cavity solitons in broad area vertical-cavity surface-emitting lasers. Sci Rep. 6, 20428 (2016).

11. 11.

Lu, G. W. et al. Flexible generation of 28 Gbps PAM4 60 GHz/80 GHz radio over fibre signal by injection locking of direct multilevel modulated laser to spacing-tunable two-tone light. Opt. Exp. 26, 20603–20613 (2018).

12. 12.

Xing, Z. et al. 100 Gb/s PAM4 transmission system for datacenter interconnects using a SiP ME-MZM based DAC-less transmitter and a VSB self-coherent receiver. Opt. Exp. 26, 23969–23979 (2018).

13. 13.

Li, F. et al. 100 Gbit/s PAM4 signal transmission and reception for 2-km interconnect with adaptive notch filter for narrowband interference. Opt. Exp. 26, 24066–24074 (2018).

14. 14.

Lu, H. H. et al. 64 Gb/s PAM4 VCSEL-based FSO link. Opt. Exp. 25, 5749–5757 (2017).

15. 15.

Wang, Y. C. et al. A high-speed 84 Gb/s VSB-PAM8 VCSEL transmitter-based fibre-FSO integration. IEEE Photonics J. 10, 7203608 (2018).

16. 16.

Chen, H. Y., Lee, J., Kaneda, N., Chen, J. & Chen, Y. K. Comparison of VSB PAM4 and OOK signal in an EML-based 80-km transmission system. IEEE Photon. Technol. Lett. 29, 2063–2066 (2017).

17. 17.

Wu, B. et al. Polarisation-insensitive remote access unit for radio-over-fibre mobile fronthaul system by reusing polarisation orthogonal light waves. IEEE Photonics J. 8, 7200108 (2016).

18. 18.

Chen, Y. W., Tseng, C. Y., Liu, S. J. & Feng, K. M. PDM DDO-OFDM with self-polarisation diversity for the backhaul of radio-over-fibre system. Optical Fibre Communication Conference (OFC), Th4A.2 (2016).

19. 19.

Chen, Y. W. et al. Over 210 Gb/s PDM multiband DDO-OFDM LR-PON downstream with simple self-polarisation diversity. Opt. Exp. 23, 18525–18533 (2015).

20. 20.

Zheng, J. et al. Photonic microwave-signal-mixing technique using phase-coherent orthogonal optical carriers for radio-over-fibre application. Opt. Lett. 39, 5263–5266 (2014).

21. 21.

Jung, S. M., Mun, S. K. H., Kang, M. & Han, S. K. Optical signal suppression by a cascaded SOA/RSOA for wavelength reusing reflective PON upstream transmission. Opt. Exp. 25, 22851–22858 (2017).

22. 22.

Lu, H. H. et al. Bidirectional fibre-wireless and fibre-VLLC transmission system based on an OEO-based BLS and a RSOA. Opt. Lett. 41, 476–479 (2016).

23. 23.

Mhatli, S. et al. Long-reach OFDM WDM–PON delivering 100 Gb/s of data downstream and 2 Gb/s of data upstream using a continuous-wave laser and a reflective semiconductor optical amplifier. Opt. Lett. 39, 6711–6714 (2014).

24. 24.

Oh, C. W. et al. Low-crosstalk full-duplex all-optical indoor wireless transmission with carrier recovery. IEEE Photon. Technol. Lett. 29, 539–542 (2017).

25. 25.

Zhan, W. et al. Optimization of modulation-canceling reflective semiconductor optical amplifier for colorless WDM transmitter applications. J. Lightw. Technol. 35, 274–279 (2017).

26. 26.

Lu, H. H. et al. Bidirectional fibre-wireless and fibre-FSO integrated system based on polarisation-orthogonal modulation scheme. Opt. Exp. 24, 17250–17258 (2016).

27. 27.

Li, C. Y. et al. Real-time PAM4 fibre-FSO and fibre-wireless hybrid system with parallel/orthogonally polarised dual-wavelength scheme. OSA Continuum 1, 320–331 (2018).

28. 28.

Murakami, A., Kawashima, K. & Atsuki, K. Cavity resonance shift and bandwidth enhancement in semiconductor lasers with strong light injection. IEEE J. Quantum Electron. 39, 1196–1204 (2003).

29. 29.

Chang, C. H., Lu, H. H., Su, H. S., Shih, C. L. & Chen, K. J. A broadband ASE light source-based full-duplex FTTX/ROF transport systems. Opt. Exp. 17, 22246–22253 (2009).

30. 30.

Lai, P. C. & Lu, H. H. Hybrid wavelength-division-multiplexing system based on a broadband amplified spontaneous emission optical source. Opt. Eng. 43, 773–777 (2004).

31. 31.

Chang, C. H. et al. A 100-Gbps multiple-input-multiple-output visible laser light communication system. IEEE/OSA J. Light. Technol. 32, 4723–4729 (2014).

32. 32.

Grüner-Nielsen, L. et al. Dispersion-compensating fibres. J. Lightw. Technol. 23, 3566–3579 (2005).

33. 33.

Litchinitser, N. M., Eggleton, B. J. & Patterson, D. B. Fibre Bragg gratings for dispersion compensation in transmission: theoretical model and design criteria for nearly ideal pulse recompression. J. Lightw. Technol. 15, 1303–1313 (1997).

34. 34.

Huang, X. H. et al. WDM free-space optical communication system of high-speed hybrid signals. IEEE Photonics J. 10, 7204207 (2018).

35. 35.

Szczerba, K., Lengyel, T., Karlsson, M., Andrekson, P. A. & Larsson, A. 94-Gb/s 4-PAM using an 850-nm VCSEL, pre-emphasis, and receiver equalization. IEEE Photon. Technol. Lett. 28, 2519–2521 (2016).

36. 36.

Kim, D., Kim, B. G., Bo, T. & Kim, H. Dither-frequency tuning technique for RSOA-based coherent WDM PON. IEEE Photon. Technol. Lett. 31, 7–10 (2019).

37. 37.

Cho, K. Y., Takushima, Y. & Chung, Y. C. 10-Gb/s operation of RSOA for WDM PON. IEEE Photon. Technol. Lett. 20, 1533–1535 (2008).

## Acknowledgements

This work was supported by the Ministry of Science and Technology of Taiwan, under grants MOST-106-2221-E-131-018-MY2, MOST-107-2636-E-027-002, MOST-107-2221-E-027-077-MY3, and MOST-107-2221-E-027-078-MY3.

## Author information

Hai-Han Lu and Wen-Shing Tsai contributed to the experiment design. Yong-Cheng Huang, Shi-Cheng Tu and Qi-Ping Huang contributed to the experimental construction and measurement. Wen-Shing Tsai contributed to the data analysis. Hai-Han Lu and Wen-Shing Tsai contributed to the manuscript writing.

Correspondence to Hai-Han Lu.

## Ethics declarations

### Competing Interests

The authors declare no competing interests.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions


text:
PhyloBetaEntropy {entropart} R Documentation

## Phylogenetic Beta Entropy of a community

### Description

Calculates the phylogenetic beta entropy of order q of a a community belonging to a metacommunity.

### Usage

PhyloBetaEntropy(NorP, NorPexp = NULL, q = 1, Tree, Normalize = TRUE, ...)
bcPhyloBetaEntropy(Ns, Nexp, q = 1, Tree, Normalize = TRUE,
Correction = "Best", CheckArguments = TRUE)
## S3 method for class 'ProbaVector'
PhyloBetaEntropy(NorP, NorPexp = NULL, q = 1, Tree, Normalize = TRUE,
..., CheckArguments = TRUE, Ps = NULL, Pexp = NULL)
## S3 method for class 'AbdVector'
PhyloBetaEntropy(NorP, NorPexp = NULL, q = 1, Tree, Normalize = TRUE,
Correction = "Best", ..., CheckArguments = TRUE, Ns = NULL, Nexp = NULL)
## S3 method for class 'integer'
PhyloBetaEntropy(NorP, NorPexp = NULL, q = 1, Tree, Normalize = TRUE,
Correction = "Best", ..., CheckArguments = TRUE, Ns = NULL, Nexp = NULL)
## S3 method for class 'numeric'
PhyloBetaEntropy(NorP, NorPexp = NULL, q = 1, Tree, Normalize = TRUE,
Correction = "Best", ..., CheckArguments = TRUE, Ps = NULL, Ns = NULL,
Pexp = NULL, Nexp = NULL)

### Arguments

 Ps The probability vector of species of the community. Pexp The probability vector of species of the metacommunity. Ns A numeric vector containing species abundances of the community. Nexp A numeric vector containing species abundances of the metacommunity. NorP A numeric vector, an integer vector, an abundance vector (AbdVector) or a probability vector (ProbaVector). Contains either abundances or probabilities of the community. NorPexp A numeric vector, an integer vector, an abundance vector (AbdVector) or a probability vector (ProbaVector). Contains either abundances or probabilities of the metacommunity. q A number: the order of entropy. Default is 1. Tree An object of class hclust, phylo, phylog or PPtree. The tree must be ultrametric. Normalize If TRUE (default), the entropy returned by the function is normalized by the height of the tree (it is the weighted average value of the entropy in each slice). If FALSE, it is the unnormalized weighted sum of the results. Correction A string containing one of the possible corrections: currently, only "ChaoShen". "Best" is the default value, it is equivalent to "ChaoShen". ... Additional arguments. Unused. CheckArguments Logical; if TRUE, the function arguments are verified. Should be set to FALSE to save time when the arguments have been checked elsewhere.

### Details

The phylogenetic entropy is the generalization of HCDT entropy to unequal species distances (Pavoine et al., 2009).

Calculation relies on TsallisBeta and PhyloApply.

Bias correction requires the number of individuals to estimate sample Coverage. Use bcPhyloBetaEntropy and choose the Correction.

Note that beta entropy value is related to alpha entropy (if q is not 1) and cannot be compared accross communities (Jost, 2007). Beta entropy of a community is not meaningful in general, do rather calculate the PhyloDiversity of the metacommunity.

The functions are designed to be used as simply as possible. PhyloBetaEntropy is a generic method. If its first argument is an abundance vector, an integer vector or a numeric vector which does not sum to 1, the bias corrected function bcPhyloBetaEntropy is called. Explicit calls to bcPhyloBetaEntropy (with bias correction) or to PhyloBetaEntropy.ProbaVector (without correction) are possible to avoid ambiguity. The .integer and .numeric methods accept Ps or Ns arguments instead of NorP for backward compatibility.

### Value

A PhyloEntropy object containing entropy values at each cut of the tree.

### References

Jost (2007), Partitioning diversity into independent alpha and beta components. Ecology 88(10): 2427-2439.

Marcon, E., Herault, B. (2015). Decomposing Phylodiversity. Methods in Ecology and Evolution 6(3): 333-339.

Pavoine, S., Love, M. S. and Bonsall, M. B. (2009). Hierarchical partitioning of evolutionary and ecological patterns in the organization of phylogenetically-structured species assemblages: Application to rockfish (genus: Sebastes) in the Southern California Bight. Ecology Letters 12(9): 898-908.

TsallisBeta, bcPhyloBetaEntropy, PhyloDiversity

### Examples

# Load Paracou data (number of trees per species in two 1-ha plot of a tropical forest
#      and their taxonomy)
data(Paracou618)
# Ps is the vector of probabilities
Ps <- as.ProbaVector(Paracou618.MC$Ps) # Probability distribution of the first plot Ps1 <- as.ProbaVector(Paracou618.MC$Psi[, 1])
# Calculate the phylogenetic Shannon beta entropy of the plot
summary(PhyloBetaEntropy(Ps1, Ps, 1, Paracou618.Taxonomy) -> e)
plot(e)

# Ns is the vector of abundances of the metacommunity
Ns <- as.AbdVector(Paracou618.MC$Ns) # Abundances in the first plot Ns1 <- as.AbdVector(Paracou618.MC$Nsi[, 1])
# Calculate the phylogenetic Shannon beta entropy of the plot
summary(bcPhyloBetaEntropy(Ns1, Ns, 1, Paracou618.Taxonomy, Correction = "Best") -> e)
plot(e)

[Package entropart version 1.6-10 Index]

format: plain


text:
- - - -

Talk:User ManualTalk:User Manual

(Bit of a n00b at wiki discussions, apologies for breaches of etiquette...) I just added the "adding a game" section. I wasn't sure whether to use "thumb" (small images) or "frame" (full-size image). I plumped for "thumb" in the end, because it looks better, but "frame" may be easier to follow. Thoughts? Spookypeanut 00:37, 23 November 2007 (CET)

Looks fine to me. I wouldn't worry about it too much, though, just keep it consistent. We can change the layout later on again. Hmm, one idea, though: Maybe we could create a special template for those pics, which is used to include them. This way, if we want to change the style of the pictures, we only have to edit that single template... Does that make sense? Anyway, thanks a lot for helping! Fingolfin 14:07, 23 November 2007 (CET)
Yes, that makes a lot of sense. I'll try and set up a template when I get chance. Spookypeanut 14:18, 23 November 2007 (CET)

Should ScummVM Launcher have a capital L? Spookypeanut 00:22, 26 November 2007 (CET)

Yes. Fingolfin 12:40, 26 November 2007 (CET)

Looking at the table of contents of the Manual, it bothers me that the top heading, i.e. "1." is "User Manual", which refers to the whole page. It means that all of the other content in the manual is 1.x, which seems silly. IMHO, the table looks neater without the
=User Manual=
line. Could that title be removed? Maybe renaming the page to Documentation/User_Manual to make it more obvious to the layman? Spookypeanut 23:31, 26 November 2007 (CET)
Done Fingolfin 19:01, 1 December 2007 (CET)

## Split the article?

Guys, it becomes huge. What about splitting it into subparts and put them hierarchially? --Sev 20:56, 30 November 2007 (CET)

Yeah, that's more sensible than the giant page. -Clone2727 21:32, 30 November 2007 (CET)
We can do that. It has various pros and cons, of coursed. Shorter pages, and split version histories. Are boons. On the other hand, moving around stuff is a bit more annoying, so is finding stuff (right now you can just Ctrl-F on the page). But since we have a Wiki search feature... Fingolfin 19:05, 1 December 2007 (CET)
So. somebody, please, do it. --Sev 20:27, 1 December 2007 (CET)
I'll do it, at some point this week, if nobody else does it first. Does the manual need to be Documentation/Manual? No other pages are under Documentation. Can't we move it to, say, UserManual? I can change that while I'm splitting it up. That way, the subsections can be, e.g., UserManual/Configuration. Spookypeanut 22:04, 3 December 2007 (CET)
I agree with that also. Though, I would say to User_Manual, to make it more legible. -Clone2727 22:40, 3 December 2007 (CET)
I was forgetting that wiki page names can have spaces :-) Have moved it now, I hope that my assumption that strong feelings of "NO!" would have shown up here is correct. I haven't changed the pages that link to it yet, as that's much more work (which I don't want to have to undo if someone thinks it shouldn't be moved). There is a redirect tho. Spookypeanut 22:29, 5 December 2007 (CET)

## First steps section

What should the "first steps" section contain? All the simple stuff (adding a game and running it) is already in the "Running and Using" section, so I can't see what should be in "first steps" at all...Spookypeanut 22:32, 5 December 2007 (CET)

The first step section has been removed. A new Quick Start page has been created though along with a reorganization of the manual pages. Hopefully it should better fit the needs. Criezy 14:22, 1 August 2009 (UTC)


text:
Image segmentation with a U-Net-like architecture. ... optimizer = keras.optimizers.Adam(lr=0.01) model.compile(optimizer=optimizer, loss=loss) Share. Multi-class weighted loss for semantic image segmentation in keras/tensorflow. ; We specify some configuration options for the model. Parameters: backbone_name – name of classification model (without last dense layers) used as feature extractor to build segmentation model. Segmentation models with pretrained backbones. Image segmentation has many applications in medical imaging, self-driving cars and satellite imaging to name a few. Keras even provides a summary function on models that will show the network’s topology from a high level perspective. Basically, it gives me the following error "Segmentation fault (core dumped)" when I try to fit a model with a conv2d layer. Python Awesome Images Implememnation of various Deep Image Segmentation models in keras Aug 30, 2018 2 min read. We use cookies on Kaggle to deliver our services, analyze web traffic, and improve your experience on the site. Check out our Introduction to Keras for engineers.. Are you a machine learning researcher? The Mask Region-based Convolutional Neural Network, or Mask R-CNN, model is one of the state-of-the-art approaches for object recognition tasks. This report will build a semantic segmentation model and train it on Oxford-IIIT Pet Dataset. Helper package with multiple U-Net implementations in Keras as well as useful utility tools helpful when working with image segmentation tasks. Segmentation models is python library with Neural Networks for The functional API in Keras is an alternate way of creating models that offers a lot net = importKerasNetwork(modelfile,Name,Value) imports a pretrained TensorFlow-Keras network and its weights with additional options specified by one or more name-value pair arguments.. For example, importKerasNetwork(modelfile,'WeightFile',weights) imports the network from the model file modelfile and weights from the weight file weights. In this three part series, we walked through the entire Keras pipeline for an image segmentation task. net = importKerasNetwork(modelfile,Name,Value) imports a pretrained TensorFlow-Keras network and its weights with additional options specified by one or more name-value pair arguments.. For example, importKerasNetwork(modelfile,'WeightFile',weights) imports the network from the model file modelfile and weights from the weight file weights. In this three part series, we walked through the entire Keras pipeline for an image segmentation task. Some times, it is useful to train only randomly initialized ... Our SemanticLogger is a custom Keras callback. Ask Question Asked 1 year ago. For Unet construction, we will be using Pavel Yakubovskiys library called segmentation_models, for data augmentation albumentation library. # Generate predictions for all images in the validation set, """Quick utility to display a model's prediction. from_config (config[, custom_objects]) Instantiates a Model from its config (output of get_config()). # continue with usual steps: compile, fit, etc.. High level API (just two lines to create NN), Train network from scratch with randomly initialized weights. Author: fchollet Date created: 2019/03/20 Last modified: 2020/04/20 Description: Image segmentation model trained from scratch on the Oxford Pets dataset. # Note that the model only sees inputs at 150x150. from keras_segmentation.pretrained import pspnet_50_ADE_20K , pspnet_101_cityscapes, pspnet_101_voc12 model = pspnet_50_ADE_20K() # load the pretrained model trained on ADE20k dataset model = pspnet_101_cityscapes() # load the pretrained model trained on Cityscapes dataset model = pspnet_101_voc12() # load the pretrained model trained on Pascal VOC 2012 dataset # load … So we are given a set of seismic images that are $101 \\times 101$ pixels each and each pixel is classified as either salt or sediment. Both libraries get updated pretty frequently, so I prefer to update them directly from git. Are you an engineer or data scientist? The main features of this library are: High level API (just two lines to create NN) 4 models architectures for binary and multi class segmentation (including legendary Unet) 25 available backbones for each architecture. It can be seen as an image classification task, except that instead of classifying the whole image, you’re classifying each pixel individually. In case you have non RGB images (e.g. We load the EMNIST dataset, reshape the data (to make it compatible with TensorFlow), convert the data into float32 format (read here why), and then scale the data to the $$[0, 1]$$ range. The structure follow the Tensorflow tutorial on how to do GAN closely. If you want to make your own dataset, a tool like labelme or GIMP can be used to manually generate the ground truth segmentation masks.Assign each class a unique ID. Image segmentation has many applications in medical imaging, self-driving cars and satellite imaging to name a few. The goal of image segmentation is to label each pixel of an image with a corresponding class of what is being represented. Trains the model on data generated batch-by-batch by a Python generator (or an instance of Sequence). Active 8 months ago. However, if you take a look at the IOU values it is near 1 which is almost perfect. image-segmentation-keras. while initializing the model. keras-rcnn. (Tensorflow) framework. Object detection is a challenging computer vision task that involves predicting both where the objects are in the image and what type of objects were detected. Suppose we want to know where an object is located in the image and the shape of that object. This article is a comprehensive overview including a step-by-step guide to implement a deep learning image segmentation model.. We shared a new updated blog on Semantic Segmentation here: A 2021 guide to Semantic Segmentation Nowadays, semantic segmentation is one of the key problems in the field of computer vision. Features: [x] U-Net models implemented in Keras [x] Vanilla U-Net implementation based on the original paper [x] Customizable U-Net Pixel-wise image segmentation is a well-studied problem in computer vision. The output itself is a high-resolution image (typically of the same size as input image). from keras_segmentation.models.model_utils import transfer_weights from keras_segmentation.pretrained import pspnet_50_ADE_20K from keras_segmentation.models.pspnet import pspnet_50 pretrained_model = pspnet_50_ADE_20K() This is the task of assigning a label to each pixel of an images. 4.3 Model Architecture: What does one input image and corresponding segmentation mask look like. I will write more details about them later. A set of models which allow easy creation of Keras models to be used for segmentation tasks. Object detection is a challenging computer vision task that involves predicting both where the objects are in the image and what type of objects were detected. There are several ways to choose framework: Provide environment variable SM_FRAMEWORK=keras / SM_FRAMEWORK=tf.keras before import segmentation_models; Change framework sm.set_framework('keras') / sm.set_framework('tf.keras'); You can also specify what kind of … In this post, we will discuss how to use deep convolutional neural networks to do image segmentation. From this perspective, semantic segmentation is actually very simple. Image segmentation with a U-Net-like architecture, Prepare paths of input images and target segmentation masks. Image Segmentation. Features: [x] U-Net models implemented in Keras [x] Vanilla U-Net implementation based on the original paper [x] Customizable U-Net First of all, you need Keras with TensorFlow to be installed. Python Awesome Images Implememnation of various Deep Image Segmentation models in keras Aug 30, 2018 2 min read. This report will build a semantic segmentation model and train it on Oxford-IIIT Pet Dataset. I will start by merely importing the libraries that we need for Image Segmentation. We also use the extra_keras_datasets module as we are training the model on the EMNIST dataset. For more detailed information about models API and use cases Read the Docs. Provide environment variable SM_FRAMEWORK=keras / SM_FRAMEWORK=tf.keras before import segmentation_models. Ladder Network in Kerasmodel achives 98% test accuracy on MNIST with just 100 labeled examples My network outputs gradient-rich images, which look like … We would need the input RGB images and the corresponding segmentation images. We import the TensorFlow imports that we need. By crunching data collected from a player’s personal swing history, the virtual caddie can recommend an optimal strategy for any golf cours… keras.models.Model. This is nice, but a bit useless if we cannot save the models that we’ve trained. These models can be used for prediction, feature extraction, and fine-tuning. ### [Second half of the network: upsampling inputs] ###, # Free up RAM in case the model definition cells were run multiple times, __________________________________________________________________________________________________, ==================================================================================================, # Split our img paths into a training and a validation set, # Instantiate data Sequences for each split, # We use the "sparse" version of categorical_crossentropy. The diagram generated by model.summary() shows important high level information about the model such as the output shapes of each layer, the number of … Image segmentation models with pre-trained backbones with Keras. grayscale or some medical/remote sensing data) In this case, all you need is just pass encoder_freeze = True argument from keras_segmentation.models.model_utils import transfer_weights from keras_segmentation.pretrained import pspnet_50_ADE_20K from keras_segmentation.models.pspnet import pspnet_50 pretrained_model = pspnet_50_ADE_20K() In the TGS Salt Identification Challenge, you are asked to segment salt deposits beneath the Earth’s surface. Skip to main content Switch to mobile version Warning Some features may not work without JavaScript. Now let’s learn about Image Segmentation by digging deeper into it. Implementation of the paper The One Hundred Layers Tiramisu : Fully Convolutional DenseNets for Semantic Segmentation… From this perspective, semantic segmentation is actually very simple. This could be because the non-tumor area is large when compared to the tumorous one. Segmentation models is python library with Neural Networks for Image Segmentation based on Keras ( Tensorflow) framework. .. code:: python import keras # or from tensorflow import keras keras.backend.set_image_data_format('channels_last') # or keras.backend.set_image_data_format('channels_first') Created segmentation model is just an instance of Keras Model, which can be build as easy as: .. code:: python model = sm.Unet() Depending on the … from keras_segmentation.pretrained import pspnet_50_ADE_20K , pspnet_101_cityscapes, pspnet_101_voc12 model = pspnet_50_ADE_20K() # load the pretrained model trained on ADE20k dataset model = pspnet_101_cityscapes() # load the pretrained model trained on Cityscapes dataset model = pspnet_101_voc12() # load the pretrained model trained on Pascal VOC 2012 dataset # load … Of course, there’s so much more one could do. Weights are downloaded automatically when instantiating a model. Fine-tuning from existing segmentation model. Let’s see how we can build a model using Keras to perform semantic segmentation. Both libraries get updated pretty frequently, so I prefer to update them directly from git. Subtract one to make them 0, 1, 2: ### [First half of the network: downsampling inputs] ###. We are generating a new solution to the business problem with available libraries: tensorflow, keras and segmentation_models. I'm having issues with Keras. Content 1.What is semantic segmentation 2.Implementation of Segnet, FCN, UNet , PSPNet and other models in Keras 3. I'm using a GAN to generate pixel-art images. Image Segmentation works by studying the image at the lowest level. % Total % Received % Xferd Average Speed Time Time Time Current, # Display auto-contrast version of corresponding target (per-pixel categories), """Helper to iterate over the data (as Numpy arrays). Image For Unet construction, we will be using Pavel Yakubovskiys library called segmentation_models, for data augmentation albumentation library. It’s even effective with limited dataset images. We import the TensorFlow imports that we need. Since the library is built on the Keras framework, created segmentation model is just a Keras Model, which can be created as easy as: Depending on the task, you can change the network architecture by choosing backbones with fewer or more parameters and use pretrainded weights to initialize it: Change number of output classes in the model: Same manimulations can be done with Linknet, PSPNet and FPN. The Keras Python library makes creating deep learning models fast and easy. Today I’m going to write about a kaggle competition I started working on recently. In the segmentation images, the pixel value should denote the class ID of the corresponding pixel. We also use the extra_keras_datasets module as we are training the model on the EMNIST dataset. We load the EMNIST dataset, reshape the data (to make it compatible with TensorFlow), convert the data into float32 format (read here why), and then scale the data to the $$[0, 1]$$ range. # Blocks 1, 2, 3 are identical apart from the feature depth. They are stored at ~/.keras/models/. By using Kaggle, you agree to our use of cookies. Follow answered Dec … Today I’m going to write about a kaggle competition I started working on recently. Unlike object detection models, image segmentation models can provide the exact outline of the object within an image. Image Segmentation Keras : Implementation of Segnet, FCN, UNet and other models in Keras. you have few different options: © Copyright 2018, Pavel Yakubovskiy I will write more detailed about them later. The presentation of this architecture was first realized through the analysis of biomedical images. As previously featured on the Developer Blog, golf performance tracking startup Arccos joined forces with Commercial Software Engineering (CSE) developers in March in hopes of unveiling new improvements to their “virtual caddie” this summer. Segmentation based import matplotlib.pyplot as plt import seaborn as sns import keras from keras.models import Sequential from keras.layers import Dense, Conv2D , MaxPool2D , Flatten , Dropout from keras.preprocessing.image import ImageDataGenerator from keras.optimizers import Adam from sklearn.metrics import classification_report,confusion_matrix import tensorflow as tf import cv2 import os import numpy as np on Keras Skip to main content Switch to mobile version Warning Some features may not work without JavaScript. Now, fortunately, the Keras … Because we’re predicting for every pixel in the image, this task is commonly referred to as dense prediction. Date created: 2019/03/20 As you can see from the above results, the ResUNet model performs best compared to other models. As the model file was a data conversion from another weights file in another format, I went and regenerated the Keras model for the latest version of Keras. """, # Display results for validation image #10. The first step in training our segmentation model is to prepare the dataset. With our model trained, we’ll implement a second Python script, this one to handle inference (i.e., making object detection predictions) on new input images. # Train the model, doing validation at the end of each epoch. The Matterport Mask R-CNN project provides a library that allows you to develop and train By default it tries to import keras, if it is not installed, it will try to start with tensorflow.keras framework. Change framework sm.set_framework ('keras') / sm.set_framework ('tf.keras') You can also specify what kind of image_data_format to use, segmentation-models works with both: channels_last and channels_first . We can pass it to model.fit to log our model's predictions on a small validation set. image-segmentation-keras. Keras Applications are deep learning models that are made available alongside pre-trained weights. The Matterport Mask R-CNN project provides a library that allows you to develop and train From structuring our data, to creating image generators to finally training our model, we’ve covered enough for a beginner to get started. There are several ways to choose framework: Provide environment variable SM_FRAMEWORK=keras / SM_FRAMEWORK=tf.keras before import segmentation_models; Change framework sm.set_framework('keras') / sm.set_framework('tf.keras'); You can also specify what kind of … Helper package with multiple U-Net implementations in Keras as well as useful utility tools helpful when working with image segmentation tasks. First of all, you need Keras with TensorFlow to be installed. View in Colab • GitHub source 4.3 Model Architecture: It is limited in that it does not allow you to create models that share layers or have multiple inputs or outputs. In this article,we’ll discuss about PSPNet and implementation in Keras. The sequential API allows you to create models layer-by-layer for most problems. We will interactively visualize our models' predictions in Weights & Biases. Segmentation Models (Keras / TF) & Segmentation Models PyTorch (PyTorch) A set of popular neural network architectures for semantic segmentation like Unet, Linknet, FPN, PSPNet, DeepLabV3(+) with pretrained on imagenet state-of-the-art encoders (resnet, resnext, efficientnet and others). Do you ship real-world machine learning solutions? Fine-tuning from existing segmentation model. Fully Connected DenseNets for Semantic Segmentation. About Keras Getting started Introduction to Keras for engineers Introduction to Keras for researchers The Keras ecosystem Learning resources Frequently Asked Questions Developer guides Keras API reference Code examples Why choose Keras? Author: fchollet Pre-trained models and datasets built by Google and the community Tools Ecosystem of tools to help you use TensorFlow The following example shows how to fine-tune a model with 10 classes . By default it tries to import keras, if it is not installed, it will try to start with tensorflow.keras framework. The Mask Region-based Convolutional Neural Network, or Mask R-CNN, model is one of the state-of-the-art approaches for object recognition tasks. Of course, there’s so much more one could do. The task of semantic image segmentation is to classify each pixel in the image. We have to assign a label to every pixel in the image, such that pixels with the same label belongs to that object. View in Colab • GitHub source Keras and TensorFlow Keras. Getting started. Training is expensive and we shouldn’t want to retrain a model every time we want to use it. ; We specify some configuration options for the model. Keras Segmentation Models. Pre-trained models and datasets built by Google and the community Tools Ecosystem of tools to help you use TensorFlow It can be seen as an image classification task, except that instead of classifying the whole image, you’re classifying each pixel individually. Now It works. The main features of this library are: High level API (just two lines of code to create model for segmentation) 4 models architecture U-Net is more successful than conventional models, in terms of architecture and in terms pixel-based image segmentation formed from convolutional neural network layers. Keras documentation. Revision 94f624b7. Image segmentation models with pre-trained backbones with Keras. Description: Image segmentation model trained from scratch on the Oxford Pets dataset. All backbones have weights trained on 2012 ILSVRC ImageNet dataset (, # set all layers trainable and recompile model. From structuring our data, to creating image generators to finally training our model, we’ve covered enough for a beginner to get started. # Ground truth labels are 1, 2, 3.

Grainger Today Facebook, Carbon Microphone Design, Luigi's Mansion 3 Gamecube Controller, Empire Apartments Philadelphia Reviews, Used Crawler Crane For Sale, Tiktok Tanning Salon Girl, Registered Bernedoodle Breeders, Leaving Hbo Max July 2020, Fantasy Lyrics Vedo, Ntu Gerpe Result, Helicopter Game Online, New York Deli Westborough,


text:
# Math 112 Solutions for 3.7, p. 286

2.
describes a parabola opening upward with zeros at 0 and and vertex . has the same zeros but vertex and opens downward.
3.
Let . The critical points are 0 and . and . Hence if and , has global minima at and a local maximum at 0. If and , has only a global minimum at 0. If and , has global maxima at and a local minimum at 0. If and , has only a global maximum at 0.

4.
Let . The critical points are at and the inflection point is at . The average of the -coordinates of the critical points is clearly the -coordinate of the inflection point. The same is true for the -coordinates.
5.
If the cubic in Problem 4 has only one critical point, it is because and the critical point is , the inflection point.
8.
Let the cubic have the equation . If (3, 0) is the local maximum and (5, -1) is the inflection point, then the local minimum is at (7, -2) and is positive. The inflection point is at , so . Since the first critical point is , we get . Finally, using , we get . Hence with .

10.
(a) Since 120 mi/hr = 176 ft/sec, the maximum steepness is , so the minimum slope is .

(b) With , we let the inflection point be the origin; since the inflection point is at , we have . Since , we have . The slope at the inflection point is . The critical points are then at and . Therefore the run is and the rise is .

(c) If the rise is -12,000 feet, then the run is feet miles.

(d) From part (c) we have .

11.
Increasing magnifies the graph vertically. Increasing compresses the graph horizontally.

13.
If the maximum of is (7,55), then and . It follows that is less than 20 when , which is true for .

14.
.

17.
.

20.
Demand will be about 24 sales per week, and it will take another four weeks to reach it.

22.

25.
Think of as an amplitude to a sunisoid.

26.
The graph is a bell curve having its maximum at and inflection points at .

27.
Let the chain hang from the points and with its low point at the origin. Since the low point is a critical point, we get , so . Since the origin is on the curve, we have , and . Then , so . Thus .


text:
The 35th International Symposium on Lattice Field Theory

18-24 June 2017
Palacio de Congresos
Europe/Madrid timezone
Home > Timetable > Session details > Contribution details

Contribution Parallel

Andalucía II
Hadron Spectroscopy and Interactions

Baryon interactions from lattice QCD with physical masses --- strangeness $S=-1$ sector ---

Speakers

• Dr. Hidekatsu NEMURA

Content

We present our recent results of baryon interactions with strangeness $S=-1$ based on Nambu-Bethe-Salpeter (NBS) correlation functions calculated from lattice QCD with almost physical quark masses corresponding to $(m_\pi,m_K)\approx(146,525)$ MeV and large volume $(La)^4=(96a)^4\approx$ (8.1 fm)$^4$. In order to perform a comprehensive study of baryon interactions[1], a large number of NBS correlation functions from NN to $\Xi\Xi$ are calculated simultaneously by using large scale computer resources. In this contribution, we focus on the strangeness $S=-1$ channels[2] of the hyperon interactions by means of HAL QCD method. The latest results will be presented.

References:

[1] H. Nemura, Comput. Phys. Commun. 207, 91 (2016) [arXiv:1510.00903 [hep-lat]].

[2] H. Nemura et al., PoS LATTICE2016, 101 (2017) [arXiv:1702.00734 [hep-lat]].

Preferred track (if multiple tracks have been selected)

Hadron Spectroscopy and Interactions


text:
# Partial Derivative Q: continuity and directional deriv's

Tags:
1. Mar 16, 2015

### kostoglotov

1. The problem statement, all variables and given/known data

a) Show that the function $$f(x,y)=\sqrt[3]{xy}$$ is continuous and the partial derivatives $$f_x$$ and $$f_y$$ exist at the origin but the directional derivatives in all other directions do not exist

b) Graph f near the origin and comment on how the graph confirms part (a).

2. The attempt at a solution

This is an odd question to me. First off, what do they mean by "directional derivatives in all other directions" ? They only specify the origin as a point at which # f_x # and $$f_y$$ should exist, but no direction is specified, so how can there be other directions? What other directions? All directions? Ie, $$f_x$$ and $$f_y$$ exist at (0,0,0) but in no direction do valid directional derivatives exist??

The other quandary for me is how to show that it is continuous at the origin. I have found the grad vector

$$\nabla f(x,y) = \frac{1}{3}\left<\frac{y^{\frac{1}{3}}}{x^{\frac{2}{3}}},\frac{x^{\frac{1}{3}}}{y^{\frac{2}{3}}}\right>$$

But there is no general L'Hopital rule for part. deriv's, so I can't decide these limits as $$(x,y) \rightarrow (0,0)$$. And if y is taken as a constant for $$f_x$$ then it just goes to infinity as x approaches zero and likewise for $$f_y$$.

So I decided to just go ahead and graph this function in Matlab. I used meshgrid and symbolics to get two different but similar graphs.

1) meshgrid

[x,y] = meshgrid(-1:0.01:1);
z = (x.*y).^(1/3);
z = real(z);
surf(x,y,z), axis equal, xlabel('x'), ylabel('y'), zlabel('z'), shading interp, colormap(jet)

2) using symbolics and ezsurf

syms x y
ezsurf((x*y)^(1/3)), axis equal, xlabel('x'), ylabel('y'), zlabel('z')

Why does using the symbolic ezsurf throw out half the surface? It must be related to taking only the real values for z in meshgrid yes??

So the meshgrid form interpolates points around the origin, suggesting that the surface exists at the origin, but the symbolic toolbox suggests that the surface doesn't exist at the origin.

I am very confused. Please help :)

2. Mar 16, 2015

### RUber

The question about directional derivatives is based on the definition of a derivative as the $\lim_{|h|\to0} \frac{f( x+h_x,y+h_y) - f(x,y) }{|h|}$ looking at it this way, you allow for h to be a vector in any direction on the x-y plane.
You should be able to show that this limit is continuous if h only has x components or y components, but as soon as you have both, the limit is not the same for h as it is for -h and therefore the derivative does not exist in that direction.
Considering that explanation, look again at your plot, and you should see what I am saying.

3. Mar 16, 2015

### Ray Vickson

A directional derivative at $\vec{p}_0 = (x_0,y_0)$ in direction $\vec{d}$---YOU pick whatever $\vec{d}$ YOU want!---is defined as
$$D_{\vec{d}} f(\vec{p}_0) = \lim_{t \to 0+} \frac{ f(\vec{p}_0 + t \vec{d}) - f(\vec{p}_0)}{t}$$
In other words, for small $t > 0$ we should have
$$f(\vec{p}_0 + t \vec{d}) \approx f(\vec{p}_0) + t \cdot D_{\vec{d}} f(\vec{p}_0)$$

So, pick any direction non-axial direction $\vec{d}$ that you want, and see if the required limit exists.

Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted


text:
## sign challenge review.

A comparison of many possible Python implementations of the sign function.

I challenged you 🏆...

You delivered 💪!

I asked you to implement the sign function in Python 🐍.

Now I'll go over some alternatives and tell you what I like ✅ and dislike ❌ about them.

I'll also tell you which one I think is the best, most Pythonic ✨ one.

👇

# sign: returns 1 for positive numbers,
# 0 for 0, and -1 for negative numbers.
>>> sign(73.73)
1
>>> sign(0)
0
>>> sign(-42)
-1

By the way, for reference, here is the original challenge:

Let's start with the “basic” if: ... elif: ... else: ... approach first.

Gets the job done, is simple, and is easy to read.

Funnily enough, the only one to share that solution was a very seasoned Python 🐍 programmer, author, and trainer, @dabeaz 🙃

Now, a couple of notes:

def sign(x):
if x > 0:
return 1
elif x < 0:
return -1
else:
return 0

For the arg name, we don't need something VERY long like number, but don't shorten it to n.

n has a connotation for positive whole numbers, and sign accepts other numbers.

I also prefer to have the else: return the 0, and that's to preserve the symmetry of 1 and -1.

We can use this as the cornerstone for many interesting implementations.

For such a simple function, it might look like a bummer that we have to spend 6 lines implementing it.

Can we shorten it a bit, perhaps?

Using conditional expressions, we can:

def sign(x):
if x == 0:
return 0
else:
return 1 if x > 0 else -1

We can also get rid of the else, which may reinforce the sense of “0 is just an edge-case”...

Depending on how you usually read Python code.

Here it is:

def sign(x):
if x == 0:
return 0

return 1 if x > 0 else -1

Now, you might be thinking...

Instead of returning the conditional expression...

Can I compute Abacus the 1 or -1?

Well, you can!

And many of you submitted things like what follows.

But that gives rise to another issue...

def sign(x):
if x == 0:
return 0

return int(abs(x) // x)

Do we really need the int?

Well, depends on what you want your function to return, because abs(x) // x returns a float if x is a float.

And what about floating point inaccuracies? Does int(abs(x) // x) always return 1?

To be honest with you, I am not entirely sure!

I couldn't find a value of x for which abs(x) // x evaluated to 0.99999(something).

However, to be extra safe, you could replace int with round.

Ok, and are there “safer” alternatives?

There are!

Have you heard about the Truthy and Falsy values of things in Python?

That makes it so that all objects can be interpreted as Booleans when needed.

But did you know that Booleans (True and False) can be handled as integers??

Hence, you can write this

What⁉

def sign(x):
if x == 0:
return 0

return (x > 0) - (x < 0)

I like the fact that it looks like a face:

(x > 0) - (x < 0)

The - is the nose, and the (...) are the eyes!

And there's even the added benefit that you can drop the if statement:

def sign(x):
return (x > 0) - (x < 0)

In my opinion, this is very cool!

BUT it's not the “way to go” in Python, generally.

This looks more like a thing you'd do in an array-oriented language, like APL.

So, we steered away from Pythonic solutions.

Can we get back on track?

Yes!

But let's not, for now 😂

I just wanna show you another interesting one, making use of the integer value of Boolean values 👇

It looks interesting, and it is useful to understand how it works.

But I'm sure the majority won't feel this is Pythonic.

def sign(x):
return -1 if x < 0 else int(bool(x))

I think the conditional expression wasn't that bad, right?

Maybe we could improve on it?

Well, we can try!

What if we nest two conditional expressions to handle the three cases?

def sign(x):
return 1 if x > 0 else -1 if x < 0 else 0

I like to have 0 at the end, because I like the symmetry between

• x > 0 -> 1
• x < 0 -> -1

This might sound silly to you, but these symmetries and patterns really make my life easier!

That's why I prefer this ordering over, say, this one:

def sign(x):
return 0 if x == 0 else 1 if x > 0 else -1

But we can still be friends if you order things your way 😁

Another thing to notice is that all the ifs with if x == 0 could be replaced by if not x.

Some might prefer it, some might not.

I have no strong feelings for neither 🤷♂

Is there any other way to emulate the “choose one of -1, 0, or 1” behaviour without using long ifs..?

Someone submitted this, and I gotta say:

It looks odd, but it is incredibly easy to read.

Is it Pythonic? I don't think so 😢

def sign(x):
return 0 if x == 0 else {True: 1, False: -1}[x > 0]

Instead of accessing a dictionary, we can also try to index into a list.

Now, close your eyes and skip this if you don't want to be horrified 👇

Again, not recommended style, but understanding how it works gives insights into how Python works.

def sign(x):
return 0 if x == 0 else [-1, 1][x > 0]

This can be taken one step further.

Should you?

Nah.

Will I do it nonetheless?

Absolutely!

Here it goes 👇 Can you understand how it works?

def sign(x):
return [-1, 0, 1][(x >= 0) + (x > 0)]

Hmmm, 'kay.

That's all nice and fine.

What if I don't want to use ifs, but also no weird Boolean computations?

Then, maybe you could try the new pattern matching features from Python 3.10!

In case you need to learn it:

Can we use match to solve this?

Well, not directly.

Which shows that match really isn't the tool for this job.

But we can use guards and get away with it 👇

But if you look closely, the case statements aren't doing anything!

So, can we get rid of them..?

def sign(x):
match x:
case x if x > 0: return 1
case x if x < 0: return -1
case _: return 0

Absolutely, and you end up with this 👇

Equivalent to the if from the beginning, but condensed.

Canned if-blocks, everyone!

def sign(x):
if x > 0: return 1
elif x < 0: return -1
else: return 0

Ok!

So, we used match and ended up getting away from it?

Unacceptable!

One more try:

match is supposed to be useful for structural pattern matching.

Can we match patterns, then?

Of course. Here's a solution no one submitted, but that I came up with 👇

What do you make of it?

def sign(x):
match x > 0, x < 0:
case True, False: return 1
case False, True: return -1
case False, False: return 0

All in all, there are plenty of interesting solutions in this thread.

So, which ones do I prefer?

Well, personally, I love weird snippets of code 😂 But I don't write them in production.

So, that means I end up with a split preference, between 👇

and the slightly condensed if: ... elif: ... else: ... block 👇

I don't know why, but it really bothers me that such a simple function takes 6 lines of code as a “full” if: ... elif: ... else: ... block!

(Does anyone else feel like that?)

Thanks for making it so far!

Finally, if you have ideas for future challenges, feel free to send them to me!

Write them down in the comments, or send me a DM!

Then, if you want to keep learning a lot about Python 🐍, follow me @mathsppblog.

I'll see you soon! 👋

P.S. I should give credit to everyone who contributed to this thread by sending their solutions.

Many people replied, so I don't think I should mention all of them..? You can find all of them in the original challenge, linked at the beginning.

Thanks a lot for participating!

I hope you learned something new! If you did, consider following the footsteps of the readers who bought me a slice of pizza 🍕. Your small contribution helps me produce this content for free and without spamming you with annoying ads.


text:
# Cannot find the maximum value of a function

Consider the following function:

$a Sin(x)+b Cos(x)$

I tried to obtain the maximum value of this function using MaxValue[]:

MaxValue[a Sin[x] + b Cos[x], x, Reals]

I expect Mathematica to return the following answer:

$\sqrt{a^2+b^2}$

But Mathematica cannot find the answer. Why does this happen?

• note it cant even do this: MaxValue[a Sin[x], x, Reals] – george2079 Aug 26 '13 at 15:53
• @george2079 But I am puzzled, is it a bug or a feature ? – yarchik Mar 16 '17 at 20:47

This is essentially what b.g did, except by using Max instead of the second derivative constraint we get the result without rewriting the original expression.

f[x_] = a Sin[x] + b Cos[x];
Simplify[
Max[f[x] /. Solve[ {f'[x] == 0 }, x] ],
Assumptions -> {Element[a, Reals], Element[b, Reals],
Element[C[1], Integers]}]

Sqrt[a^2 + b^2]

I'm a bit puzzled why the conditional expression on C[1] doesn't simplify out on its own when the C[1] is gone from the expression..?

You can use a more down to earth method :

func = Sqrt[a^2 + b^2] Sin[x + f];
sol = First@Solve[{D[func, {x, 1}] == 0, D[func, {x, 2}] < 0,
f \[Element] Reals, a \[Element] Reals, b \[Element] Reals}, x, Reals];

Simplify[func /. sol, Assumptions -> C[1] \[Element] Integers]
(* Sqrt[a^2 + b^2] *)

• Would you please explain why a Sin[x] + b Cos[x] is defined as Sqrt[a^2 + b^2] Sin[x + f]. – M6299 Aug 24 '13 at 10:42
• You can do TrigExpand[ Sin[x + f]] and match Cos[f] to a/Sqrt[a^2 + b^2] and Sin[f] to b/Sqrt[a^2 + b^2]. – b.gates.you.know.what Aug 24 '13 at 11:36

Try this:

MaxValue[a Sin[x] + b Cos[x] /. x -> 2 ArcTan[t] // TrigExpand, t, Reals]

MaxValue[{y, Reduce[{y == a Sin[x] + b Cos[x], #}, y, {x}, Reals]}, y]& /@
{Xor[a == 0, b == 0], a!=0 && b!=0} // FullSimplify

• not quite.. if a==0 the max is abs[b]. I suppose because ArcTan doesn't admit the full range of angles. – george2079 Aug 24 '13 at 20:11
• Thanks for the answer. The ranges of x and ArcTan[t] are different. How come x can be substituted with ArcTan[t]? – M6299 Aug 25 '13 at 5:52


text:
# Effect Sizes: Why Significance Alone is Not Enough

So, you performed a test for significance and obtained a positive result. That’s great but it’s not time to celebrate yet. You may ask: Why? Isn’t a significant test result sufficient to show the existence of an effect? This statement, however, is not true for two reasons. First, a significant result only indicates the existence of an effect but doesn’t prove it. For example, at a significance level of 5%, an exact test will yield a false positive result in 5% of the cases. Second, a significant result does not necessarily make a statement about the magnitude of the effect. In this post, we’ll investigate the difference between statistical significance and the effect size, which describes the magnitude of an effect.

## Significance depends on sample size and effect size

To exemplify the difference between statistical significance and effect size, let’s assume that we are conducting a study investigating two groups, $$G_1$$ and $$G_2$$, with respect to two outcomes, $$Y_1$$ and $$Y_1$$. For this purpose, we’ll generate artificial data and determine whether measurements are independent of the groups using the chi-squared test.

### Generation of artificial data

To explore how significance depends on both sample size and effect size, we are going to define a function that builds contingency tables with the following properties:

• The number of samples is sample.size
• Each group contains 50% of the measurements
• The relative difference between the frequencies of the two groups is diff

This is the function:

build.contingency.table <- function(sample.size, diff) {
g.size <- sample.size / 2 # group size
data <- matrix(c(g.size/2 - (diff * g.size/2),
g.size/2 + (diff * g.size/2),
g.size/2, g.size/2),
nrow = 2, byrow = TRUE,
dimnames = list(c("G1", "G2"), c("Y1", "Y2")))
return(data.frame(data))
}

### Data generation

We’ll generate three confusion matrices:

1. A confusion matrix for small sample size and relatively large effect size
2. A confusion matrix for medium sample size and medium effect size
3. A confusion matrix for large sample size and relatively small effect size

In the following, the results for these three data sets are described.

#### Smallest sample size, largest effect

For the first data set, let’s assume a sample size of 40 and let the difference between the group frequencies be 60%. This means that $$0.6 \cdot 20 = 12$$ samples in $$G_1$$ will have a different outcome than in $$G_2$$. Under this assumption, we obtain the following contingency table and significance result when testing whether the observed frequencies are independent of the groups using the $$\chi^2$$ test:

data.L <- build.contingency.table(40, 0.6)
print(data.L)
##    Y1 Y2
## G1  4 16
## G2 10 10
chi.result.L <- chisq.test(data.L)
print(chi.result.L$p.value) ## [1] 0.09742169 Interestingly, the p-value of 0.0974 is not significant at the 5% level despite the large difference between the two groups. #### Medium sample size, medium effect Let’s now assume that we’ve collected a greater number of samples than in the first study, namely a total of $$10\,000$$ samples. This time, however, the effect size is smaller: the difference between the two groups is only $$\frac{200}{5000} = 4\%$$. What do you think, will a $$\chi^2$$ test on the corresponding contingency table be insignificant? Let’s see the result: data.M <- build.contingency.table(10000, 0.04) print(data.M) ## Y1 Y2 ## G1 2400 2600 ## G2 2500 2500 chi.result.M <- chisq.test(data.M) print(chi.result.M$p.value)
## [1] 0.04765904

Indeed, the p-value of 0.0477 is sufficiently small for a significant result at the 5% level. This could come as a surprise considering that the outcomes were more similarly split across both groups than in the first experiment where the result was not significant.

#### Largest sample size, smallest effect

To illustrate what can happen at a very large sample size, let’s generate a data set with an even larger sample size and an even smaller difference between the groups. We’ll take a sample of 1 million measurements and enforce a difference between the groups of only $$1\%$$:

data.S <- build.contingency.table(1000000, 0.01)
print(data.S)
##        Y1     Y2
## G1 247500 252500
## G2 250000 250000
chi.result.S <- chisq.test(data.S)
print(chi.result.S$p.value) ## [1] 5.790922e-07 In this case, the p-value is even smaller than for the previous data set although the difference between the two groups has been reduced from 4% to 1%. ### Discussion How can these results be explained? We have to remember that the p-value indicates how likely it is to obtain a test result that is at least as extreme by chance. Since chance is reduced with greater sample sizes, it becomes easier to show differences between groups (i.e. the statistical power) increases). Therefore, at large sample sizes, even small effects can become significant, while for small sample sizes, even large effects may not be significant. ## Determining the effect size with Cramer’s V The effect size of the $$\chi^2$$ test can be determined using Cramer’s V. Cramer’s V is a normalized version of the $$\chi^2$$ test statistic. It is defined by $V = \sqrt{\frac{\chi^2}{n \cdot (c - 1)}}$ where $$n$$ is the sample size and $$c = \min(m,n)$$ is the minimum of the number of rows $$m$$ and columns $$n$$ in the contingency table. Interpretation of Cramer’s V is easy due to $$V \in [0,1]$$. For large effects, $$V$$ will approach 1 but if there’s no effect $$V$$ will be close to 0. ### An R function for computing Cramer’s V Since there’s no function to compute Cramer’s V in base R, we’ll implement it ourselves as follows: cramer.v <- function(contingency.tab) { chi <- chisq.test(contingency.tab, correct = FALSE)$statistic
n <- sum(contingency.tab)
c <- min(nrow(contingency.tab), ncol(contingency.tab))
V <- sqrt(chi / (n * (c-1)))
return(as.numeric(V))
}

### Interpreting Cramer’s V

To interpret Cramer’s V, the following approach is often used:

• $$V \in [0.1, 0.3]$$: weak association
• $$V \in [0.4, 0.5]$$: medium association
• $$V > 0.5$$: strong association

### Cramer’s V for the three data sets

We will determine Cramer’s V for the three data sets data.L, data.M, and data.S, which exhibit relatively large, medium, and small effect sizes, respectively.

data <- list("Largest_Effect" = data.L,
"Medium_Effect" = data.M,
"Smallest_Effect" = data.S)
Vs <- sapply(data, cramer.v)
print(Vs)
##  Largest_Effect   Medium_Effect Smallest_Effect
##     0.314485451     0.020004001     0.005000063

The obtained values for Cramer’s V agree well with our expectation: Cramer’s V is the largest for the first data set and negligible for the second and third data set for which the groups were quite similar. According to the typical interpretation of Cramer’s V, only the first data set exhibited a weak association, while the other two data sets, albeit significant, exhibited no association.

## Conclusions

If you are reporting the significance of a test, it is similarly important to report the effect size because significance alone does not allow conclusions about the magnitude of an effect.. The available measures for the effect size depend on which significance test was used. Here, we have demonstrated Cramer’s V as a measure for the effect size for the $$\chi^2$$ test.

What importance do you assign to effect sizes? Do you think that they are often overlooked? If so, why?

Author: Matthias Döring

There aren't any comments yet. Be the first to comment!


text:
Tracy–Widom distribution

The Tracy–Widom distribution, introduced by Craig Tracy and Harold Widom (1993, 1994), is the probability distribution of the normalized largest eigenvalue of a random Hermitian matrix.[1]

In practical terms, Tracy-Widom is the crossover function between the two phases of weakly versus strongly coupled components in a system.[2] It also appears in the distribution of the length of the longest increasing subsequence of random permutations (Baik, Deift & Johansson 1999), in current fluctuations of the asymmetric simple exclusion process (ASEP) with step initial condition (Johansson 2000, Tracy & Widom 2009), and in simplified mathematical models of the behavior of the longest common subsequence problem on random inputs (Majumdar & Nechaev 2005). See (Takeuchi & Sano 2010, Takeuchi et al. 2011) for experimental testing (and verifying) that the interface fluctuations of a growing droplet (or substrate) are described by the TW distribution $F_2$ (or $F_1$) as predicted by (Prähofer & Spohn 2000).

The distribution F1 is of particular interest in multivariate statistics (Johnstone 2007, 2008, 2009). For a discussion of the universality of Fβ, β = 1, 2, and 4, see Deift (2007). For an application of F1 to inferring population structure from genetic data see Patterson, Price & Reich (2006).

Definition

The Tracy-Widom distribution is defined as the limit:[3]

$F_2(s) = \lim\limits_{n\rightarrow \infty}{\rm Prob}\left((\lambda_{\rm max}-\sqrt{2n})(\sqrt{2})n^{1/6}\leq s\right)$,

The shift by $\sqrt{2n}$ is used to keep the distributions centered at 0. The multiplication by $(\sqrt{2})n^{1/6}$ is used because the standard deviation of the distributions scales as $n^{-1/6}$.

Equivalent formulations

The cumulative distribution function of the Tracy–Widom distribution can be given as the Fredholm determinant

$F_2(s) = \det(I - A_s)\,$

of the operator As on square integrable functions on the half line (s, ∞) with kernel given in terms of Airy functions Ai by

$\frac{\mathrm{Ai}(x)\mathrm{Ai}'(y) - \mathrm{Ai}'(x)\mathrm{Ai}(y)}{x-y}.\,$

It can also be given as an integral

$F_2(s) = \exp\left(-\int_s^\infty (x-s)q^2(x)\,dx\right)$

in terms of a solution of a Painlevé equation of type II

$q^{\prime\prime}(s) = sq(s)+2q(s)^3\,$

where q, called the Hastings–McLeod solution, satisfies the boundary condition

$\displaystyle q(s) \sim \textrm{Ai}(s), s\rightarrow\infty.$

Other Tracy-Widom Distributions

The distribution F2 is associated to unitary ensembles in random matrix theory. There are analogous Tracy–Widom distributions F1 and F4 for orthogonal (β = 1) and symplectic ensembles (β = 4) that are also expressible in terms of the same Painlevé transcendent q (Tracy & Widom 1996):

$F_1(s)=\exp\left(-\frac{1}{2}\int_s^\infty q(x)\,dx\right)\, \left(F_2(s)\right)^{1/2}$

and

$F_4(s/\sqrt{2})=\cosh\left(\frac{1}{2}\int_s^\infty q(x)\, dx\right)\, \left(F_2(s)\right)^{1/2}.$

For an extension of the definition of the Tracy–Widom distributions Fβ to all β > 0 see Ramírez, Rider & Virág (2006).

Numerical approximations

Numerical techniques for obtaining numerical solutions to the Painlevé equations of the types II and V, and numerically evaluating eigenvalue distributions of random matrices in the beta-ensembles were first presented by Edelman & Persson (2005) using MATLAB. These approximation techniques were further analytically justified in Bejan (2005) and used to provide numerical evaluation of Painlevé II and Tracy–Widom distributions (for β=1,2, and 4) in S-PLUS. These distributions have been tabulated in Bejan (2005) to four significant digits for values of the argument in increments of 0.01; a statistical table for p-values was also given in this work. Bornemann (2009) gave accurate and fast algorithms for the numerical evaluation of Fβ and the density functions fβ(s) = dFβ/ds for β = 1, 2, and 4. These algorithms can be used to compute numerically the mean, variance, skewness and kurtosis of the distributions Fβ.

β Mean Variance Skewness Kurtosis
1 −1.2065335745820 1.607781034581 0.29346452408 0.1652429384
2 −1.771086807411 0.8131947928329 0.224084203610 0.0934480876
4 −2.306884893241 0.5177237207726 0.16550949435 0.0491951565

Functions for working with the Tracy–Widom laws are also presented in the R package 'RMTstat' by Johnstone et al. (2009) and MATLAB package 'RMLab' by Dieng (2006).

For a simple approximation based on a shifted gamma distribution see Chiani (2012).

Footnotes

1. ^ Dominici, D. (2008) Special Functions and Orthogonal Polynomials American Math. Soc.
2. ^ Mysterious Statistical Law May Finally Have an Explanation, wired.com 2014-10-27
3. ^ Tracy, C. A.; Widom, H. (1996), "On orthogonal and symplectic matrix ensembles", Communications in Mathematical Physics 177 (3): 727–754, Bibcode:1996CMaPh.177..727T, doi:10.1007/BF02099545, MR 1385083


text:
## Cryptology ePrint Archive: Report 2019/944

Efficient zero-knowledge arguments in the discrete log setting, revisited

Max Hoffmann and Michael Klooß and Andy Rupp

Abstract: Zero-knowledge arguments have become practical, and widely used, especially in the world of Blockchain, for example in Zcash.

This work revisits zero-knowledge proofs in the discrete logarithm setting. First, we identify and carve out basic techniques (partly being used implicitly before) to optimize proofs in this setting. In particular, the linear combination of protocols is a useful tool to obtain zero-knowledge and/or reduce communication. With these techniques, we are able to devise zero-knowledge variants of the logarithmic communication arguments by Bootle et al.\ (EUROCRYPT '16) and Bünz et al. (S\&P '18) thereby introducing almost no overhead. We then construct a conceptually simple commit-and-prove argument for satisfiability of a set of quadratic equations. Unlike previous work, we are not restricted to rank 1 constraint systems (R1CS). This is, to the best of our knowledge, the first work demonstrating that general quadratic constraints, not just R1CS, are a natural relation in the dlog (or ideal linear commitment) setting. This enables new possibilities for optimisation, as, eg., any degree $n^2$ polynomial $f(X)$ can now be evaluated'' with at most $2n$ quadratic constraints.

Our protocols are modular. We easily construct an efficient, logarithmic size shuffle proof, which can be used in electronic voting.

Additionally, we take a closer look at quantitative security measures, eg. the efficiency of an extractor. We formalise short-circuit extraction, which allows us to give tighter bounds on the efficiency of an extractor.

Category / Keywords: cryptographic protocols / zero-knowledge argument, quadratic equations, arithmetic circuit satisfiability, discrete logarithm assumption

Original Publication (with major differences): ACM CCS 2019

Date: received 18 Aug 2019, last revised 21 Nov 2019

Contact author: michael klooss at kit edu

Available format(s): PDF | BibTeX Citation

Note: Full version. Fixed acknowledgements and URL.

Short URL: ia.cr/2019/944

[ Cryptology ePrint archive ]