text:
• ### Bayes Shrinkage at GWAS scale: Convergence and Approximation Theory of a Scalable MCMC Algorithm for the Horseshoe Prior(1705.00841)

Oct. 15, 2018 stat.CO
The horseshoe prior is frequently employed in Bayesian analysis of high-dimensional models, and has been shown to achieve minimax optimal risk properties when the truth is sparse. While optimization-based algorithms for the extremely popular Lasso and elastic net procedures can scale to dimension in the hundreds of thousands, algorithms for the horseshoe that use Markov chain Monte Carlo (MCMC) for computation are limited to problems an order of magnitude smaller. This is due to high computational cost per step and growth of the variance of time-averaging estimators as a function of dimension. We propose two new MCMC algorithms for computation in these models that have improved performance compared to existing alternatives. One of the algorithms also approximates an expensive matrix product to give orders of magnitude speedup in high-dimensional applications. We prove that the exact algorithm is geometrically ergodic, and give guarantees for the accuracy of the approximate algorithm using perturbation theory. Versions of the approximation algorithm that gradually decrease the approximation error as the chain extends are shown to be exact. The scalability of the algorithm is illustrated in simulations with problem size as large as $N=5,000$ observations and $p=50,000$ predictors, and an application to a genome-wide association study with $N=2,267$ and $p=98,385$. The empirical results also show that the new algorithm yields estimates with lower mean squared error, intervals with better coverage, and elucidates features of the posterior that were often missed by previous algorithms in high dimensions, including bimodality of posterior marginals indicating uncertainty about which covariates belong in the model.
• ### Model-free inference on extreme dependence via waiting times(1512.07848)

May 21, 2018 math.PR, math.ST, stat.TH, stat.ME
A variety of methods have been proposed for inference about extreme dependence for multivariate or spatially-indexed stochastic processes and time series. Most of these proceed by first transforming data to some specific extreme value marginal distribution, often the unit Fr\'echet, then fitting a family of max-stable processes to the transformed data and exploring dependence within the framework of that model. The marginal transformation, model selection, and model fitting are all possible sources of misspecification in this approach. We propose an alternative model-free approach, based on the idea that substantial information on the strength of tail dependence and its temporal structure are encoded in the distribution of the waiting times between exceedances of high thresholds at different locations. We propose quantifying the strength of extremal dependence and assessing uncertainty by using statistics based on these waiting times. The method does not rely on any specific underlying model for the process, nor on asymptotic distribution theory. The method is illustrated by applications to climatological, financial, and electrophysiology data. To put the proposed approach within the context of the existing literature, we construct a class of spacetime-indexed stochastic processes whose waiting time distributions are available in closed form by endowing the support points in de Haan's spectral representation of max-stable processes with random birth times, velocities, and lifetimes, and applying Smith's model to these processes. We show that waiting times in this model are stochatically decreasing in mean speed, and the sample mean of the waiting times obeys a central limit theorem with a uniform convergence rate under mild conditions. This indicates that our procedure can be implemented in this setting using standard $t$ statistics and associated hypothesis tests.
• ### Exact Limits of Inference in Coalescent Models(1711.05724)

Jan. 16, 2018 math.ST, stat.TH
Recovery of population size history from molecular sequence data is an important problem in population genetics. Inference commonly relies on a coalescent model linking the population size history to genealogies. The high computational cost of estimating parameters from these models usually compels researchers to select a subset of the available data or to rely on non-sufficient summary statistics for statistical inference. We consider the problem of recovering the true population size history from two possible alternatives on the basis of coalescent time data. We give exact expressions for the probability of selecting the correct alternative in a variety of biologically interesting cases as a function of the separation between the alternative size histories, the number of individuals, loci, and the sampling times. The results are applied to human population history. This work has significant implications for optimal design when the inferential goal is to test scientific hypotheses about population size trajectories in coalescent models with and without recombination.
• ### A Decision Theoretic Approach to A/B Testing(1710.03410)

Oct. 10, 2017 math.ST, stat.TH, stat.ME
A/B testing is ubiquitous within the machine learning and data science operations of internet companies. Generically, the idea is to perform a statistical test of the hypothesis that a new feature is better than the existing platform---for example, it results in higher revenue. If the p value for the test is below some pre-defined threshold---often, 0.05---the new feature is implemented. The difficulty of choosing an appropriate threshold has been noted before, particularly because dependent tests are often done sequentially, leading some to propose control of the false discovery rate (FDR) rather than use of a single, universal threshold. However, it is still necessary to make an arbitrary choice of the level at which to control FDR. Here we suggest a decision-theoretic approach to determining whether to adopt a new feature, which enables automated selection of an appropriate threshold. Our method has the basic ingredients of any decision-theory problem: a loss function, action space, and a notion of optimality, for which we choose Bayes risk. However, the loss function and the action space differ from the typical choices made in the literature, which has focused on the theory of point estimation. We give some basic results for Bayes-optimal thresholding rules for the feature adoption decision, and give some examples using eBay data. The results suggest that the 0.05 p-value threshold may be too conservative in some settings, but that its widespread use may reflect an ad-hoc means of controlling multiplicity in the common case of repeatedly testing variants of an experiment when the threshold is not reached.
• ### Scaling up Data Augmentation MCMC via Calibration(1703.03123)

Sept. 26, 2017 stat.ME
There has been considerable interest in making Bayesian inference more scalable. In big data settings, most literature focuses on reducing the computing time per iteration, with less focused on reducing the number of iterations needed in Markov chain Monte Carlo (MCMC). This article focuses on data augmentation MCMC (DA-MCMC), a widely used technique. DA-MCMC samples tend to become highly autocorrelated in large data samples, due to a miscalibration problem in which conditional posterior distributions given augmented data are too concentrated. This makes it necessary to collect very long MCMC paths to obtain acceptably low MC error. To combat this inefficiency, we propose a family of calibrated data augmentation algorithms, which appropriately adjust the variance of conditional posterior distributions. A Metropolis-Hastings step is used to eliminate bias in the stationary distribution of the resulting sampler. Compared to existing alternatives, this approach can dramatically reduce MC error by reducing autocorrelation and increasing the effective number of DA-MCMC samples per computing time. The approach is simple and applicable to a broad variety of existing data augmentation algorithms, and we focus on three popular models: probit, logistic and Poisson log-linear. Dramatic gains in computational efficiency are shown in applications.
• ### Optimal approximating Markov chains for Bayesian inference(1508.03387)

Aug. 29, 2017 stat.CO, math.ST, stat.TH
The Markov Chain Monte Carlo method is the dominant paradigm for posterior computation in Bayesian analysis. It is common to control computation time by making approximations to the Markov transition kernel. Comparatively little attention has been paid to computational optimality in these approximating Markov Chains, or when such approximations are justified relative to obtaining shorter paths from the exact kernel. We give simple, sharp bounds for uniform approximations of uniformly mixing Markov chains. We then suggest a notion of optimality that incorporates computation time and approximation error, and use our bounds to make generalizations about properties of good approximations in the uniformly mixing setting. The relevance of these properties is demonstrated in applications to a minibatching-based approximate MCMC algorithm for large $n$ logistic regression and low-rank approximations for Gaussian processes.
• ### MCMC for Imbalanced Categorical Data(1605.05798)

June 26, 2017 stat.CO, math.ST, stat.TH, cs.CC
Many modern applications collect highly imbalanced categorical data, with some categories relatively rare. Bayesian hierarchical models combat data sparsity by borrowing information, while also quantifying uncertainty. However, posterior computation presents a fundamental barrier to routine use; a single class of algorithms does not work well in all settings and practitioners waste time trying different types of MCMC approaches. This article was motivated by an application to quantitative advertising in which we encountered extremely poor computational performance for common data augmentation MCMC algorithms but obtained excellent performance for adaptive Metropolis. To obtain a deeper understanding of this behavior, we give strong theory results on computational complexity in an infinitely imbalanced asymptotic regime. Our results show computational complexity of Metropolis is logarithmic in sample size, while data augmentation is polynomial in sample size. The root cause of poor performance of data augmentation is a discrepancy between the rates at which the target density and MCMC step sizes concentrate. In general, MCMC algorithms that have a similar discrepancy will fail in large samples - a result with substantial practical impact.
• ### Coupling and Decoupling to bound an approximating Markov Chain(1706.02040)

June 7, 2017 math.PR
This simple note lays out a few observations which are well known in many ways but may not have been said in quite this way before. The basic idea is that when comparing two different Markov chains it is useful to couple them is such a way that they agree as often as possible. We construct such a coupling and analyze it by a simple dominating chain which registers if the two processes agree or disagree. We find that this imagery is useful when thinking about such problems. We are particularly interested in comparing the invariant measures and long time averages of the processes. However, since the paths agree for long runs, it also provides estimates on various stopping times such as hitting or exit times. We also show that certain bounds are tight. Finally, we provide a simple application to a Markov Chain Monte Carlo algorithm and show numerically that the results of the paper show a good level of approximation at considerable speed up by using an approximating chain rather than the original sampling chain.
• ### Theoretical Limits of Record Linkage and Microclustering(1703.04955)

March 15, 2017 math.ST, stat.TH
There has been substantial recent interest in record linkage, attempting to group the records pertaining to the same entities from a large database lacking unique identifiers. This can be viewed as a type of "microclustering," with few observations per cluster and a very large number of clusters. A variety of methods have been proposed, but there is a lack of literature providing theoretical guarantees on performance. We show that the problem is fundamentally hard from a theoretical perspective, and even in idealized cases, accurate entity resolution is effectively impossible when the number of entities is small relative to the number of records and/or the separation among records from different entities is not extremely large. To characterize the fundamental difficulty, we focus on entity resolution based on multivariate Gaussian mixture models, but our conclusions apply broadly and are supported by simulation studies inspired by human rights applications. These results suggest conservatism in interpretation of the results of record linkage, support collection of additional data to more accurately disambiguate the entities, and motivate a focus on coarser inference. For example, results from a simulation study suggest that sometimes one may obtain accurate results for population size estimation even when fine scale entity resolution is inaccurate.
• ### An algorithm for removing sensitive information: application to race-independent recidivism prediction(1703.04957)

March 15, 2017 stat.AP
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing or augmenting human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the predictions they produce will be equally free from prejudice. There is growing recognition that employing algorithms does not remove the potential for bias, and can even amplify it if the training data were generated by a process that is itself biased. In this paper, we provide a probabilistic notion of algorithmic bias. We propose a method to eliminate bias from predictive models by removing all information regarding protected variables from the data to which the models will ultimately be trained. Unlike previous work in this area, our framework is general enough to accommodate data on any measurement scale. Motivated by models currently in use in the criminal justice system that inform decisions on pre-trial release and parole, we apply our proposed method to a dataset on the criminal histories of individuals at the time of sentencing to produce "race-neutral" predictions of re-arrest. In the process, we demonstrate that a common approach to creating "race-neutral" models-- omitting race as a covariate-- still results in racially disparate predictions. We then demonstrate that the application of our proposed method to these data removes racial disparities from predictions with minimal impact on predictive accuracy.
• ### Estimating the observable population size from biased samples: a new approach to population estimation with capture heterogeneity(1606.02235)

June 7, 2016 stat.ME
Capture-recapture methods aim to estimate the size of a closed population on the basis of multiple incomplete enumerations of individuals. In many applications, the individual probability of being recorded is heterogeneous in the population. Previous studies have suggested that it is not possible to reliably estimate the total population size when capture heterogeneity exists. Here we approach population estimation in the presence of capture heterogeneity as a latent length biased nonparametric density estimation problem on the unit interval. We show that in this setting it is generally impossible to estimate the density on the entire unit interval in finite samples, and that estimators of the population size have high and sometimes unbounded risk when the density has significant mass near zero. As an alternative, we propose estimating the population of individuals with capture probability exceeding some threshold. We provide methods for selecting an appropriate threshold, and show that this approach results in estimators with substantially lower risk than estimators of the total population size, with correspondingly smaller uncertainty, even when the parameter of interest is the total population. The alternative paradigm is demonstrated in extensive simulation studies and an application to snowshoe hare multiple recapture data.
• ### Optimal Gaussian approximations to the posterior for log-linear models with Diaconis-Ylvisaker priors(1511.00764)

Nov. 3, 2015 stat.ME
In contingency table analysis, sparse data is frequently encountered for even modest numbers of variables, resulting in non-existence of maximum likelihood estimates. A common solution is to obtain regularized estimates of the parameters of a log-linear model. Bayesian methods provide a coherent approach to regularization, but are often computationally intensive. Conjugate priors ease computational demands, but the conjugate Diaconis-Ylvisaker priors for the parameters of log-linear models do not give rise to closed form credible regions, complicating posterior inference. Here we derive the optimal Gaussian approximation to the posterior for log-linear models with Diaconis-Ylvisaker priors, and provide convergence rate and finite-sample bounds for the Kullback-Leibler divergence between the exact posterior and the optimal Gaussian approximation. We demonstrate empirically in simulations and a real data application that the approximation is highly accurate, even in relatively small samples. The proposed approximation provides a computationally scalable and principled approach to regularized estimation and approximate Bayesian inference for log-linear models.
• ### Tensor decompositions and sparse log-linear models(1404.0396)

April 1, 2014 math.ST, stat.TH
Contingency table analysis routinely relies on log linear models, with latent structure analysis providing a common alternative. Latent structure models lead to a low rank tensor factorization of the probability mass function for multivariate categorical data, while log linear models achieve dimensionality reduction through sparsity. Little is known about the relationship between these notions of dimensionality reduction in the two paradigms. We derive several results relating the support of a log-linear model to the nonnegative rank of the associated probability tensor. Motivated by these findings, we propose a new collapsed Tucker class of tensor decompositions, which bridge existing PARAFAC and Tucker decompositions, providing a more flexible framework for parsimoniously characterizing multivariate categorical data. Taking a Bayesian approach to inference, we illustrate advantages of the new decompositions in simulations and an application to functional disability data.


text:
How to convert metre to ångström? [closed]

In my book it says

Convert $1.213\cdot10^{-11}$ metre into Angstrom.

The answer is $12.13\cdot10^{-2}$ Angstrom.

How is it possible? I thought of multiplying the metre with $10^{-10}$, but I didn't get the correct answer. The same applies to the conversion of centimetres to Angstrom.

The ångström (symbol: Å) is a widely used non-SI unit of length. Its value in SI units is:

$$1\ \mathring{\mathrm{A}}=0.1\ \mathrm{nm}=100\ \mathrm{pm}=10^{-10}\ \mathrm m$$

Thus, the required conversion factor is given by

$$\frac{1\ \mathring{\mathrm{A}}}{10^{-10}\ \mathrm m}=1$$

or

$$\frac{10^{-10}\ \mathrm m}{1\ \mathring{\mathrm{A}}}=1$$

This conversion factor can be used to express the given length of $1.213\cdot10^{-11}\ \mathrm m$ in terms of the unit ångström:

$$1.213\cdot10^{-11}\ \mathrm m=1.213\cdot10^{-11}\ \mathrm m\cdot\frac{1\ \mathring{\mathrm{A}}}{10^{-10}\ \mathrm m}=0.1213\ \mathring{\mathrm{A}}$$

Note that $0.1213\ \mathring{\mathrm{A}}=12.13\cdot10^{-2}\ \mathring{\mathrm{A}}$, which is the answer given in your book.

Also note that you do not simply divide the given length by $10^{-10}$ since the units are part of the conversion factor. You actually divide the value by $10^{-10}\ \mathrm m/\mathring{\mathrm{A}}$.

You're correct in saying you need to do is multiply by $10^{-10}$. The answer you gave is equivalent to $1.123\cdot10^{-1}$ (just divide by $10$), which is exactly $(1.123\cdot10^{-11}) \cdot (10^{-10})$. If you're using a calculator, make sure you wrap each term in brackets so that the order of operations is correct.

format: plain


text:
# Set Notation: Write set of ordered pairs of odd integers

I am trying to write down the set of all ordered pairs $(n,m)$ such that $n=2k+1:k\in\mathbb{Z}^{+}$ and $m=2l+1:l\in\mathbb{Z}^{+}$. How would that be written in set notation?

I would use $$\left\{(n,m)\in \mathbb{Z}^2 \mid \exists k, l \in \mathbb{Z}^+:n=2k+1,m=2l+1\right\}$$
• @BMehta should the $\exists$ be replaced with $\forall$ if I want all ordered pairs $(n,m)$? – Aaron Hendrickson Feb 23 '17 at 19:57
• @AaronHendrickson No, "all" is implicit in the set notation: $\{ x \mid P(x) \}$ denotes the set of all $x$ such that $P(x)$ holds. In this particular case, you want the set of all $(n,m)$ for which there exist $k$ and $l$ with the desired property. – Uwe Feb 23 '17 at 21:04
One way is: $$\{ (2k+1, 2l+1) \mid k, l \in \mathbb{Z}^+ \}$$
one another way is: $$\{x|\exists y, z\in \Bbb{Z}: x=(y,z) \wedge \exists k, l \in \Bbb{Z}^+: y=2k+1 \wedge z=2l+1\}$$


text:
# Massive MIMO for Maximal Spectral Efficiency: How Many Users and Pilots Should Be Allocated?

Journal Article

### Authors:

Emil Björnson; E G. Larsson; Mérouane Debbah

### Source:

IEEE Transactions on Signal Processing, Volume 15, Issue 2, p.1293-1308 (2016)

### Abstract:

Massive MIMO is a promising technique for increasing the spectral efficiency (SE) of cellular networks, by deploying antenna arrays with hundreds or thousands of active elements at the base stations and performing coherent transceiver processing. A common rule-of-thumb is that these systems should have an order of magnitude more antennas M than scheduled users K because the users’ channels are likely to be near-orthogonal when $M/K > 10$. However, it has not been proved that this ruleof-thumb actually maximizes the SE. In this paper, we analyze how the optimal number of scheduled users K depends on M and other system parameters. To this end, new SE expressions are derived to enable efficient system-level analysis with power control, arbitrary pilot reuse, and random user locations. The value of K in the large-M regime is derived in closed form, while simulations are used to show what happens at finite M, in different interference scenarios, with different pilot reuse factors, and for different processing schemes. Up to half the coherence block should be dedicated to pilots and the optimal M/K is less than 10 in many cases of practical relevance. Interestingly, K depends strongly on the processing scheme and hence it is unfair to compare different schemes using the same K.

Full Text:


text:
# Bosonic operators

[EDITED QUESTION] We have a vector of bosonic operators, such that: $$\begin{equation*} \vec{\phi} = (a, b, c)^{\text{T}} \; , \end{equation*}$$ and the following commutation rules, $$\begin{equation*} \begin{split} [A_{\ell}, A_{\ell^{'}}] &=0 \\ [A^{\dagger}_{\ell}, A^{\dagger}_{\ell^{'}}] &=0 \\ [A_{\ell}, A^{\dagger}_{\ell^{'}}] &=\delta_{\ell,\ell^{'}} \; , \end{split} \end{equation*}$$ for each operator, where $$A_{\ell} = a,b,c$$.

If now we applied a rotation to the vector, such that: $$\begin{equation*} \vec{\phi^{'}} = R^{-1}\vec{\phi} \; , \end{equation*}$$ where:

$$R = \frac{1}{2} \begin{bmatrix} -1 & 1 & \sqrt{2} \\ \sqrt{2} & \sqrt{2} & 0 \\ -1 & 1 & -\sqrt{2} \end{bmatrix}$$ and $$R^{-1} = \frac{1}{2} \begin{bmatrix} -1 & \sqrt{2} & -1 \\ 1 & \sqrt{2} & 1 \\ \sqrt{2} & 0 & -\sqrt{2} \end{bmatrix} = R^{\dagger} \; ,$$

[EDIT: the matrix $$R$$ had a typo and wasn't unitary. Where it reads $$1/2$$, it was $$1/\sqrt{2}$$.]

are the new operators also bosonic? Is this type of linear combination of bosonic operators also a bosonic operator?

[EDIT. Answer: Now that the matrix is unitary, the linear combination of the bosonics operators are also bosonic operators.]

• You wrote down the conditions for operators to be bosonic. Did you try to straightforwardly check if your new operators fulfil those conditions?
– noah
Sep 10 at 13:45
• What is the connection between the ${\boldsymbol \phi}$ and the $A$'s? And why write the inverse of the orthogonal matrix $Ri$ n such a strange way? It's just the transpose of $R$ surely? Sep 10 at 18:56
• The vector $\phi$ is formed by three bosonic operators $a$, $b$ and $c$. They follow these three conditions, where $Aℓ=a,b,c$. The matrix $R$ was given like that, so the operation $RR^{\dagger}=2\mathbb{I}$, that might be the source of the issue here. Thank you.
– koy
Sep 10 at 20:21

As you have written it, $$R$$ is not unitary nor orthogonal, as $$RR^\dagger=R^\dagger R=2\mathbb{I}.$$ Using the proper normalization for $$R$$ (with $$1/2$$ out front instead of $$1/\sqrt{2}$$) will ensure that it is unitary and thus that the resulting operators will also satisfy bosonic commutation relations.


text:
MathSciNet bibliographic data MR2069786 46L05 Raeburn, Iain; Sims, Aidan; Yeend, Trent The \$C\sp *\$$C\sp *$-algebras of finitely aligned higher-rank graphs. J. Funct. Anal. 213 (2004), no. 1, 206–240. Article

For users without a MathSciNet license , Relay Station allows linking from MR numbers in online mathematical literature directly to electronic journals and original articles. Subscribers receive the added value of full MathSciNet reviews.


text:
| Next --- Slide 0 of 46
Back to Lecture Thumbnails
whdawn

It seems that it should be lecture 18 instead of 17

kmcrane

Yep. :-)


text:
Theses Doctoral

# Efficient Simulation Methods for Estimating Risk Measures

Du, Yiping

In this thesis, we analyze the computational problem of estimating financial risk in nested Monte Carlo simulation. An outer simulation is used to generate financial scenarios, and an inner simulation is used to estimate future portfolio values in each scenario. Mean squared error (MSE) for standard nested simulation converges at the rate $k^{-2/3}$, where $k$ is the computational budget.
In the first part of this thesis, we focus on one risk measure, the probability of a large loss, and we propose a new algorithm to estimate this risk. Our algorithm sequentially allocates computational effort in the inner simulation based on marginal changes in the risk estimator in each scenario. Theoretical results are given to show that the risk estimator has an asymptotic MSE of order $k^{-4/5+\epsilon}$, for all positive $\epsilon$, that is faster compared to the conventional uniform inner sampling approach. Numerical results consistent with the theory are presented.
In the second part of this thesis, we introduce a regression-based nested Monte Carlo simulation method for risk estimation. The proposed regression method combines information from different risk factor realizations to provide a better estimate of the portfolio loss function. The MSE of the regression method converges at the rate $k^{-1}$ until reaching an asymptotic bias level which depends on the magnitude of the regression error. Numerical results consistent with our theoretical analysis are provided and numerical comparisons with other methods are also given.
In the third part of this thesis, we propose a method based on weighted regression. Similar to the unweighted regression method, the MSE of the weighted regression method converges at the rate $k^{-1}$ until reaching an asymptotic bias level, which depends on the size of the regression error. However, the weighted approach further reduces MSE by emphasizing scenarios that are more important to the calculation of the risk measure. We find a globally optimal weighting strategy for general risk measures in an idealized setting. For applications, we propose and test a practically implementable two-pass method, where the first pass uses an unweighted regression and the second pass uses weights based on the first pass.


text:
# Kafka connector#

This connector allows the use of Apache Kafka topics as tables in Trino. Each message is presented as a row in Trino.

Topics can be live. Rows appear as data arrives, and disappear as segments get dropped. This can result in strange behavior if accessing the same table multiple times in a single query (e.g., performing a self join).

The connector reads and writes message data from Kafka topics in parallel across workers to achieve a significant performance gain. The size of data sets for this parallelization is configurable and can therefore be adapted to your specific needs.

See the Kafka connector tutorial.

## Requirements#

To connect to Kafka, you need:

• Kafka broker version 0.10.0 or higher.

• Network access from the Trino coordinator and workers to the Kafka nodes. Port 9092 is the default port.

## Configuration#

To configure the Kafka connector, create a catalog properties file etc/catalog/kafka.properties with the following contents, replacing the properties as appropriate.

In some cases, such as when using specialized authentication methods, it is necessary to specify additional Kafka client properties in order to access your Kafka cluster. To do so, add the kafka.config.resources property to reference your Kafka config files. Note that configs can be overwritten if defined explicitly in kafka.properties:

connector.name=kafka
kafka.table-names=table1,table2
kafka.nodes=host1:port,host2:port
kafka.config.resources=/etc/kafka-configuration.properties

### Multiple Kafka clusters#

You can have as many catalogs as you need, so if you have additional Kafka clusters, simply add another properties file to etc/catalog with a different name (making sure it ends in .properties). For example, if you name the property file sales.properties, Trino creates a catalog named sales using the configured connector.

### Log levels#

Kafka consumer logging can be verbose and pollute Trino logs. To lower the log level, simply add the following to etc/log.properties:

org.apache.kafka=WARN

## Configuration properties#

The following configuration properties are available:

Property name

Description

kafka.default-schema

Default schema name for tables

kafka.nodes

List of nodes in the Kafka cluster

kafka.buffer-size

kafka.hide-internal-columns

Controls whether internal columns are part of the table schema or not

kafka.messages-per-split

Number of messages that are processed by each Trino split, defaults to 100000

kafka.timestamp-upper-bound-force-push-down-enabled

Controls if upper bound timestamp push down is enabled for topics using CreateTime mode

kafka.security-protocol

Security protocol for connection to Kafka cluster, defaults to PLAINTEXT

kafka.ssl.keystore.location

Location of the keystore file

kafka.ssl.keystore.password

kafka.ssl.keystore.type

File format of the keystore file, defaults to JKS

kafka.ssl.truststore.location

Location of the truststore file

kafka.ssl.truststore.password

kafka.ssl.truststore.type

File format of the truststore file, defaults to JKS

kafka.ssl.key.password

Password for the private key in the keystore file

kafka.ssl.endpoint-identification-algorithm

Endpoint identification algorithm used by clients to validate server host name, defaults to https

kafka.config.resources

A comma-separated list of Kafka client configuration files. These files must exist on the machines running Trino. Only specify this if absolutely necessary to access Kafka. Example: /etc/kafka-configuration.properties

In addition, you need to configure table schema and schema registry usage with the relevant properties.

### kafka.default-schema#

Defines the schema which contains all tables that were defined without a qualifying schema name.

This property is optional; the default is default.

### kafka.nodes#

A comma separated list of hostname:port pairs for the Kafka data nodes.

This property is required; there is no default and at least one node must be defined.

Note

Trino must still be able to connect to all nodes of the cluster even if only a subset is specified here as segment files may be located only on a specific node.

### kafka.buffer-size#

Size of the internal data buffer for reading data from Kafka. The data buffer must be able to hold at least one message and ideally can hold many messages. There is one data buffer allocated per worker and data node.

This property is optional; the default is 64kb.

### kafka.timestamp-upper-bound-force-push-down-enabled#

The upper bound predicate on _timestamp column is pushed down only for topics using LogAppendTime mode.

For topics using CreateTime mode, upper bound push down must be explicitly allowed via kafka.timestamp-upper-bound-force-push-down-enabled config property or timestamp_upper_bound_force_push_down_enabled session property.

This property is optional; the default is false.

### kafka.hide-internal-columns#

In addition to the data columns defined in a table description file, the connector maintains a number of additional columns for each table. If these columns are hidden, they can still be used in queries but do not show up in DESCRIBE <table-name> or SELECT *.

This property is optional; the default is true.

### kafka.security-protocol#

Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL.

This property is optional; default is PLAINTEXT.

### kafka.ssl.keystore.location#

Location of the keystore file used for connection to Kafka cluster.

This property is optional.

### kafka.ssl.keystore.password#

Password for the keystore file used for connection to Kafka cluster.

This property is optional, but required when kafka.ssl.keystore.location is given.

### kafka.ssl.keystore.type#

File format of the keystore file. Valid values are: JKS, PKCS12.

This property is optional; default is JKS.

### kafka.ssl.truststore.location#

Location of the truststore file used for connection to Kafka cluster.

This property is optional.

### kafka.ssl.truststore.password#

Password for the truststore file used for connection to Kafka cluster.

This property is optional, but required when kafka.ssl.truststore.location is given.

### kafka.ssl.truststore.type#

File format of the truststore file. Valid values are: JKS, PKCS12.

This property is optional; default is JKS.

### kafka.ssl.key.password#

Password for the private key in the keystore file used for connection to Kafka cluster.

This property is optional. This is required for clients only if two-way authentication is configured i.e. ssl.client.auth=required.

### kafka.ssl.endpoint-identification-algorithm#

The endpoint identification algorithm used by clients to validate server host name for connection to Kafka cluster. Kafka uses https as default. Use disabled to disable server host name validation.

This property is optional; default is https.

## Internal columns#

For each defined table, the connector maintains the following columns:

Column name

Type

Description

_partition_id

BIGINT

ID of the Kafka partition which contains this row.

_partition_offset

BIGINT

Offset within the Kafka partition for this row.

_segment_start

BIGINT

Lowest offset in the segment (inclusive) which contains this row. This offset is partition specific.

_segment_end

BIGINT

Highest offset in the segment (exclusive) which contains this row. The offset is partition specific. This is the same value as _segment_start of the next segment (if it exists).

_segment_count

BIGINT

Running count for the current row within the segment. For an uncompacted topic, _segment_start + _segment_count is equal to _partition_offset.

_message_corrupt

BOOLEAN

True if the decoder could not decode the message for this row. When true, data columns mapped from the message should be treated as invalid.

_message

VARCHAR

Message bytes as an UTF-8 encoded string. This is only useful for a text topic.

_message_length

BIGINT

Number of bytes in the message.

_headers

map(VARCHAR, array(VARBINARY))

Headers of the message where values with the same key are grouped as array.

_key_corrupt

BOOLEAN

True if the key decoder could not decode the key for this row. When true, data columns mapped from the key should be treated as invalid.

_key

VARCHAR

Key bytes as an UTF-8 encoded string. This is only useful for textual keys.

_key_length

BIGINT

Number of bytes in the key.

_timestamp

TIMESTAMP

Message timestamp.

For tables without a table definition file, the _key_corrupt and _message_corrupt columns will always be false.

## Table schema and schema registry usage#

The table schema for the messages can be supplied to the connector with a configuration file or a schema registry. It also provides a mechanism for the connector to discover tables.

You have to configure the supplier with the kafka.table-description-supplier property, setting it to FILE or CONFLUENT. Each table description supplier has a separate set of configuration properties.

Refer to the following subsections for more detail. The FILE table description supplier is the default and the value is case insensitive.

### File table description supplier#

In order to use the file based table description supplier, the kafka.table-description-supplier must be set to FILE which is the default.

In addition, you must set kafka.table-names and kafka.table-description-dir as described in the following sections:

#### kafka.table-names#

Comma-separated list of all tables provided by this catalog. A table name can be unqualified (simple name), and is then placed into the default schema (see below), or it can be qualified with a schema name (<schema-name>.<table-name>).

For each table defined here, a table description file (see below) may exist. If no table description file exists, the table name is used as the topic name on Kafka and no data columns are mapped into the table. The table still contains all internal columns (see below).

This property is required; there is no default and at least one table must be defined.

#### kafka.table-description-dir#

References a folder within Trino deployment that holds one or more JSON files (must end with .json) which contain table description files.

This property is optional; the default is etc/kafka.

#### Table definition files#

Kafka maintains topics only as byte messages and leaves it to producers and consumers to define how a message should be interpreted. For Trino, this data must be mapped into columns to allow queries against the data.

Note

For textual topics that contain JSON data, it is entirely possible to not use any table definition files, but instead use the Trino JSON functions and operators to parse the _message column which contains the bytes mapped into an UTF-8 string. This is, however, pretty cumbersome and makes it difficult to write SQL queries. This only works when reading data.

A table definition file consists of a JSON definition for a table. The name of the file can be arbitrary but must end in .json. Place the file in the directory configured with the kafka.table-description-dir property. The table definition file must be accessible from all Trino nodes.

{
"tableName": ...,
"schemaName": ...,
"topicName": ...,
"key": {
"dataFormat": ...,
"fields": [
...
]
},
"message": {
"dataFormat": ...,
"fields": [
...
]
}
}

Field

Required

Type

Description

tableName

required

string

Trino table name defined by this file.

schemaName

optional

string

Schema which will contain the table. If omitted, the default schema name is used.

topicName

required

string

Kafka topic that is mapped.

key

optional

JSON object

Field definitions for data columns mapped to the message key.

message

optional

JSON object

Field definitions for data columns mapped to the message itself.

#### Key and message in Kafka#

Starting with Kafka 0.8, each message in a topic can have an optional key. A table definition file contains sections for both key and message to map the data onto table columns.

Each of the key and message fields in the table definition is a JSON object that must contain two fields:

Field

Required

Type

Description

dataFormat

required

string

Selects the decoder for this group of fields.

fields

required

JSON array

A list of field definitions. Each field definition creates a new column in the Trino table.

Each field definition is a JSON object:

{
"name": ...,
"type": ...,
"dataFormat": ...,
"mapping": ...,
"formatHint": ...,
"hidden": ...,
"comment": ...
}

Field

Required

Type

Description

name

required

string

Name of the column in the Trino table.

type

required

string

Trino type of the column.

dataFormat

optional

string

Selects the column decoder for this field. Defaults to the default decoder for this row data format and column type.

dataSchema

optional

string

The path or URL where the Avro schema resides. Used only for Avro decoder.

mapping

optional

string

Mapping information for the column. This is decoder specific, see below.

formatHint

optional

string

Sets a column specific format hint to the column decoder.

hidden

optional

boolean

Hides the column from DESCRIBE <table name> and SELECT *. Defaults to false.

comment

optional

string

Adds a column comment which is shown with DESCRIBE <table name>.

There is no limit on field descriptions for either key or message.

### Confluent table description supplier#

The Confluent table description supplier uses the Confluent Schema Registry to discover table definitions. It is only tested to work with the Confluent Schema Registry.

The benefits of using the Confluent table description supplier over the file table description supplier are:

• New tables can be defined without a cluster restart.

• Schema updates are detected automatically.

• There is no need to define tables manually.

Set kafka.table-description-supplier to CONFLUENT to use the schema registry and configure the additional properties in the following table:

Note

Inserts are not supported and the only data format supported is AVRO.

Confluent table description supplier properties#

Property name

Description

Default value

kafka.confluent-schema-registry-url

Comma-separated list of URL addresses for the Confluent schema registry. For example, http://schema-registry-1.example.org:8081,http://schema-registry-2.example.org:8081

kafka.confluent-schema-registry-client-cache-size

The maximum number of subjects that can be stored in the local cache. The cache stores the schemas locally by subjectId, and is provided by the Confluent CachingSchemaRegistry client.

1000

kafka.empty-field-strategy

Avro allows empty struct fields but this is not allowed in Trino. There are three strategies for handling empty struct fields:

• IGNORE, ignore structs with no fields. This propagate to parents. For example an array of structs with no fields is ignored.

• FAIL, fail the query if a struct with no fields is defined.

• DUMMY, add a dummy boolean field called dummy which is null. This may be desired if the struct represents a marker field.

IGNORE

kafka.confluent-subjects-cache-refresh-interval

The interval used for refreshing the list of subjects and the definition of the schema for the subject in the subjects cache

1s

#### Confluent subject to table name mapping#

The subject naming strategy determines how a subject is resolved from the table name.

The default strategy is the TopicNameStrategy where the key subject is defined as <topic-name>-key and the value subject is defined as <topic-name>-value. If other strategies are used there is no way to determine the subject name beforehand so it must be specified manually in the table name.

To manually specify the key and value subjects just append to the table name, for example: <topic name>&key-subject=<key subject>&value-subject=<value subject. Both the key-subject and value-subject parameters are optional. If either is not specified then the default TopicNameStrategy is used to resolve the subject name via the topic name. Note that a case insensitive match must be done, as identifiers cannot contain upper case characters.

## SQL support#

In addition to the globally available and read operation statements, the connector supports the following features:

## Kafka inserts#

The Kafka connector supports the use of INSERT statements to write data to a Kafka topic. Table column data is mapped to Kafka messages as defined in the table definition file. There are four supported data formats for key and message encoding:

These data formats each have an encoder that maps column values into bytes to be sent to a Kafka topic.

Trino supports at-least-once delivery for Kafka producers. This means that messages are guaranteed to be sent to Kafka topics at least once. If a producer acknowledgement times out or if the producer receives an error, it might retry sending the message. This could result in a duplicate message being sent to the Kafka topic.

The Kafka connector does not allow the user to define which partition will be used as the target for a message. If a message includes a key, the producer will use a hash algorithm to choose the target partition for the message. The same key will always be assigned the same partition.

## Row encoding#

Encoding is required to allow writing data and defines how table columns in Trino map to Kafka keys and message data.

The Kafka connector contains the following encoders:

• raw encoder - Table columns are mapped to a Kafka message as raw bytes

• CSV encoder - Kafka message is formatted as comma separated value

• JSON encoder - Table columns are mapped to JSON fields

• Avro encoder - Table columns are mapped to Avro fields based on an Avro schema

Note

A table definition file must be defined for the encoder to work.

### Raw encoder#

The raw encoder formats the table columns as raw bytes using the mapping information specified in the table definition file.

The following field attributes are supported:

• dataFormat - Specifies the width of the column data type

• type - Trino data type

• mapping - start and optional end position of bytes to convert (specified as start or start:end)

The dataFormat attribute selects the number of bytes converted. If absent, BYTE is assumed. All values are signed.

Supported values:

• BYTE - one byte

• SHORT - two bytes (big-endian)

• INT - four bytes (big-endian)

• LONG - eight bytes (big-endian)

• FLOAT - four bytes (IEEE 754 format, big-endian)

• DOUBLE - eight bytes (IEEE 754 format, big-endian)

The type attribute defines the Trino data type.

Different values of dataFormat are supported, depending on the Trino data type:

Trino data type

dataFormat values

BIGINT

BYTE, SHORT, INT, LONG

INTEGER

BYTE, SHORT, INT

SMALLINT

BYTE, SHORT

TINYINT

BYTE

REAL

FLOAT

DOUBLE

FLOAT, DOUBLE

BOOLEAN

BYTE, SHORT, INT, LONG

VARCHAR / VARCHAR(x)

BYTE

The mapping attribute specifies the range of bytes in a key or message used for encoding.

Note

Both a start and end position must be defined for VARCHAR types. Otherwise, there is no way to know how many bytes the message contains. The raw format mapping information is static and cannot be dynamically changed to fit the variable width of some Trino data types.

If only a start position is given:

• For fixed width types, the appropriate number of bytes are used for the specified dataFormat (see above).

If both a start and end position are given, then:

• For fixed width types, the size must be equal to number of bytes used by specified dataFormat.

• All bytes between start (inclusive) and end (exclusive) are used.

Note

All mappings must include a start position for encoding to work.

The encoding for numeric data types (BIGINT, INTEGER, SMALLINT, TINYINT, REAL, DOUBLE) is straightforward. All numeric types use big-endian. Floating point types use IEEE 754 format.

Example raw field definition in a table definition file for a Kafka message:

{
"tableName": "your-table-name",
"schemaName": "your-schema-name",
"topicName": "your-topic-name",
"key": { "..." },
"message": {
"dataFormat": "raw",
"fields": [
{
"name": "field1",
"type": "BIGINT",
"dataFormat": "LONG",
"mapping": "0"
},
{
"name": "field2",
"type": "INTEGER",
"dataFormat": "INT",
"mapping": "8"
},
{
"name": "field3",
"type": "SMALLINT",
"dataFormat": "LONG",
"mapping": "12"
},
{
"name": "field4",
"type": "VARCHAR(6)",
"dataFormat": "BYTE",
"mapping": "20:26"
}
]
}
}

Columns should be defined in the same order they are mapped. There can be no gaps or overlaps between column mappings. The width of the column as defined by the column mapping must be equivalent to the width of the dataFormat for all types except for variable width types.

Example insert query for the above table definition:

INSERT INTO example_raw_table (field1, field2, field3, field4)
VALUES (123456789, 123456, 1234, 'abcdef');

When inserting variable width types, the value must be exactly equal to the width defined in the table definition file. If the inserted value is longer it gets truncated, resulting in data loss. If the inserted value is shorter the decoder will not be able to properly read the value because there is no defined padding character. Due to these constraints the encoder fails if the width of the inserted value is not equal to the mapping width.

### CSV encoder#

The CSV encoder formats the values for each row as a line of comma-separated-values (CSV) using UTF-8 encoding. The CSV line is formatted with a comma ‘,’ as the column delimiter.

The type and mapping attributes must be defined for each field:

• type - Trino data type

• mapping - The integer index of the column in the CSV line (the first column is 0, the second is 1, and so on)

dataFormat and formatHint are not supported and must be omitted.

The following Trino data types are supported by the CSV encoder:

• BIGINT

• INTEGER

• SMALLINT

• TINYINT

• DOUBLE

• REAL

• BOOLEAN

• VARCHAR / VARCHAR(x)

Column values are converted to strings before they are formatted as a CSV line.

Example CSV field definition in a table definition file for a Kafka message:

{
"tableName": "your-table-name",
"schemaName": "your-schema-name",
"topicName": "your-topic-name",
"key": { "..." },
"message": {
"dataFormat": "csv",
"fields": [
{
"name": "field1",
"type": "BIGINT",
"mapping": "0"
},
{
"name": "field2",
"type": "VARCHAR",
"mapping": "1"
},
{
"name": "field3",
"type": "BOOLEAN",
"mapping": "2"
}
]
}
}

Example insert query for the above table definition:

INSERT INTO example_csv_table (field1, field2, field3)
VALUES (123456789, 'example text', TRUE);

### JSON encoder#

The JSON encoder maps table columns to JSON fields defined in the table definition file according to RFC 4627.

For fields, the following attributes are supported:

• type - Trino type of column.

• mapping - slash-separated list of field names to select a field from the JSON object

• dataFormat - name of formatter (required for temporal types)

• formatHint - pattern to format temporal data (only use with custom-date-time formatter)

The following Trino data types are supported by the JSON encoder:

Trino data types

BIGINT

INTEGER

SMALLINT

TINYINT

DOUBLE

REAL

BOOLEAN

VARCHAR

DATE

TIME

TIME WITH TIME ZONE

TIMESTAMP

TIMESTAMP WITH TIME ZONE

The following dataFormats are available for temporal data:

• iso8601

• rfc2822

• custom-date-time - formats temporal data according to Joda Time pattern given by formatHint field

• milliseconds-since-epoch

• seconds-since-epoch

All temporal data in Kafka supports milliseconds precision

The following table defines which temporal data types are supported by dataFormats:

Trino data type

Decoding rules

DATE

custom-date-time, iso8601

TIME

custom-date-time, iso8601, milliseconds-since-epoch, seconds-since-epoch

TIME WITH TIME ZONE

custom-date-time, iso8601

TIMESTAMP

custom-date-time, iso8601, rfc2822, milliseconds-since-epoch, seconds-since-epoch

TIMESTAMP WITH TIME ZONE

custom-date-time, iso8601, rfc2822, milliseconds-since-epoch, seconds-since-epoch

Example JSON field definition in a table definition file for a Kafka message:

{
"tableName": "your-table-name",
"schemaName": "your-schema-name",
"topicName": "your-topic-name",
"key": { "..." },
"message": {
"dataFormat": "json",
"fields": [
{
"name": "field1",
"type": "BIGINT",
"mapping": "field1"
},
{
"name": "field2",
"type": "VARCHAR",
"mapping": "field2"
},
{
"name": "field3",
"type": "TIMESTAMP",
"dataFormat": "custom-date-time",
"formatHint": "yyyy-dd-MM HH:mm:ss.SSS",
"mapping": "field3"
}
]
}
}

Example insert query for the above table definition:

INSERT INTO example_json_table (field1, field2, field3)
VALUES (123456789, 'example text', TIMESTAMP '2020-07-15 01:02:03.456');

### Avro encoder#

The Avro encoder serializes rows to Avro records as defined by the Avro schema. Trino does not support schema-less Avro encoding.

Note

The Avro schema is encoded with the table column values in each Kafka message

The dataSchema must be defined in the table definition file to use the Avro encoder. It points to the location of the Avro schema file for the key or message

Avro schema files can be retrieved via HTTP or HTTPS from remote server with the syntax:

"dataSchema": "http://example.org/schema/avro_data.avsc"

Local files need to be available on all Trino nodes and use an absolute path in the syntax, for example:

"dataSchema": "/usr/local/schema/avro_data.avsc"

The following field attributes are supported:

• name - Name of the column in the Trino table.

• type - Trino type of column.

• mapping - slash-separated list of field names to select a field from the Avro schema. If the field specified in mapping does not exist in the original Avro schema, then a write operation fails.

The following table lists supported Trino types, which can be used in type for the equivalent Avro field type.

Trino data type

Avro data type

BIGINT

INT, LONG

REAL

FLOAT

DOUBLE

FLOAT, DOUBLE

BOOLEAN

BOOLEAN

VARCHAR / VARCHAR(x)

STRING

Example Avro field definition in a table definition file for a Kafka message:

{
"tableName": "your-table-name",
"schemaName": "your-schema-name",
"topicName": "your-topic-name",
"key": { "..." },
"message":
{
"dataFormat": "avro",
"dataSchema": "/avro_message_schema.avsc",
"fields":
[
{
"name": "field1",
"type": "BIGINT",
"mapping": "field1"
},
{
"name": "field2",
"type": "VARCHAR",
"mapping": "field2"
},
{
"name": "field3",
"type": "BOOLEAN",
"mapping": "field3"
}
]
}
}

Example Avro schema definition for the above table definition:

{
"type" : "record",
"name" : "example_avro_message",
"namespace" : "io.trino.plugin.kafka",
"fields" :
[
{
"name":"field1",
"type":["null", "long"],
"default": null
},
{
"name": "field2",
"type":["null", "string"],
"default": null
},
{
"name":"field3",
"type":["null", "boolean"],
"default": null
}
],
"doc:" : "A basic avro schema"
}

Example insert query for the above table definition:

INSERT INTO example_avro_table (field1, field2, field3)
VALUES (123456789, 'example text', FALSE);

## Row decoding#

For key and message, a decoder is used to map message and key data onto table columns.

The Kafka connector contains the following decoders:

• raw - Kafka message is not interpreted, ranges of raw message bytes are mapped to table columns

• csv - Kafka message is interpreted as comma separated message, and fields are mapped to table columns

• json - Kafka message is parsed as JSON and JSON fields are mapped to table columns

• avro - Kafka message is parsed based on an Avro schema and Avro fields are mapped to table columns

Note

If no table definition file exists for a table, the dummy decoder is used, which does not expose any columns.

### raw decoder#

The raw decoder supports reading of raw (byte-based) values from Kafka message or key and converting it into Trino columns.

For fields, the following attributes are supported:

• dataFormat - selects the width of the data type converted

• type - Trino data type (see table below for list of supported data types)

• mapping - <start>[:<end>]; start and end position of bytes to convert (optional)

The dataFormat attribute selects the number of bytes converted. If absent, BYTE is assumed. All values are signed.

Supported values are:

• BYTE - one byte

• SHORT - two bytes (big-endian)

• INT - four bytes (big-endian)

• LONG - eight bytes (big-endian)

• FLOAT - four bytes (IEEE 754 format)

• DOUBLE - eight bytes (IEEE 754 format)

The type attribute defines the Trino data type on which the value is mapped.

Depending on Trino type assigned to column different values of dataFormat can be used:

Trino data type

Allowed dataFormat values

BIGINT

BYTE, SHORT, INT, LONG

INTEGER

BYTE, SHORT, INT

SMALLINT

BYTE, SHORT

TINYINT

BYTE

DOUBLE

DOUBLE, FLOAT

BOOLEAN

BYTE, SHORT, INT, LONG

VARCHAR / VARCHAR(x)

BYTE

The mapping attribute specifies the range of the bytes in a key or message used for decoding. It can be one or two numbers separated by a colon (<start>[:<end>]).

If only a start position is given:

• For fixed width types the column will use the appropriate number of bytes for the specified dataFormat (see above).

• When VARCHAR value is decoded all bytes from start position till the end of the message will be used.

If start and end position are given, then:

• For fixed width types the size must be equal to number of bytes used by specified dataFormat.

• For VARCHAR all bytes between start (inclusive) and end (exclusive) are used.

If no mapping attribute is specified, it is equivalent to setting start position to 0 and leaving end position undefined.

Decoding scheme of numeric data types (BIGINT, INTEGER, SMALLINT, TINYINT, DOUBLE) is straightforward. A sequence of bytes is read from input message and decoded according to either:

• big-endian encoding (for integer types)

• IEEE 754 format for (for DOUBLE).

Length of decoded byte sequence is implied by the dataFormat.

For VARCHAR data type a sequence of bytes is interpreted according to UTF-8 encoding.

### csv decoder#

The CSV decoder converts the bytes representing a message or key into a string using UTF-8 encoding and then interprets the result as a CSV (comma-separated value) line.

For fields, the type and mapping attributes must be defined:

• type - Trino data type (see table below for list of supported data types)

• mapping - the index of the field in the CSV record

dataFormat and formatHint are not supported and must be omitted.

Table below lists supported Trino types, which can be used in type and decoding scheme:

Trino data type

Decoding rules

BIGINT
INTEGER
SMALLINT
TINYINT

Decoded using Java Long.parseLong()

DOUBLE

Decoded using Java Double.parseDouble()

BOOLEAN

“true” character sequence maps to true; Other character sequences map to false

VARCHAR / VARCHAR(x)

Used as is

### json decoder#

The JSON decoder converts the bytes representing a message or key into a JSON according to RFC 4627. Note that the message or key MUST convert into a JSON object, not an array or simple type.

For fields, the following attributes are supported:

• type - Trino type of column.

• dataFormat - Field decoder to be used for column.

• mapping - slash-separated list of field names to select a field from the JSON object

• formatHint - only for custom-date-time, see below

The JSON decoder supports multiple field decoders, with _default being used for standard table columns and a number of decoders for date and time based types.

The table below lists Trino data types, which can be used as in type, and matching field decoders, which can be specified via dataFormat attribute.

Trino data type

Allowed dataFormat values

BIGINT
INTEGER
SMALLINT
TINYINT
DOUBLE
BOOLEAN
VARCHAR
VARCHAR(x)

Default field decoder (omitted dataFormat attribute)

DATE

custom-date-time, iso8601

TIME

custom-date-time, iso8601, milliseconds-since-epoch, seconds-since-epoch

TIME WITH TIME ZONE

custom-date-time, iso8601

TIMESTAMP

custom-date-time, iso8601, rfc2822, milliseconds-since-epoch, seconds-since-epoch

TIMESTAMP WITH TIME ZONE

custom-date-time, iso8601, rfc2822, milliseconds-since-epoch seconds-since-epoch

### Default field decoder#

This is the standard field decoder, supporting all the Trino physical data types. A field value is transformed under JSON conversion rules into boolean, long, double or string values. For non-date/time based columns, this decoder should be used.

### Date and time decoders#

To convert values from JSON objects into Trino DATE, TIME, TIME WITH TIME ZONE, TIMESTAMP or TIMESTAMP WITH TIME ZONE columns, special decoders must be selected using the dataFormat attribute of a field definition.

• iso8601 - text based, parses a text field as an ISO 8601 timestamp.

• rfc2822 - text based, parses a text field as an RFC 2822 timestamp.

• custom-date-time - text based, parses a text field according to Joda format pattern

specified via formatHint attribute. Format pattern should conform to https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html.

• milliseconds-since-epoch - number based, interprets a text or number as number of milliseconds since the epoch.

• seconds-since-epoch - number based, interprets a text or number as number of milliseconds since the epoch.

For TIMESTAMP WITH TIME ZONE and TIME WITH TIME ZONE data types, if timezone information is present in decoded value, it will be used in Trino value. Otherwise result time zone will be set to UTC.

### avro decoder#

The Avro decoder converts the bytes representing a message or key in Avro format based on a schema. The message must have the Avro schema embedded. Trino does not support schema-less Avro decoding.

For key/message, using avro decoder, the dataSchema must be defined. This should point to the location of a valid Avro schema file of the message which needs to be decoded. This location can be a remote web server (e.g.: dataSchema: 'http://example.org/schema/avro_data.avsc') or local file system(e.g.: dataSchema: '/usr/local/schema/avro_data.avsc'). The decoder fails if this location is not accessible from the Trino coordinator node.

For fields, the following attributes are supported:

• name - Name of the column in the Trino table.

• type - Trino type of column.

• mapping - slash-separated list of field names to select a field from the Avro schema. If field specified in mapping does not exist in the original Avro schema then a read operation returns NULL.

Table below lists supported Trino types which can be used in type for the equivalent Avro field type/s.

Trino data type

Allowed Avro data type

BIGINT

INT, LONG

DOUBLE

DOUBLE, FLOAT

BOOLEAN

BOOLEAN

VARCHAR / VARCHAR(x)

STRING

VARBINARY

FIXED, BYTES

ARRAY

ARRAY

MAP

MAP

#### Avro schema evolution#

The Avro decoder supports schema evolution feature with backward compatibility. With backward compatibility, a newer schema can be used to read Avro data created with an older schema. Any change in the Avro schema must also be reflected in Trino’s topic definition file. Newly added/renamed fields must have a default value in the Avro schema file.

The schema evolution behavior is as follows:

• Column added in new schema: Data created with an older schema produces a default value, when the table is using the new schema.

• Column removed in new schema: Data created with an older schema no longer outputs the data from the column that was removed.

• Column is renamed in the new schema: This is equivalent to removing the column and adding a new one, and data created with an older schema produces a default value when table is using the new schema.

• Changing type of column in the new schema: If the type coercion is supported by Avro, then the conversion happens. An error is thrown for incompatible types.


text:
# Homework Help: Reaction and friction coefficient on a cylinder

1. Jan 28, 2015

### Karol

1. The problem statement, all variables and given/known data
The cylinder weighs 200[N] and lies on a rough surface. a rope holds it in place.
What's the tension T in the rope and what's the minimal coefficient of friction.

2. Relevant equations
Friction: f=μN

3. The attempt at a solution
All the geometry i have drawn. T balances the torque made by the weight of the cylinder. i calculate torque round the point of contact with the inclined surface:
The distance d from the line of action of T:
$$d=\frac{R}{\tan30^0}\cdot \sin 60^0$$
Now the torque from the weight equals the torque from T:
$$200R\cdot \sin 30^0=T\cdot d\rightarrow T=66.7$$
It should be 654 and i don't continue yet to the friction coefficient.
Edit: i found my mistake, in newtons it comes out right. now i will try to find the coefficient of friction

#### Attached Files:

• ###### Snap1.jpg
File size:
25.7 KB
Views:
108
Last edited: Jan 28, 2015
2. Jan 28, 2015

### Staff: Mentor

I get 654 N if the cylinder has a mass of 200 kg instead of a weight of 200N.


text:
# Finite groups with seminormal or abnormal Sylow subgroups

Document Type: Research Paper

Authors

1 Francisk Skorina Gomel State University, Department of Mathematics, Sovetskaya str., 104, Gomel, 246019, Republic of Belarus

2 Physic and Mathematic Department, Brest State A.S. Pushkin University, Brest, Belarus

Abstract

Let $G$ be a finite group in which every Sylow subgroup is seminormal or abnormal. We prove that $G$ has a Sylow tower. We establish that if a group has a maximal subgroup with Sylow subgroups under the same conditions, then this group is soluble.

Keywords

Main Subjects


text:
# Excision theorem

Excision theorem In algebraic topology, ein Zweig der Mathematik, the excision theorem is a theorem about relative homology and one of the Eilenberg–Steenrod axioms. Given a topological space {Anzeigestil X} and subspaces {Anzeigestil A} und {Anzeigestil U} so dass {Anzeigestil U} is also a subspace of {Anzeigestil A} , the theorem says that under certain circumstances, we can cut out (excise) {Anzeigestil U} from both spaces such that the relative homologies of the pairs {Anzeigestil (Xsetminus U,Asetminus U)} hinein {Anzeigestil (X,EIN)} sind isomorph.

This assists in computation of singular homology groups, as sometimes after excising an appropriately chosen subspace we obtain something easier to compute.

Inhalt 1 Satz 1.1 Aussage 1.2 Proof Sketch 2 Anwendungen 2.1 Eilenberg–Steenrod Axioms 2.2 Mayer-Vietoris Sequences 3 Siehe auch 4 Verweise 5 Bibliography Theorem Statement If {displaystyle Usubseteq Asubseteq X} are as above, das sagen wir {Anzeigestil U} can be excised if the inclusion map of the pair {Anzeigestil (Xsetminus U,Asetminus U)} hinein {Anzeigestil (X,EIN)} induces an isomorphism on the relative homologies: {Anzeigestil H_{n}(Xsetminus U,Asetminus U)cong H_{n}(X,EIN)} The theorem states that if the closure of {Anzeigestil U} is contained in the interior of {Anzeigestil A} , dann {Anzeigestil U} can be excised.

Often, subspaces that do not satisfy this containment criterion still can be excised—it suffices to be able to find a deformation retract of the subspaces onto subspaces that do satisfy it.

Proof Sketch The proof of the excision theorem is quite intuitive, though the details are rather involved. The idea is to subdivide the simplices in a relative cycle in {Anzeigestil (X,EIN)} to get another chain consisting of "kleiner" einfach, and continuing the process until each simplex in the chain lies entirely in the interior of {Anzeigestil A} or the interior of {displaystyle Xsetminus U} . Since these form an open cover for {Anzeigestil X} and simplices are compact, we can eventually do this in a finite number of steps. This process leaves the original homology class of the chain unchanged (this says the subdivision operator is chain homotopic to the identity map on homology). In the relative homology {Anzeigestil H_{n}(X,EIN)} , dann, this says all the terms contained entirely in the interior of {Anzeigestil U} can be dropped without affecting the homology class of the cycle. This allows us to show that the inclusion map is an isomorphism, as each relative cycle is equivalent to one that avoids {Anzeigestil U} entirely.

Applications Eilenberg–Steenrod Axioms The excision theorem is taken to be one of the Eilenberg–Steenrod Axioms.

Mayer-Vietoris Sequences The Mayer–Vietoris sequence may be derived with a combination of excision theorem and the long-exact sequence.[1] See also Homotopy excision theorem References ^ See Hatcher 2002, p.149, for example Bibliography Joseph J. Rotmann, Eine Einführung in die algebraische Topologie, Springer-Verlag, ISBN 0-387-96678-1 Allen Hatcher, Algebraic Topology. Cambridge University Press, Cambridge, 2002. Kategorien: Homology theoryTheorems in topology

Wenn Sie andere ähnliche Artikel wissen möchten Excision theorem Sie können die Kategorie besuchen Homology theory.

Geh hinauf

Wir verwenden eigene Cookies und Cookies von Drittanbietern, um die Benutzererfahrung zu verbessern Mehr Informationen