Q:

What should the criteria be for Stack Exchange sites to leave beta?

Today SE removed the "beta" label from 29 established sites,
recognizing a need that has been plaguing many of our sites for years. Until today, under SE's rules there were
two states: "graduated" if a site meets the requirements and "beta" otherwise. One of those requirements is to
sustain an average of ten questions per day, which just doesn't work for some of our smaller sites.  High-quality
sites have thus been branded with the "beta" label for, in some cases, almost a decade, long after there were
questions about those sites' quality, scope, and ability to maintain their communities.
All that changed today -- yay!
Well, it started to change.  In the announcement, Catija says that this is a start and SE is still working out
what the rules and thresholds ought to be. Basically, we don't know what the criteria are, but by any reasonable
criteria these 29 sites qualify, so let's start with these ones. We've been wanting to break up with 
graduation for a while, moving from this binary system to
one where milestones happen when they make sense, but nailing down the specifics is hard.
I've been one of the most vocal people about how the "beta" system is broken, so the least I can do is try to
help fix it.  In this question, let's focus on what the criteria should be for removing the "beta" label from a site.
(We can discuss other milestones, like when to raise privilege levels or when a site gets community ads, separately.)
When a site is brand new it's clearly a beta.  The community has to work out through actual use what the scope is,
what requirements to apply to questions or answers, and so on.  It has to build a community of people working together,
on meta and on the main site, to classify, improve, and curate content.  It has to figure out what it wants to be.  As
a community grows, as the body of Q&A grows, and as time passes, what the site is starts to firm up.  Eventually you
have very-old sites clamoring to advance.  It would be good to have
some heuristics to indicate both to a community and to SE that a site is approaching, or has reached, the point
where the "beta" label should come off.
What are those heuristics?  When is a site no longer beta?
In a post on Writing Meta a year and a half ago I outlined
some things that I felt justified our advancement.  My list was subjective (there's nary a number in the whole lot!),
but here are some factors to consider:

a sizable body of questions, with old ones still getting new answers, and a steady flow of new questions (how do we recognize this? what's "sizable"?)
content quality (measured how?)
engaged users who not only ask and answer but also guide new users, review, curate, and use their moderation privileges
an active meta
enough users at various reputation levels to support higher privilege levels
enough active users to support moderator elections
site longevity: seven years is too long, seven weeks is too short -- when has a site stood the test of time?

What factors should communities and SE be looking at?  How do we know when a site is "baked"?

A:

Avid Users
We need a better idea of how many avid users a site has.
Right now Bob who answered one question 5 years ago that hit the Hot Questions list and got 200 reputation from it before disappearing off the face of the earth is counted as an avid user.
Users who show up and write one awesome answer and then leave are great but they really aren't involved in the day to day moderation such as editing/voting/closing etc.
I think that a better definition of Avid Users would include,

Interacting with the site in some way commenting/asking/answering/voting/flagging/reviewing etc.
Within a recent amount of time say last 30 days for example.

Right now the number of Avid Users only goes up with time, this way it could go down as well if users stop showing up and participating. Letting it fluctuate with the number of people who are participating in a recent amount of time would give us all a much better idea of how the site is doing.

A:

This question is basically asking something much more abstract: what's the purpose of beta? That is, why do we need this beta state to live between Area 51 idea and a "non-beta" site?
This question matters even more now, since the "beta" designation is now decoupled from most of the things we presently recognize as being consequences of being in beta: appointed moderators, lower rep thresholds, etc.
As I understood it, the thinking was that beta was about doing the following:

Firmly establishing the scope and domain of the site. What is on/off topic, what do good/poor questions look like, how much subjectivity do we allow, etc. This should ultimately result in a relatively clear set of enforceable rules, typically on the site's Meta.
Proving that the community is functional and growing. This includes having enough questions being asked that new answerers have a place, a body of experts that has room to grow, maintenance tasks being performed accurately, etc.

The recent release of beta sites seems to be designating "beta" is being primarily or solely about the first one. Most of those sites have clearly figured out what they're about, but many of them function at a lower level in various ways than most previous sites functioned at upon graduation.
So, we could essentially decide that beta is mainly about scope, with the matter of the functioning of the site having a much lower bar.
Scope definition is easy to determine. The site needs to have had enough questions, both open and closed, to allow for a large enough sample size to know what the site is about. There must be meta discussions about scope, with sufficient participation to have some idea about what the general consensus is.
That brings us to the question of how we define "functional". There are a number of criteria which could be objectively measured:

User growth is an important aspect to any site. A site which isn't growing its base of active users is set for slow decline.
So we should determine the number of users per-month who transition into being active users of the site, relative to the number of total active users. "Active user" of course is something that would have to be nailed down, but visiting the site regularly (every few days) and making some number of posts within that monthly timespan would be a good start.
One characteristic of smaller sites (or just overly narrow topics) is that there are a small cabal of answers who are involved in almost every question. This would be fine for a single tag, but on a whole site, it is indicative of an insular, isolated community of a small number of experts feeding on a drip-feed of questions. It also makes it difficult for new experts to appear, since if all the questions are handled by the existing experts, there's nothing left to do.
To detect this, we could find take the reputation of the answerer for each answer, then compute both the average for a week and the median. Note that this counts on the basis of the answer, so one high rep users answering 3 questions would count their rep 3 times. The lower the median is to the average, the less likely it is that a small cabal of users are dominating the site.
Community question closure. Elected moderators are exception handlers; in a well-functioning site, questions get closed reasonably quickly by regular users performing close votes. If moderators have to spend time patrolling the site just to make sure questions are being closed expeditiously, that's a sign that the site lacks sufficient engaged users willing to perform "curation" tasks.
One could measure the ratio of questions closed by moderators vs. those closed by the community. The focus of the measurement should be posts that only get 1 or 2 non-moderator close votes, relative to the number of closed questions.
Community consensus on closure. A well-functioning site is one where the community is pretty clear on what is an appropriate question and what is not, and they will vote accordingly. So finding a way to measure this consensus would be useful.
One objective measurement might be looking at how many close votes age away over a particular span of time, relative to the number of closed questions. Of course, the number of votes should also be taken into account; a spurious single close vote shouldn't count as much as 3 or 4 close votes.

A:

A while ago graduation was based on a fairly thorough manual review. Back then I once checked a bunch of graduated sites, compared them to older sites that appeared to be stuck in beta and tried to figure some patterns that could make reviewers feel that particular site is good enough to graduate.
The only thing I saw in common in these manually approved sites was that all of them seemed to have "question bank" above 3K (total questions asked excluding any that have been deleted). Upon thinking about this a bit I decided that this could make a fairly solid indication that topic covered by the site is good enough to provide an extensive, reliable knowledge base, meaning it's reasonably safe to graduate.
I would prefer that our criteria for leaving beta would somehow take this metric into account, because it appears to be sort of "verified" by past manual reviews.

In case if we would want to get sites out of beta automatically, I would probably prefer a bit stricter criteria that additionally accounts for auto-delete features of the system, something like requiring above 3K questions that are older than, say, one year (better one and half, to let 365-days cleanup scripts make reasonable impact).
In this case another possible additional criteria could be some minimal activity check - if we would want to prevent automatically promoting sites that have lots of old questions but weren't active for weeks or months.

