PAC learning

The goal

Distribution and the oracle

Distribution agnostic. Assumption: training distribution = testing distribution. 2 Oracle PAC model variant: Algorithm can use both \(D^{+}{c}\) and \(D^{-}{c}\), $error\le ϵ$ wrt both.

EQ oracle can be simulated by the EX oracle.

Strong PAC learning

Given $ϵ,\delta ,C$, we want to find $h\in C$ such that, with probability $1-\delta$, $Pr(c(x)\ne h(x))\le ϵ$.

Weak PAC learning algorithm

p, q polynomials; $ϵ\le 2^{-1}-g=2^{-1}-\frac{1}{p(n,size(c))}$, $\delta \le \frac{1}{q(n,size(c))}$. g: the advantage of $L$ over random guessing. Minimally better than random guessing: any g subexponential, can be achieved by memorizing previously seen examples.

h is weak hyp for $c$ iff: \ $P{r}_{x\sim D}[h(x)f(x)=-1]\le 2^{-1}+g$; or $E_{x\sim D}[h(x)c(x)]\ge 2g$.

PAC learning model

Make PAC learning algorithm

For C using H (containing C) with $m$ finite: Cast a \textbf{multivalued discrete classification function} into PAC model: Make a binary classifier concept for each bit.

Decide initial h (maybe null, universal); decide update to hypothesis corresponding to example provided by the sampling oracle : check if +ve examples enough.

If H finite: Create an Occam algorithm; maybe use size(h) to bound $|H|$; use Occam’s razor. If H finite: Create an Occamish algorithm inconsistent with at most $ϵ|S|/2$ examples; maybe use size(h) to bound $|H|$; use Chernoff to bound $Pr(\mathrm{\exists }$bad h$;|[c\mathrm{\Delta }h]\cap S|\le mϵ/2)$; thence continue Occam razor proof.

Try to find an algorithm which will grow the hypothesis decision list efficiently, as if solving a set cover problem.

Any $|H|$: Make algorithm to find h consistent with S, use VCD - Occam razor.

Make an algorithm to learn C using only EQ(c), simulate EQ using EX.

PAC reduce C over X to C’ over X'

Efficiently change $x\in X_n\to g(x)\in X_{p(n)}^{\prime }$; so that there exists proper image concept c’ for every $c$ with size(c’) = q(size(c)).

Prove efficient PAC learnability

Make PAC algorithm, bound $m$ for $(ϵ,\delta )$ approximation of c; prove that running time is polynomial in $({ϵ}^{-1},{\delta }^{-1},size(c),$m$,n)$.

Reduce to a known efficient PAC learning algorithm.

Show efficient learnability in the mistake bound model; bound $m$ using Occam razor or VCD Occam razor.

Find weak learner; boost.

If learning on uniform distribution, see U distribution Fourier analysis section.

Lower bound on sample complexity in PAC

Shatter dimension

m = $\mathrm{\Omega }(\frac{d}{ϵ})$.

\pf Take $S$ shattered by $C^{\prime }$, let $|S|=d$ and distribution $D=U$ over $S$. Then, we show that no algorithm $L$ can ensure $ϵ$ error whp. As $D$ is the support of $C^{\prime }$, without loss of generality, we will consider $C$ to be the finite set of $2^d$ concepts with different dichotomies induced on $S$.

Draw $S^{\prime }$ with $|S^{\prime }|=\frac{d}{2}$; suppose that $L$ learns $h$. $h=f_L(S^{\prime })$. But there are $|2^{S-S^{\prime }}|=2^{d/2}$ different concepts $c$ consistent with the labellings in $S^{\prime }$ and so, $h$ is chosen independently of these different concepts. Does $h$ have low error wrt all of these?

We want to show that there exists a $c$ such that the $h$ learned would have a high error rate with non-negligible probability. We will do this by considering the following process: Fix $S$ and $h$, pick $c\sim U(2^{S-S^{\prime }})$. We then show that $E_c[P{r}_x(h(x)\ne c(x))]>1/4$, which implies the existence of $c$ for which $h$ is a bad approximation.

$E_c[P{r}_x[h(x)\ne c(x)]]=E_c[P{r}_D(x\in S-S^{\prime })P{r}_c(h(x)\ne c(x)|x\in S-S^{\prime })]\ge 2^{-1}\times 2^{-1}$. Thus, there exists a $c$, for which the $h$ learned using $S^{\prime }$ as the sample would be bad.

We then need to show that $L$ cannot find a good hypothesis for this $c$ with high probability. \tbc

From halving algorithm

Halving algorithm $L$ mistakes in MB model is an approximate lower bound: take set S on which $L$ makes mistakes, make uniform distr on S, use halving algorithm in PAC model, then error probability is $(|S|-m)/(2\log |S|)\le ϵ$, so $|S|(1-1/(n^k))\le m\mathrm{\forall }k$.

PAC with mq

Any R efficiently PAC learnable using mq’s and eq’s is efficiently learnable with mq’s only.

PAC learnability of some R

Also see list of R which are PAC learnable with noise, learnable in MB.

Halfspaces

$f=sgn(w.x)$, $\mathrm{\forall }$x$,⟨w,x⟩\ne 0$, $\mathrm{\exists }{x}_i:|E_{x\sim D}[f(x)x_i]|\ge W^{-1}$; so some $x_i$ weak hypothesis; then use ADAboost.

Parities

Use the MB algorithm with Occam razor.

ae learning of parities wrt Uniform distribution is equivalent to decoding high rate random linear codes from low number of errors. So, believed to be hard.

Approximate f with orthogonal polynomials

Aka Fourier coefficients. Must have orthogonal polynomials for D.

Uniform distribution PAC learning (UPAC)

Easier than learning under general distributions: can use Fourier analysis.

Harder than learning Gaussian distr. \why

\oprob What is the lower bound for UPAC learning? Does ae learning D$L^{\prime }$ s make sense?

Shortness of DNF term length, Decn tree depth

DNF term-length or 1-D$L$ size is limited to $\log \frac{1}{ϵ}$: Probability of longer term being satisfied $<ϵ$.

Learn DNF

Collect all terms of size $log(s/ϵ)$ consistent with target DNF: use feature expansion, disjunction learning winnow algorithm.

The function space

Basis of the input space: ${\{\pm 1\}}^n$. Basis of the function space: Parity functions $p_S$. $f=\sum _{S\subseteq [n]}\hat{f}(S)p_S$. See Boolean function ref for details.

Alg to learn Fourier coefficient

Learn $\hat{f}(S)$\ using $\hat{f}(S)=E_{x{\in }_U{\{\pm 1\}}^n}[f(x)p_S(x)]$: Choose sample of size m; by Chernoff $m=\mathrm{\Omega }(\frac{\log {\delta }^{-1}}{l^2})$ for $(\delta ,l)$ approx of $\hat{f}(S)$.

Best fitting polynomial g of degree d: $\sum _{|S|\le d}\hat{f}(S)p_S$ (See Boolean function ref). $P{r}_x(sgn(g(x))\ne f(x))=E_x([sgn(g(x))\ne f(x)])\le E_x((f(x)-g(x){)}^2)$.

Low degree algorithm

If f is $(ϵ,\delta )$ concentrated, f learnable wrt U to accuracy 1-a in time $n^{O(\delta )}$: Sample to estimate $\hat{f}(S):|S|\le \delta$, make g, output $sgn(g)$. Agnostically learning big correlated parities.

Learning some R under U

Polysize DNF formulae learnable in \ $n^{O(\log |c|\log \frac{1}{ϵ}))}$. Halfspaces learnable in $n^{O({ϵ}^{-2})}$. Any function of k halfspaces learnable in $n^{O(k^2{ϵ}^{-2})}$. \oprob Do better for $\cap$ of k halfspaces.

Depth d ckts of size $|c|$. Also Decision trees. \why

Based on total influence

f learnable in $n^{O(\frac{d}{ϵ})}$ to accuracy $1-ϵ$. \why

So, monotone f learnable in $n^{O(\frac{\sqrt{n}}{ϵ})}$.

Hardness of ae learning of PARITY(k)

This is equivalent to decoding high rate random linear code C from low number of errors, a hard open problem. Also equivalent to learning PARITY with noise, in simpler case where noise is random and independent.

If you can decode $C$, you can ae learn PARITY: Simply take the examples $\left\{x_i\right\}$ you get, turn it into H, get a random G from $null(H^T)$, give it to the decoder along with the syndrome $\{p_e(x_i)\}$, return hypothesis $p_e$.

If you can learn PARITY(k) using algorithm $L$, you can learn it properly whp. Take the hypothesis h returned by $L$, turn it into an mq oracle which evaluates h(y) by taking h(x+y)+h(y) over many x. Thence make hypothesis parity $p$ attribute efficiently.

By properly ae learning parity you can decode random linear code: Given G and y, make H and yH, use learner.

UPAC with mq

Learning PARITY(k) using namq

Let H be check matrix for a code C; corrupted code y = c+e for some code $c$ and error e with $wt(e)<w$. Take BCR decoding algorithm (see coding theory ref) A which accepts y and the syndrome yH = eH, and returns e. So, using namq’s on the columns of H, A learns unknown $e\in PARITY(k)$ attribute efficiently. Another view: using e, A learns parity $c$ using noisy namq’s.

Sparse approach

For t-sparse f; or $\frac{{\parallel f\parallel }_1^2}{ϵ}$ sparse g: approximator for f.

Thus, using K-M algorithm, f is learnable to accuracy $1-ϵ$ in time \ $poly($n\(, \norm{f}{1}, \eps^{-1})\) using membership queries: find coords of g = coords of f of size \(\geq \frac{\eps}{\norm{f}{1}}\); use sgn(g) as hypothesis.

Weak parity learning

Given f with some t-heavy coefficient, given MQ oracle, find vector z with t/2 heavy $|\hat{f}(z)|$. $|\hat{f}(y)|\ge t$ iff $Pr(f=p_y)\ge 1/2+t/2$; so the weakly correlated parity $p_y$ is $1/2-t/2$ approximator of f, so called weak parity learning.

Can also view this as learning parities in agnostic setting. See agnostic learning section.

ae-namq algorithm: Like a sieve

Take an aena-mq parity learning algorithm L. $f_a:f(x\oplus a)$; S: the set of mq points of L. Estimate Fourier coefficients of f with t/4 accuracy. Also, for every y in S, find all Fourier coefficients of $f_{R,y}$ to estimate coefficients of $f_y$. ${\hat{f}}_c(a)=\hat{f}(a)p_a(c)$; so ${\hat{f}}_c(a)\hat{f}(a)$ and $p_a(c)$ have same sign.

Run $L$ for every $z\in {0,1}^m$ with \(\hat{f}R(z) \geq 3t/4\), using \(\hat{f}{R,c}(z)\hat{f}_R(z)\) to answer mq’s and learn $p_a$. If $a=zR$, add $p_a$ to a set of possible hypotheses. Repeat the process many times with different R. Really good $p_a$ occur in hypothesis sets at much higher rate. Thus, find a with $\hat{f}(a)\ge t/2$.

Can also be fixed to find t/2 heavy component of randomized fn $\psi$. Only, $m$ required for good approximation whp depends on $var(\psi )$.

K-M algorithm to find big components using membership queries

Let $|a|=k$. \ $f_a(x):=\sum _{Z\in {\{\pm 1\}}^{n-k}}\hat{f}(a,Z)p_{a,Z}(a,x)$.\ Then $f_a(x)=E_y[f(yx)p_a(y)]$: Let $Z=Z_1{Z}_2,Z_1\in {\{\pm 1\}}^k$; $E_y[f(yx)p_a(y)]=E_y[\sum _Z\hat{f}(Z)p_Z(yx)p_a(y)]=\sum _{Z_1,Z_2}\hat{f}(Z_1{Z}_2)p_{Z_2}(x)E_y[p_{Z_1}(y)p_a(y)]=\sum _{Z_2}\hat{f}(a{Z}_2)p_{Z_2}(x)$.

(Kushilevitz, Mansour). Input t; output all $|\hat{f}(S)|\ge t$ whp. Make tree: at root find ${\parallel f\parallel }^2$; at left child find ${\parallel f_0\parallel }^2$; at right child find ${\parallel f_1\parallel }^2$; if any branch has ${\parallel f_a\parallel }^2\le t^2$ prune; repeat.

Leaves of the tree are individual coefficients $\hat{f}(S)$. Height $\le n$. Breadth $\le t^{-2}$: every level is a partition of $\{\hat{f}(S)\}$; $t^{-2}$ of $\left\{f_a\right\}$ have ${\parallel f_a\parallel }^2>t^{-2}$.

To find ${\parallel f_a\parallel }^2=E_x[f_a(x{)}^2]$: Find $f_a(x)=E_y[f(yx)p_a(y)]$: pick random y’s, use membership queries.

Opcount: poly(t, n).

Results

Decision trees with t leaves approximable by t-sparse function; so learnable in polynomial time with membership queries.

Learning DNF

Aka Harmonic sieve. DNFs have a $(2s+1{)}^{-1}$ heavy component: For any D, there is $p_a$ such that $|E_D[f{p}_a]|\ge (2s+1{)}^{-1}$ and $wt(a)\le \log ((2s+1){\parallel D{2}^n\parallel }_{\mathrm{\infty }})$. \why.

$E_D[f(x)p_a(x)]=E_U[f(x)2^{n}D(x)p_a(x)]=\hat{\psi }(a)$. If ye give oracle access to D, you can evaluate $\psi$. So, use ae-namq algorithm to learn $a$ with $\hat{\psi }(a)\ge t/2$. As $var(\psi )\le \parallel 2^{n}D(x)\parallel$, $m$ required will depend on this. This is efficient only for D polynomially close to U.

Then boost using p-smooth algorithm.

Boosting weak efficient PAC algorithm L

Practical applications

Frequently used: boost decision trees.

Boost confidence

Run $L$ k times to gather k hypotheses (Get $ϵ$ approx with prob $1-(1-{\delta }_0{)}^k$), select best h by using many examples to gauge $h\mathrm{\Delta }c$.

Boosting accuracy

So you are using the guarantee that polynomial time $L$ works with any distribution to produce a hypothesis which produces very accurate results by playing with input distribution.

General idea for all techniques: put more weight on x on which weak hyp h makes mistake; rerun weak learner.

Lower bound

Number of iterations to get $ϵ$ accurate using $\gamma$ weak learner is $\mathrm{\Omega }(g^{-2}\log (\frac{1}{ϵ}))$.

Properties of booster

Noise tolerance. Smoothness: How far does the artificially generated distribution deviate from original?: Affects speed.

Boost accuracy by filtering

Reweight points using online algorithm.

Majority tree of Hypotheses Booster

(Schapire). Error b bosted to $g(b)=3b^2-2b^3$: Run $L$ to get $h_1$; filter D to get $D_2$ where $h_1$ is useless like random guessing; run $L$ to get $h_2$ - learn something new; filter D to get $D_3$ where $h_1(x)\ne h_2(x)$; run $L$ to get $h_3$; return $majority(h_1,h_2,h_3)$. Gauge errors of $h_1$ and $h_2$ to ensure efficient filterability.

Recursive accuracy boosting

For target accuracy b and distribution D: Maybe invoke self thrice for accuracy $g^{-1}(b)$ and distributions $D,D_2,D_3$. Final h like 3-ary tree; only leaves use samples for learning; others sample for error gauging.

Boost accuracy by sampling

Reweight points using offline algorithm.

Get sample S; boost accuracy till h consistant with S; bound size of h; bound $|S|$ using Occam razor.

Adaptive booster (AdaBoost)

Get sample S; use $U$ distr on it. At time t=0: wt on $s_j$ is $w_j=1$; $W_0=\sum w_i=m$; prob distr $D(x_i)=\frac{w_i}{W}$. On iteration i: run $L$, get hypothesis $h_i:X\to \{\pm 1\}$, $err(h_i)=E_i$, accuracy $a_i=1-E_i$, ${\beta }_i=\frac{E_i}{a_i}$; for each $s_j$ where $h_i$ is correct, reduce their weight: $w_j:=w_j{\beta }_i$. Output after time T: $sgn(\sum _i{a}_i{h}_i-2^{-1})$; $a_i=\frac{\lg {\beta }_i}{\sum _j\lg {\beta }_j}$. If $E=\underset{i}{max}{E}_i$, $\beta =ma{x}_i{\beta }_i$, this is $sgn(T^{-1}\sum _i{h}_i-2^{-1})$.

Final error $E_T\le e^{-2T{g}^2}$: $W_1=m(2E);W_T=m(2E{)}^T$. Weight of $x_i$ misclassified at iteration T $\ge {\beta }^{T/2}$: At $\ge \frac{T}{2}$ of $\left\{h_i\right\}$ could’ve been right on it. Total wt on misclassified points $\le ϵ$m${\beta }^{T/2}\le m(2E{)}^T$; so $ϵ\le (\frac{4E^2}{\beta }{)}^{T/2}=(4E(1-E){)}^{T/2}=(4(4^{-1}-g^2){)}^{T/2}\le exp(-2g^{2}T)$.

$e^{-2T{g}^2}<m^{-1}$ when $T>\frac{\ln m}{2g^2}$. So, final $|h|=O(\frac{\ln m}{2g^2})|c|$.

Contrast with maj tree booster

Gives simple hyphothesis; takes advantage of varying error rates of weak hypotheses. \why

Noise tolerance properties

Sensitive to noise even if $L$ resistant to noise: Distributions created by looking at actual label.

Boosting by branching programs

h = Branching program: DAG of hypotheses $\left\{h_i\right\}$. Labels x according to leaf reached. Run weak learner, get $h_0$. From distr $D_{h_0^{-}}$ and $D_{h_0^{+}}$, get $h_{1,1}$ and $h_{1,2}$. From $D_{h_{i,1}^{+}}$ get $h_{i+1,1}$; from $D_{h_{i,j}^{+}}\vee D_{h_{i,j+1}^{-}}$ get $h_{i+1,j+1}$: diamonds in the DAG; from $D_{h_{i,i+1}^{+}}$ get $h_{i+1,i+2}$.

Assume that both $er{r}_{D_{c^{-}}}(h_{i,j})$ and $er{r}_{D_{c^{-}}}(h_{i,j})$ $\le 2^{-1}-g$ : can be removed. \why

At node (j,k); k-1 hypotheses have said h(x)=1; j-k+1 hypotheses have said h(x)=0. So, in final level T; first T/2 leaves labelled 0; rest labelled 1.

Take x with c(x)=1: $h_i=h_{i,j}$ seen in iteration i: In worst case, $E_x[[h_i(x)=1]]=2^{-1}+g$; so $E_x$[leaf where x ends up] $=\frac{T}{2}+gT$. A biased random walk.

Show $P{r}_x(h(x)=1)\ge 1-ϵ$: due to lack of independence of events h(x) = 1, martingale is the right tool; let $X_i=[h_i(x)=1]$, $Y=\sum _{i=1}^T{X}_i$; $Y_i=E[Y|X_1,..X_i]$ is a Doob martingale with $Y_i-Y_{i-1}=1$; $Y_i=\sum _{j=1}^i{X}_i+\frac{T-i}{2}+g(T-i)$; by Azuma: $Pr(|Y-E[Y]|\ge gT)\le 2e^{-g^{2}T}\le ϵ$ for $T\ge \frac{10\log \frac{1}{t}}{g^2}$.

Resistant to noise if $L$ resistant to noise: Boosting program creates distributions without looking at $c(x)$.

Consequences

PAC algorithm memory bound: $O(\frac{1}{ϵ})$. \why

Hardness of PAC learning

General techniques

Specify distr where learning hard.

Any C which includes functions which can be efficiently used to solve iterated products [ cryptography ref] is inherently unpredictable.

PAC-reduce a C’ known to be hard to $C$.

Show that VCD is $\mathrm{\infty }$. Eg: $C=\left\{convexpolygons\right\}$.

Reduce it to a hard problem in another area: eg: decoding random linear codes.

If RP neq NP

By contradiction: Take NP complete language A; efficiently reduce input a to labelled sample set $S_a$ where $S_a\equiv$c$\in C$ iff $a\in A$; assume efficient PAC learning algorithm $L$; with suitable \(\eps \stackrel{?}{=} |S{a}|^{-1}\), use $L$ with timer to learn $ϵ$ close h whp; see if h is consistant with $S_a$; if yes, $a\in A$; otherwise $a\notin A$; we have an RP algorithm for A!

Hardness results for proper learning

$k\ge 3$: k-term DNF learning intractable (H is restricted to equal C), but predicting classification accurately tractable: Hardness by reduction to graph coloring problem \why; k-term DNF = some k-CNF by distr law: so improper learning using k-CNF as H is easy. So, conversion from k-CNF learnt to k-term DNF is hard.

Usually the $RP\ne NP$ assumption is used.

Cryptographic hardness results for improper learning

Aka inherent tractable-unpredictability. Learning hard despite: Polynomial examples being sufficient; or concept evaluation easy.

C which include concepts which can find Discrete cube roots

Take n bit N’s. Make discrete cube root hardness assumption [see cryptography ref]. Consider uniform distr over $Z_N^{\ast }$; any algorithm can generate examples by itself - so doesn’t learn anything new; otherwise contradicts assumption. Learning remains hard even if supplied n repeated squares of a number y mod N : no clue about d in this $O(n^2)$ input.

By PAC reduction, anything which can compute iterated products is hard to learn.

C which include concepts to find ith bit of BBS pseudorandom generator outputs

$f_{s_0,N,t}(i)$: ith output bit of BBS pseudorandom generator with seed $s_0$ and N, if $i\le t$ (see randomized algs ref). If $\mathrm{\exists }$ $O(n^{ck})$ time algorithm A to learn C with error $\le 2^{-1}-n^{-k}$; you can distinguish BBS from random string b of $n^{(d+1)ck}$ bits, where $d\in Z,dc\ne 1$; and thence factor $N$.

Distinguisher D

Input random string; Use Uniform distr over b; use $(i,b_i)$ as examples; learn with $n^{ck}$ examples; get hypothesis h with error $\le 2^{-1}-n^{-k}$; Pick another bit index j; try predicting $b_j$; if guess correct, output 1 or ‘generator’. On truly random b, $Pr(D^{rand}=1)\ge 2^{-1}+\frac{n^{ck}}{n^{(d+1)ck}}$; but $Pr(D^{f_{s_0,N,t}}=1)\le 2^{-1}+n^{-k}$: difference not negligible.

By PAC reduction, anything which can compute iterated products is hard to learn.

Classes which include RSA decryptors

$C=\left\{f_{d,N}\right\}$, with private key d, public key e: see Cryptography ref. Not learnable in polytime: else make examples over $U({\{0,1\}}^n)$: $(x^e\mathrm{mod}N,x)$; learn and violate RSA unbreakability assumption.

Inherently tractably-unpredictable classes

Polynomial sized boolean circuits of depth log n and $n^2$ inputs: Can solve Iterated products with ckt of depth ${\log }^{2}n$ (using binary tree like mult ckt). Beame et al: even with log n deep ckt. So, Poly-sized Boolean formulae inherently unpredictable.

Also, Neural networks of constant depth with boolean weights: \ (Reif). By PAC reduction from Poly-sized Boolean formulae, Log-space turing machines and DFA inherently unpredictable (but DFA learnable with membership queries).

Intersections of halfspaces. \why Agnostically learning single halfspace. \why

\oprob DNF formulae (can learn under U if RP= NP) : hard to learn? : too weak to encode pseudorandom generators, many cryptographic primitives; if you unconditionally show DNF hard to learn under U, you show $RP\ne NP$!