From 796e0a8dabca1ad4ee7efc97701ed4e8ee93fe15 Mon Sep 17 00:00:00 2001 From: thesyncim Date: Sat, 31 Oct 2026 18:30:25 +0000 Subject: [PATCH] internal/loopir, internal/rewrite: vectorize stores at a loop-invariant offset A loop that stores at an offset from the index, such as a row of a 2-D slice (b[i*ldb+j] = a[i*lda+j], c[row+j] += alpha*x[j]) or a fill that starts part way into a slice (line[off+x] = v), was rejected: loads could be at an offset, but every store had to be at exactly the loop index. A store may now be at the same offsets a load may: i+e, e+i and i-e for a loop-invariant integer e. Every store to one slice must be at the same offset, so each iteration still writes its own element; two offsets are rejected with their own reason. A load of the stored slice at that offset is the same element, ordered after the stores before it as at the index. The dependence test for a load at another offset measures the distance from the store offset instead of from the index, so a constant positive distance in a forward loop is still loaded before any store of the chunk, and a negative or variable distance is rejected as before. An if that selects between stores, and the copy builtin, take the offset too. The emitted store is capped at the limit plus the offset, so the vector loop never writes an element the scalar loop does not, and the length check and the overlap window cover the elements stored. A constant offset that puts every stored index below zero is left alone, as for a load. The vectorized set gains one loop in std (an image/jpeg fill), sixteen in gonum (dlacpy, dlaset, dlascl and dlarfb rows, and testlapack helpers) and five in github.com/thesyncim/goav1; TSVC s423 and s424 now vectorize and match their goldens. --- AGENTS.md | 35 +- README.md | 1 + bench/bench_test.go | 15 + bench/ops.go | 6 + bench/ops_simd.go | 46 ++ internal/loopir/ir.go | 12 +- internal/loopir/lower.go | 135 ++-- internal/loopir/plan.go | 19 +- internal/loopir/reason.go | 26 +- internal/rewrite/emit.go | 43 +- testdata/json_reasons.txt | 8 +- testdata/offset_negative_constant.txt | 112 ++- .../offset_negative_constant_no_rewrite.txt | 34 +- testdata/offset_no_rewrite.txt | 13 +- testdata/offset_store.txt | 671 ++++++++++++++++++ testdata/offset_store_no_rewrite.txt | 75 ++ tsvc/README.md | 6 +- tsvc/testdata/coverage.txt | 16 +- 18 files changed, 1135 insertions(+), 138 deletions(-) create mode 100644 testdata/offset_store.txt create mode 100644 testdata/offset_store_no_rewrite.txt diff --git a/AGENTS.md b/AGENTS.md index b96b922..a2e2be7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,22 +104,25 @@ safety argument stays structural: the loop runs at least once (`x` may be a nil pointer the scalar loop never reads), and the overlap check and the vector loop use the locals; the fallback is the original loop. Arrays, pointers to slices, maps, and the result of a call stay rejected. -- A store is `ident[i]` where `i` is *exactly* the loop variable (`index`). A load - may also be `ident[i+e]`, `ident[e+i]` or `ident[i-e]` for a loop-invariant integer - `e` (`indexAt`, `Load.Off`). A slice that is also stored may be read only at a - constant positive offset in a forward loop: iteration i then reads what a later - iteration writes, so the scalar loop sees the old value and the emitter loads it - before any store of the chunk (`statements` hoists those loads). A negative - offset is a recurrence (`ReasonCarriedDependence`); a variable distance or a - loop that counts down is `ReasonOffsetOfStored`. A constant offset that puts an - index below zero, the lowest one read with a constant start or every one with a - constant limit, is `ReasonNegativeIndex`: the scalar loop panics and the emitted - code would index at a negative constant. A positive one that pushes a constant - start or limit past the largest int is rejected as that bound, since the emitted - code adds them. The length check is against the last index read (`lastIndex`), - and offset loads are not capped at the limit. Every other store and load is `ident[i]` (`index`) — `dst[i+1]`, `dst[2*i]` are rejected outright, regardless of - whether they're actually dependent. That is what guarantees no two - iterations touch the same element. +- A store or a load is `ident[i]`, `ident[i+e]`, `ident[e+i]` or `ident[i-e]` where + `i` is *exactly* the loop variable and `e` a loop-invariant integer (`indexAt`, + `Store.Off`, `Load.Off`). Every store to one slice is at the same offset + (`ReasonStoreOffsets` otherwise), so each iteration writes its own element. A slice + that is also stored may be read at that offset, or at a constant positive distance + past it in a forward loop: iteration i then reads what a later iteration writes, so + the scalar loop sees the old value and the emitter loads it before any store of the + chunk (`statements` hoists those loads). A negative distance is a recurrence + (`ReasonCarriedDependence`); a variable distance or a loop that counts down is + `ReasonOffsetOfStored`. A constant offset that puts an index below zero, the lowest + one touched with a constant start or every one with a constant limit, is + `ReasonNegativeIndex`: the scalar loop panics and the emitted code would index at a + negative constant. A positive one that pushes a constant start or limit past the + largest int is rejected as that bound, since the emitted code adds them. The length + check is against the last index touched (`lastIndex`); offset loads are not capped + at the limit, and a store at an offset is capped at the limit plus the offset + (`destination`), so it writes nothing the scalar loop does not. `dst[2*i]` and + `dst[i][j]` are rejected outright, regardless of whether they're actually dependent. + That is what guarantees no two iterations touch the same element. - A non-slice operand becomes an `Invariant` only if it is a variable, a constant, a conversion or negation of one, or a sum, difference, product or bitwise combination of two (`invariant`). It is evaluated diff --git a/README.md b/README.md index cae8ccc..5fc1446 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,7 @@ operations that lower to **AVX-512/AVX2/NEON** depending on the target CPU: | `for i := range x { if x[i] < 0 { x[i] = 0 } }`, `if c { dst[i] = u } else if d { dst[i] = v } else { dst[i] = w }` | compare, then select (`IfElse`): ReLU, leaky ReLU, ReLU6, hard tanh, clamp, sign, step; also when the branches assign a temporary (`t := x[i]+3; if t < 0 { t = 0 } else if t > 6 { t = 6 }`: hard sigmoid, hard swish, cubic SiLU) | | `for i := range v.Values { v.Values[i] *= a }`, `dst.f[i] += src.f[i]` | a slice reached through a field (`x.f`, `x.f.g`); the root must not be assigned in the loop, and the field is copied into a local once | | `for i := range dst { dst[i] = a[i+1] - a[i] }`, `work[j] += a[row*lda+j]` | read at an offset from the index (stencil, row of a 2-D slice); a slice the loop also writes is read only at a constant positive offset (`a[i] = a[i+1] + b[i]`) | +| `b[i*ldb+j] = a[i*lda+j]`, `c[row+j] += alpha * x[j]`, `line[off+x] = v` | store at an offset from the index (row of a 2-D slice); every store to a slice is at the same offset, and the slice is read only there or at a constant positive distance past it | Supported element types: `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`. diff --git a/bench/bench_test.go b/bench/bench_test.go index 8821213..cff789e 100644 --- a/bench/bench_test.go +++ b/bench/bench_test.go @@ -451,6 +451,21 @@ func BenchmarkAccumRowInt32s(b *testing.B) { } } +// StoreRowFloat32s adds alpha*x into one row of a two-dimensional slice. +func BenchmarkStoreRowFloat32s(b *testing.B) { + for _, n := range benchSizes { + b.Run(fmt.Sprintf("%d", n), func(b *testing.B) { + x := make([]float32, n) + c := make([]float32, 2*n) + setBytes(b, n, 4, 3) + b.ResetTimer() + for b.Loop() { + StoreRowFloat32s(c, x, 2, 1, n) + } + }) + } +} + func BenchmarkMinUint16s(b *testing.B) { for _, n := range benchSizes { b.Run(fmt.Sprintf("%d", n), func(b *testing.B) { diff --git a/bench/ops.go b/bench/ops.go index 12abaf9..6a143c0 100644 --- a/bench/ops.go +++ b/bench/ops.go @@ -332,3 +332,9 @@ func HardSigmoidFloat32s(dst, x []float32) { dst[i] = t / 6 } } + +func StoreRowFloat32s(c, x []float32, alpha float32, row, ldc int) { + for j := range x { + c[row*ldc+j] += alpha * x[j] + } +} diff --git a/bench/ops_simd.go b/bench/ops_simd.go index 5cf91a3..142e379 100644 --- a/bench/ops_simd.go +++ b/bench/ops_simd.go @@ -1994,6 +1994,52 @@ func HardSigmoidFloat32s(dst, x []float32) { } } +func StoreRowFloat32s(c, x []float32, alpha float32, row, ldc int) { + _vcAFloat32s46 := simd.BroadcastFloat32s(alpha) + if _loopvecOverlap(c, x, 0+(min(0, row*ldc)), len(x)+(max(0, row*ldc)), 0, len(x)) { + for j := range x { + c[row*ldc+j] += alpha * x[j] + } + } else { + if len(x) > 0 { + _ = c[len(x)+(row*ldc)-1] + _ = x[len(x)-1] + { + _w1 := c[row*ldc:][:len(x):len(x)] + _count := len(_w1) + _w2 := x[:_count:_count] + _i, _lanes := 0, simd.BroadcastFloat32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadFloat32s(_c2) + _v2 := simd.LoadFloat32s(_c1) + _v1.MulAdd(_vcAFloat32s46, _v2).Store(_c1) + _v3 := simd.LoadFloat32s(_c2[_lanes:]) + _v4 := simd.LoadFloat32s(_c1[_lanes:]) + _v3.MulAdd(_vcAFloat32s46, _v4).Store(_c1[_lanes:]) + _v5 := simd.LoadFloat32s(_c2[2*_lanes:]) + _v6 := simd.LoadFloat32s(_c1[2*_lanes:]) + _v5.MulAdd(_vcAFloat32s46, _v6).Store(_c1[2*_lanes:]) + _v7 := simd.LoadFloat32s(_c2[3*_lanes:]) + _v8 := simd.LoadFloat32s(_c1[3*_lanes:]) + _v7.MulAdd(_vcAFloat32s46, _v8).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v9 := simd.LoadFloat32s(_w2[_i:]) + _v10 := simd.LoadFloat32s(_w1[_i:]) + _v9.MulAdd(_vcAFloat32s46, _v10).Store(_w1[_i:]) + } + for _i < _count { + _v11, _n := simd.LoadFloat32sPart(_w2[_i:]) + _v12, _ := simd.LoadFloat32sPart(_w1[_i:]) + _v11.MulAdd(_vcAFloat32s46, _v12).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + func _loopvecOverlap[T any](a, b []T, aLo, aHi, bLo, bHi int) bool { if aLo >= aHi || bLo >= bHi { return false diff --git a/internal/loopir/ir.go b/internal/loopir/ir.go index 9b596b0..8f4d59f 100644 --- a/internal/loopir/ir.go +++ b/internal/loopir/ir.go @@ -172,9 +172,12 @@ type Loop struct { // Stmt is a statement of a loop body: a Store or a Let. type Stmt interface{ isStmt() } -// Store writes Val to Dst[i]. +// Store writes Val to Dst[i+Off]. Off is source text for an integer expression +// that is the same on every iteration, "" for zero, as in Load. Every store to +// one slice is at the same offset, so no two iterations write the same element. type Store struct { Dst *Ref + Off string Val Value } @@ -213,9 +216,10 @@ type Value interface{ isValue() } // Load reads Ref[i+Off]. Off is source text for an integer expression that is the // same on every iteration, "" for zero. A slice that is also stored in the loop is -// read only at a constant positive offset: iteration i then reads the element a -// later iteration writes, and the vector loop, which loads the chunk before any -// store of the body, sees the same old value. +// read only at the offset it is stored at, or a constant positive distance past +// it: iteration i then reads the element a later iteration writes, and the vector +// loop, which loads the chunk before any store of the body, sees the same old +// value. // FromRange is set when the source is the range value variable, which holds the // element as it was when the iteration began: a store to the same slice earlier in // the body does not change it. diff --git a/internal/loopir/lower.go b/internal/loopir/lower.go index cf9e6d9..f46188d 100644 --- a/internal/loopir/lower.go +++ b/internal/loopir/lower.go @@ -616,7 +616,7 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6 // dst[i] = src[i] is a copy, emitted as the copy builtin. Copying a // slice onto itself does nothing; at an offset it shifts the elements, // which copy does correctly where the ranges overlap. - if v.Ref.Obj == stmt.Dst.Obj && v.Off == "" { + if v.Ref.Obj == stmt.Dst.Obj && v.Off == stmt.Off { return nil, ReasonUnsupportedOperand } case *Invariant: @@ -652,58 +652,75 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6 } } - // A slice that is both stored and read at an offset is a dependence between - // iterations: iteration i reads the element iteration i+d writes. The vector - // loop runs a whole chunk of iterations statement by statement, so it keeps the - // scalar result only when the read is of a later iteration's element (d > 0, in a - // forward loop): that iteration has not run yet, so the scalar loop reads the - // value from before the loop, and the emitted loop loads it before any store of - // the chunk. A read of an earlier iteration's element (d < 0) is a recurrence, - // and a distance that is not a constant, or a loop that counts down, has no - // order loopvec can check. + // Every store to a slice is at one offset, so each iteration writes its own + // element. Two offsets would have different iterations write the same element, + // in an order the vector loop, which runs one statement over a whole chunk + // before the next, does not keep. // - // A constant offset that puts an index below zero makes the scalar loop panic - // when it runs. Start is the lowest index the loop visits, whichever way it - // counts, so a constant start puts the lowest index read at Start+off, and a - // constant limit puts the highest at Limit-1+off. Such a loop is left alone, - // since the vector loop would index or slice at that negative constant, which - // does not compile. Start is never negative and Limit never below one (see - // rangeReason), so only a negative offset can do this, and it is at least - // -maxOffset, so negating it cannot overflow. A positive offset that pushes a - // constant start or limit past the largest int would not compile either, as the - // emitted code adds them. - stored := map[types.Object]bool{} + // A slice that is both stored and read at another offset is a dependence + // between iterations: iteration i reads the element iteration i+d writes, d + // the read offset less the store offset. The vector loop runs a whole chunk of + // iterations statement by statement, so it keeps the scalar result only when + // the read is of a later iteration's element (d > 0, in a forward loop): that + // iteration has not run yet, so the scalar loop reads the value from before the + // loop, and the emitted loop loads it before any store of the chunk. A read of + // an earlier iteration's element (d < 0) is a recurrence, and a distance that + // is not a constant, or a loop that counts down, has no order loopvec can check. + stored := map[types.Object]string{} for _, stmt := range body { if store, ok := stmt.(Store); ok { - stored[store.Dst.Obj] = true + if off, ok := stored[store.Dst.Obj]; ok && off != store.Off { + return nil, ReasonStoreOffsets + } + stored[store.Dst.Obj] = store.Off } } + // A constant offset that puts an index below zero makes the scalar loop panic + // when it reaches that index. Start is the lowest index the loop visits, + // whichever way it counts, so a constant start puts the lowest index touched at + // Start+off, and a constant limit puts the highest at Limit-1+off. Such a loop + // is left alone, since the vector loop would index or slice at that negative + // constant, which does not compile. Start is never negative and Limit never + // below one (see rangeReason), so only a negative offset can do this, and it is + // at least -maxOffset, so negating it cannot overflow. A positive offset that + // pushes a constant start or limit past the largest int would not compile + // either, as the emitted code adds them. + offsetReason := func(offset string) Reason { + off, err := strconv.ParseInt(offset, 10, 64) + switch { + case err != nil: + return "" + case off < 0 && (trip.StartConst && trip.StartValue < -off || trip.LimitConst && trip.LimitValue <= -off): + return ReasonNegativeIndex + case off > 0 && trip.StartConst && trip.StartValue > math.MaxInt64-off: + return ReasonUnsupportedStart + case off > 0 && trip.LimitConst && trip.LimitValue > math.MaxInt64-off: + return ReasonUnsupportedBound + } + return "" + } for _, stmt := range body { + if store, ok := stmt.(Store); ok { + if reason := offsetReason(store.Off); reason != "" { + return nil, reason + } + } for _, leaf := range Leaves(stmtValue(stmt)) { load, ok := leaf.(*Load) - if !ok || load.Off == "" { + if !ok { continue } - distance, err := strconv.ParseInt(load.Off, 10, 64) - if stored[load.Ref.Obj] { + if off, isStored := stored[load.Ref.Obj]; isStored && load.Off != off { + d, ok := distance(load.Off, off) switch { - case err != nil || step != 1: + case !ok || step != 1: return nil, ReasonOffsetOfStored - case distance < 0: + case d < 0: return nil, ReasonCarriedDependence } } - if err == nil && distance < 0 && - (trip.StartConst && trip.StartValue < -distance || trip.LimitConst && trip.LimitValue <= -distance) { - return nil, ReasonNegativeIndex - } - if err == nil && distance > 0 { - switch { - case trip.StartConst && trip.StartValue > math.MaxInt64-distance: - return nil, ReasonUnsupportedStart - case trip.LimitConst && trip.LimitValue > math.MaxInt64-distance: - return nil, ReasonUnsupportedBound - } + if reason := offsetReason(load.Off); reason != "" { + return nil, reason } } } @@ -879,7 +896,7 @@ func (l *lowerer) deadOutsideLoop(v *types.Var) bool { return true } -// store lowers one assignment to dst[i]. +// store lowers one assignment to dst[i], or to dst[i+e] for a loop-invariant e. func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { var op Op switch assign.Tok { @@ -892,7 +909,7 @@ func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { return Store{}, ReasonUnsupportedOperand } } - dst, ok := l.index(assign.Lhs[0]) + dst, off, ok := l.indexAt(assign.Lhs[0]) if !ok { return Store{}, ReasonUnsupportedDestination } @@ -908,18 +925,18 @@ func (l *lowerer) store(assign *ast.AssignStmt) (Store, Reason) { if assign.Tok == token.SHR_ASSIGN { shift = token.SHR } - val, reason = l.shiftOf(shift, &Load{Ref: dst}, assign.Rhs[0]) + val, reason = l.shiftOf(shift, &Load{Ref: dst, Off: off}, assign.Rhs[0]) default: // dst[i] op= rhs is dst[i] = dst[i] op rhs. var rhs Value if rhs, reason = l.value(assign.Rhs[0]); reason == "" { - val = &Binary{Op: op, X: &Load{Ref: dst}, Y: rhs} + val = &Binary{Op: op, X: &Load{Ref: dst, Off: off}, Y: rhs} } } if reason != "" { return Store{}, reason } - return Store{Dst: dst, Val: val}, "" + return Store{Dst: dst, Off: off, Val: val}, "" } // loadKey identifies the element a load reads: the range value variable reads the @@ -1060,7 +1077,7 @@ func (l *lowerer) ifStore(s *ast.IfStmt) (Store, Reason) { } sel := store.Val.(*Select) known := loadKeys(sel.Cond) - known[loadKey{store.Dst.Obj, ""}] = true + known[loadKey{store.Dst.Obj, store.Off}] = true for _, leaf := range Leaves(sel) { if load, ok := leaf.(*Load); ok && !known[loadKey{load.Ref.Obj, load.Off}] { return Store{}, ReasonConditionalLoad @@ -1087,16 +1104,16 @@ func (l *lowerer) ifChain(s *ast.IfStmt) (Store, Reason) { case nil: // No else: the element keeps its value. The vector loop stores it back, which // is only unobservable when the condition already reads the element. - if !loadKeys(cond)[loadKey{then.Dst.Obj, ""}] { + if !loadKeys(cond)[loadKey{then.Dst.Obj, then.Off}] { return Store{}, ReasonConditionalStore } - other = &Load{Ref: then.Dst} + other = &Load{Ref: then.Dst, Off: then.Off} case *ast.BlockStmt: store, reason := l.branch(e) if reason != "" { return Store{}, reason } - if store.Dst.Obj != then.Dst.Obj { + if store.Dst.Obj != then.Dst.Obj || store.Off != then.Off { return Store{}, ReasonUnsupportedIf } other = store.Val @@ -1105,12 +1122,12 @@ func (l *lowerer) ifChain(s *ast.IfStmt) (Store, Reason) { if reason != "" { return Store{}, reason } - if store.Dst.Obj != then.Dst.Obj { + if store.Dst.Obj != then.Dst.Obj || store.Off != then.Off { return Store{}, ReasonUnsupportedIf } other = store.Val } - return Store{Dst: then.Dst, Val: &Select{Cond: cond, Then: then.Val, Else: other}}, "" + return Store{Dst: then.Dst, Off: then.Off, Val: &Select{Cond: cond, Then: then.Val, Else: other}}, "" } // branch lowers the body of an if: one assignment to an element. @@ -1396,10 +1413,22 @@ func (l *lowerer) load(expr ast.Expr) (*Load, bool) { return &Load{Ref: ref, Off: off}, true } -// index matches slice[i] where slice is an identifier and i the loop index. -func (l *lowerer) index(expr ast.Expr) (*Ref, bool) { - ref, off, ok := l.indexAt(expr) - return ref, ok && off == "" +// distance is the constant read offset less the constant store offset, each "" +// for zero; ok is false when either is not a constant. +func distance(read, store string) (d int64, ok bool) { + r, s := int64(0), int64(0) + var err error + if read != "" { + if r, err = strconv.ParseInt(read, 10, 64); err != nil { + return 0, false + } + } + if store != "" { + if s, err = strconv.ParseInt(store, 10, 64); err != nil { + return 0, false + } + } + return r - s, true } // maxOffset bounds a constant offset, so that adding it to a limit cannot diff --git a/internal/loopir/plan.go b/internal/loopir/plan.go index bbe9e61..63e6d06 100644 --- a/internal/loopir/plan.go +++ b/internal/loopir/plan.go @@ -64,9 +64,9 @@ type Plan struct { // NonEmpty is set when the loop is known to run at least once. NonEmpty bool // Checked lists every slice whose length must be verified against the - // limit before the loop runs: for each store in order, its destination and - // then each slice it reads, in the order the loads are emitted. A slice read - // at several offsets is listed once for each. + // limit before the loop runs: for each store in order, its destination at + // its offset and then each slice it reads, in the order the loads are + // emitted. A slice read at several offsets is listed once for each. Checked []Access // Hoists lists the slices the loop reaches through a selector (x.f), in order: the // emitted code assigns each Name from Src once before the loop. @@ -87,9 +87,11 @@ type Access struct { } // PlanStmt is a statement whose value has been normalized; see Normalize. It -// stores to Dst, defines Temp, or folds into Acc with Op, whichever is set. +// stores to Dst at offset Off, defines Temp, or folds into Acc with Op, whichever +// is set. type PlanStmt struct { Dst *Ref + Off string Temp *Temp Acc types.Object Op Op @@ -127,7 +129,7 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { if !types.Identical(stmt.Dst.Elem, elem) { return nil, ReasonMixedTypes } - planned = PlanStmt{Dst: stmt.Dst, Root: Normalize(stmt.Val)} + planned = PlanStmt{Dst: stmt.Dst, Off: stmt.Off, Root: Normalize(stmt.Val)} case Let: if !types.Identical(stmt.Temp.Var.Type(), elem) { return nil, ReasonMixedTypes @@ -180,9 +182,10 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { p.Start, p.Limit, p.NonEmpty, p.TripRef = trip.Start, trip.Limit, trip.NonEmpty, trip.Slice // The generated loop stops at the length of the stored slice, so unless the // limit is exactly that, the limit is spelled out on every slice or a longer - // one would be read or written past it. + // one would be read or written past it. A store at an offset always needs it: + // the elements past limit+offset are ones the scalar loop does not write. for _, stmt := range p.Stmts { - if stmt.Dst != nil && (trip.Slice == nil || stmt.Dst.Obj != trip.Slice.Obj) { + if stmt.Dst != nil && (trip.Slice == nil || stmt.Dst.Obj != trip.Slice.Obj || stmt.Off != "") { p.Bound = trip.Limit break } @@ -193,7 +196,7 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) { for _, stmt := range p.Stmts { if stmt.Dst != nil { stored[stmt.Dst.Obj] = true - p.Checked = appendAccess(p.Checked, Access{Ref: stmt.Dst}) + p.Checked = appendAccess(p.Checked, Access{Ref: stmt.Dst, Off: stmt.Off}) refs = appendRef(refs, stmt.Dst) } for _, leaf := range Leaves(stmt.Root) { diff --git a/internal/loopir/reason.go b/internal/loopir/reason.go index 20dfb66..d8a0357 100644 --- a/internal/loopir/reason.go +++ b/internal/loopir/reason.go @@ -18,8 +18,8 @@ const ( // ReasonUnsupportedStart covers a counting loop whose start is not a // loop-invariant integer expression (a constant, an int variable, len of a // slice, and sums and products of those), is a negative constant, or is a - // constant that a read at a positive constant offset pushes past the largest - // int. + // constant that a read or store at a positive constant offset pushes past the + // largest int. ReasonUnsupportedStart Reason = "loop start is not a loop-invariant integer expression" // ReasonUnsupportedCondition covers a loop condition that is not // i < limit or i <= limit, or i >= limit or i > limit when counting down. @@ -31,23 +31,27 @@ const ( // expression, is a constant that is not positive, or is written so that // adding one to it could overflow (a <= limit, or the start of a loop that // counts down, that is not a constant or something minus a constant), or is - // a constant that a read at a positive constant offset pushes past the + // a constant that a read or store at a positive constant offset pushes past the // largest int. ReasonUnsupportedBound Reason = "loop limit is not a loop-invariant integer expression" - // ReasonUnsupportedDestination covers a destination that isn't a plain - // slice[i] element: an offset (dst[i+1]), a stride (dst[2*i]), or a - // nested index (dst[i][j]). + // ReasonUnsupportedDestination covers a destination that isn't a slice + // element at the loop index or a loop-invariant offset from it: a stride + // (dst[2*i]), a nested index (dst[i][j]), or an offset that changes. ReasonUnsupportedDestination Reason = "destination is not a simple, same-index slice element" - // ReasonOffsetOfStored covers a slice that is read at an offset from the loop - // index (a[i+1]) and also written in the loop, in a way the dependence test - // cannot clear: the offset is not a constant, or the loop counts down. + // ReasonOffsetOfStored covers a slice that is read at another offset from the + // loop index than it is written at (a[i+1] and a[i]), in a way the dependence + // test cannot clear: the distance is not a constant, or the loop counts down. ReasonOffsetOfStored Reason = "a slice is read at an offset and written in the same loop" + // ReasonStoreOffsets covers a slice stored at two different offsets from the + // loop index (a[i] and a[i+1]): two iterations write the same element, and the + // vector loop would not keep the order the scalar loop writes it in. + ReasonStoreOffsets Reason = "a slice is stored at more than one offset" // ReasonCarriedDependence covers a slice read at an earlier index than the // iteration writes (a[i] = a[i-1] + x): iteration i reads what iteration i-1 // wrote, so the iterations cannot run as a vector. ReasonCarriedDependence Reason = "an iteration reads an element an earlier iteration wrote" - // ReasonNegativeIndex covers a slice read at a constant offset that puts an - // index below zero: the one read at the lowest index the loop visits, with a + // ReasonNegativeIndex covers a slice read or stored at a constant offset that + // puts an index below zero: the one at the lowest index the loop visits, with a // constant start (a[i-1] in a loop over 0..n-1, whichever way it counts), or // every one, with a constant limit (a[i-20] in a loop that stops before 16). The // scalar loop panics when it reaches that index, and the vector loop would index diff --git a/internal/rewrite/emit.go b/internal/rewrite/emit.go index d338c7d..f798ea9 100644 --- a/internal/rewrite/emit.go +++ b/internal/rewrite/emit.go @@ -89,10 +89,10 @@ func (e *emitter) copyCall() string { destination, source := store.Dst.Name, load.Ref.Name switch { case e.plan.Start != "0": - destination += "[" + e.plan.Start + ":" + e.plan.Bound + "]" + destination += "[" + addOffset(e.plan.Start, store.Off) + ":" + addOffset(e.plan.Bound, store.Off) + "]" source += "[" + addOffset(e.plan.Start, load.Off) + ":]" case e.plan.Bound != "": - destination += "[:" + e.plan.Bound + "]" + destination += "[" + store.Off + ":" + addOffset(e.plan.Bound, store.Off) + "]" if load.Off != "" { source += "[" + load.Off + ":]" } @@ -140,6 +140,18 @@ func (e *emitter) operand(ref *loopir.Ref, pos, off string) string { return ref.Name + "[" + addOffset(pos, off) + ":]" } +// destination slices the stored ref from position pos at offset off. Inside the +// element-wise loop the store's window does that, since it holds exactly the +// elements the loop stores. Elsewhere a store at an offset is capped at the limit +// plus the offset, so it never writes an element past the last one the scalar +// loop writes. +func (e *emitter) destination(ref *loopir.Ref, pos, off string) string { + if _, ok := e.windows[windowKey{ref.Obj, off}]; ok || off == "" { + return e.operand(ref, pos, off) + } + return ref.Name + "[" + addOffset(pos, off) + ":" + addOffset(e.plan.Bound, off) + "]" +} + // addOffset is the expression pos + off, where off is "" for zero, a constant, or // an expression. func addOffset(pos, off string) string { @@ -321,20 +333,23 @@ func (e *emitter) statements(b *strings.Builder, full bool, copy, copies int) { } fmt.Fprintf(b, "\t%s, %s := simd.Load%sPart(%s)\n", name, lanes, e.plan.SimdType, e.operand(load.Ref, offset, load.Off)) } - // A slice stored in the loop and read ahead of the index (a[i+1]) is read - // before any store of the body: the scalar loop reads the value from before - // the loop, because the iteration that writes it has not run yet, and a - // store earlier in the body would already have changed it in the vector. - stored := map[types.Object]bool{} + // A slice stored in the loop and read ahead of the element it is stored at + // (a[i+1] beside a[i]) is read before any store of the body: the scalar loop + // reads the value from before the loop, because the iteration that writes it + // has not run yet, and a store earlier in the body would already have changed + // it in the vector. + stored := map[types.Object]string{} for _, stmt := range e.plan.Stmts { if stmt.Dst != nil { - stored[stmt.Dst.Obj] = true + stored[stmt.Dst.Obj] = stmt.Off } } for _, stmt := range e.plan.Stmts { for _, leaf := range loopir.Leaves(stmt.Root) { - if load, ok := leaf.(*loopir.Load); ok && stored[load.Ref.Obj] && load.Off != "" { - emitLoad(load) + if load, ok := leaf.(*loopir.Load); ok { + if off, isStored := stored[load.Ref.Obj]; isStored && load.Off != off { + emitLoad(load) + } } } } @@ -356,16 +371,16 @@ func (e *emitter) statements(b *strings.Builder, full bool, copy, copies int) { acc := accName(accs, copy, copies) fmt.Fprintf(b, "\t%s = %s\n", acc, e.fold(stmt, acc)) case full: - fmt.Fprintf(b, "\t%s.Store(%s)\n", e.expr(stmt.Root), e.operand(stmt.Dst, offset, "")) - delete(loaded, key{stmt.Dst.Obj, ""}) + fmt.Fprintf(b, "\t%s.Store(%s)\n", e.expr(stmt.Root), e.destination(stmt.Dst, offset, stmt.Off)) + delete(loaded, key{stmt.Dst.Obj, stmt.Off}) default: - text := fmt.Sprintf("%s.StorePart(%s)", e.expr(stmt.Root), e.operand(stmt.Dst, offset, "")) + text := fmt.Sprintf("%s.StorePart(%s)", e.expr(stmt.Root), e.destination(stmt.Dst, offset, stmt.Off)) if !haveLanes { // Nothing was loaded, so the store reports how many lanes it wrote. text, haveLanes = "_n := "+text, true } fmt.Fprintf(b, "\t%s\n", text) - delete(loaded, key{stmt.Dst.Obj, ""}) + delete(loaded, key{stmt.Dst.Obj, stmt.Off}) } } } diff --git a/testdata/json_reasons.txt b/testdata/json_reasons.txt index a778ca2..2c800a1 100644 --- a/testdata/json_reasons.txt +++ b/testdata/json_reasons.txt @@ -63,9 +63,9 @@ func DownBoundCall(dst, a []float32) { } } -func DestinationOffset(dst, a []float32) { - for i := range dst { - dst[i+1] = a[i] + 1 +func DestinationStride(dst, a []float32) { + for i := range a { + dst[2*i] = a[i] + 1 } } @@ -132,7 +132,7 @@ func Vectorized(dst, a []float32) { {"file":"$WORK/ops.go","line":37,"func":"BoundDivideByVariable","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} {"file":"$WORK/ops.go","line":43,"func":"DownStartNotLast","vectorized":false,"reason":"loop start is not a loop-invariant integer expression","stage":"lower"} {"file":"$WORK/ops.go","line":49,"func":"DownBoundCall","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} -{"file":"$WORK/ops.go","line":55,"func":"DestinationOffset","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} +{"file":"$WORK/ops.go","line":55,"func":"DestinationStride","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} {"file":"$WORK/ops.go","line":61,"func":"OperandCall","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":69,"func":"BodyIfGuardedStore","vectorized":false,"reason":"a store under a condition may write elements the scalar loop leaves alone","stage":"lower"} {"file":"$WORK/ops.go","line":78,"func":"BodyTwoStatements","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} diff --git a/testdata/offset_negative_constant.txt b/testdata/offset_negative_constant.txt index 64904d9..74c4c02 100644 --- a/testdata/offset_negative_constant.txt +++ b/testdata/offset_negative_constant.txt @@ -1,7 +1,7 @@ -# A read at a negative constant offset is rewritten while every index it reads can -# be zero or more: from a constant start at least as large as the offset, a named +# A read or store at a negative constant offset is rewritten while every index it +# touches can be zero or more: from a constant start at least as large as the offset, a named # one included, which the emitted code keeps by name, and up to a constant limit -# that leaves the highest index read at zero or more. Loops whose index is a +# that leaves the highest index touched at zero or more. Loops whose index is a # negative constant are left alone (see offset_negative_constant_no_rewrite.txt). exec go run ./testmain @@ -54,6 +54,18 @@ func LastIndexZero(dst, a []int32, k int) { dst[i] = a[i-15] + 1 } } + +func StoreFirstIndexZero(dst, a []int32) { + for i := 15; i < 31; i++ { + dst[i-15] = a[i] * 3 + } +} + +func StoreNamedStart(dst, a []int32, n int) { + for i := s1; i < n; i++ { + dst[i-1] = a[i] * 3 + } +} -- ops_simd.go.want -- //go:build goexperiment.simd @@ -238,6 +250,84 @@ func LastIndexZero(dst, a []int32, k int) { } } +func StoreFirstIndexZero(dst, a []int32) { + _vcInt32s5 := simd.BroadcastInt32s(3) + if _loopvecOverlap(dst, a, 0, 31, 15, 31) { + for i := 15; i < 31; i++ { + dst[i-15] = a[i] * 3 + } + } else { + _ = dst[31-16] + _ = a[31-1] + { + _w1 := dst[0:][:(31 - 15):(31 - 15)] + _count := len(_w1) + _w2 := a[15:][:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v1.Mul(_vcInt32s5).Store(_c1) + _v2 := simd.LoadInt32s(_c2[_lanes:]) + _v2.Mul(_vcInt32s5).Store(_c1[_lanes:]) + _v3 := simd.LoadInt32s(_c2[2*_lanes:]) + _v3.Mul(_vcInt32s5).Store(_c1[2*_lanes:]) + _v4 := simd.LoadInt32s(_c2[3*_lanes:]) + _v4.Mul(_vcInt32s5).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v5 := simd.LoadInt32s(_w2[_i:]) + _v5.Mul(_vcInt32s5).Store(_w1[_i:]) + } + for _i < _count { + _v6, _n := simd.LoadInt32sPart(_w2[_i:]) + _v6.Mul(_vcInt32s5).StorePart(_w1[_i:]) + _i += _n + } + } + } +} + +func StoreNamedStart(dst, a []int32, n int) { + _vcInt32s6 := simd.BroadcastInt32s(3) + if _loopvecOverlap(dst, a, s1-1, n, s1, n) { + for i := s1; i < n; i++ { + dst[i-1] = a[i] * 3 + } + } else { + if n > s1 { + _ = dst[n-2] + _ = a[n-1] + { + _w1 := dst[s1-1:][:(n - s1):(n - s1)] + _count := len(_w1) + _w2 := a[s1:][:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v1.Mul(_vcInt32s6).Store(_c1) + _v2 := simd.LoadInt32s(_c2[_lanes:]) + _v2.Mul(_vcInt32s6).Store(_c1[_lanes:]) + _v3 := simd.LoadInt32s(_c2[2*_lanes:]) + _v3.Mul(_vcInt32s6).Store(_c1[2*_lanes:]) + _v4 := simd.LoadInt32s(_c2[3*_lanes:]) + _v4.Mul(_vcInt32s6).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v5 := simd.LoadInt32s(_w2[_i:]) + _v5.Mul(_vcInt32s6).Store(_w1[_i:]) + } + for _i < _count { + _v6, _n := simd.LoadInt32sPart(_w2[_i:]) + _v6.Mul(_vcInt32s6).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + func _loopvecOverlap[T any](a, b []T, aLo, aHi, bLo, bHi int) bool { if aLo >= aHi || bLo >= bHi { return false @@ -248,7 +338,7 @@ func _loopvecOverlap[T any](a, b []T, aLo, aHi, bLo, bHi int) bool { return aBase+uintptr(aLo)*size < bBase+uintptr(bHi)*size && bBase+uintptr(bLo)*size < aBase+uintptr(aHi)*size } -- stderr.want -- -loopvec: rewrote 5 loop(s) in $WORK/ops.go +loopvec: rewrote 7 loop(s) in $WORK/ops.go -- testmain/main.go -- package main @@ -288,6 +378,16 @@ func main() { test.LastIndexZero(dst, ramp(1), 15) fmt.Println(dst) fmt.Println(try(func() { test.LastIndexZero(make([]int32, 16), ramp(1), 14) })) + + dst = make([]int32, 16) + test.StoreFirstIndexZero(dst, ramp(31)) + fmt.Println(dst) + fmt.Println(try(func() { test.StoreFirstIndexZero(make([]int32, 15), ramp(31)) })) + + dst = make([]int32, 20) + test.StoreNamedStart(dst, ramp(21), 21) + fmt.Println(dst) + fmt.Println(try(func() { test.StoreNamedStart(make([]int32, 19), ramp(21), 21) })) } -- testmain/output -- [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17] @@ -299,3 +399,7 @@ true true [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2] true +[48 51 54 57 60 63 66 69 72 75 78 81 84 87 90 93] +true +[6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63] +true diff --git a/testdata/offset_negative_constant_no_rewrite.txt b/testdata/offset_negative_constant_no_rewrite.txt index abad41e..0b50c0c 100644 --- a/testdata/offset_negative_constant_no_rewrite.txt +++ b/testdata/offset_negative_constant_no_rewrite.txt @@ -1,11 +1,11 @@ -# A read at a negative constant offset is left alone when an index it reads is a -# negative constant: the one at the lowest index the loop visits, with a constant +# A read or store at a negative constant offset is left alone when an index it +# touches is a negative constant: the one at the lowest index the loop visits, with a constant # start (a[i-1] over 0..n-1, whichever way the loop counts, and over a range of a # slice), or every one, with a constant limit (a[i-16] in a loop that stops before # 16). The scalar loop panics when it reaches that index, and the vector loop would # index at a negative constant, which does not compile. The start and limit may be # literals, named constants or constant expressions (see offset_negative_constant.txt -# for the loops that are rewritten). A read at a positive constant offset that +# for the loops that are rewritten). A read or store at a positive constant offset that # pushes a constant start or limit past the largest int is left alone too, since the # emitted code would add them. @@ -95,6 +95,30 @@ func StartPastLargest(dst, a []int32, n int) { dst[i] = a[i+2] + 1 } } + +func StoreEveryIndexNegative(dst, a []int32) { + for i := 0; i < 16; i++ { + dst[i-16] = a[i] * 3 + } +} + +func StoreFirstIndexNegative(dst, a []int32, n int) { + for i := s0; i < n; i++ { + dst[i-1] = a[i] * 3 + } +} + +func StoreCountingDownToZero(dst, a []int32) { + for i := 15; i >= 0; i-- { + dst[i-1] = a[i] + 1 + } +} + +func StoreLimitPastLargest(dst, a []int32) { + for i := 0; i < math.MaxInt64; i++ { + dst[i+1] = a[i] + 1 + } +} -- stdout.want -- {"file":"$WORK/ops.go","line":6,"func":"EveryIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} {"file":"$WORK/ops.go","line":12,"func":"FarBelow","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} @@ -107,3 +131,7 @@ func StartPastLargest(dst, a []int32, n int) { {"file":"$WORK/ops.go","line":56,"func":"RangeOverSlice","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} {"file":"$WORK/ops.go","line":62,"func":"LimitPastLargest","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} {"file":"$WORK/ops.go","line":68,"func":"StartPastLargest","vectorized":false,"reason":"loop start is not a loop-invariant integer expression","stage":"lower"} +{"file":"$WORK/ops.go","line":74,"func":"StoreEveryIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":80,"func":"StoreFirstIndexNegative","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":86,"func":"StoreCountingDownToZero","vectorized":false,"reason":"a constant offset puts an index below zero","stage":"lower"} +{"file":"$WORK/ops.go","line":92,"func":"StoreLimitPastLargest","vectorized":false,"reason":"loop limit is not a loop-invariant integer expression","stage":"lower"} diff --git a/testdata/offset_no_rewrite.txt b/testdata/offset_no_rewrite.txt index b3dd6bb..2918f61 100644 --- a/testdata/offset_no_rewrite.txt +++ b/testdata/offset_no_rewrite.txt @@ -4,8 +4,8 @@ # carries a value from one iteration to the next, so it is rejected with its own # reason. A distance that is not a constant and a loop that counts down have no # order loopvec can check. -# Offsets that are not the same on every iteration, a stride, and a store at an -# offset are left alone too. +# Offsets that are not the same on every iteration and a stride are left alone +# too. cp ops.go ops.go.orig loopvec -split . @@ -61,12 +61,6 @@ func OffsetAssigned(dst, a []int32, off int) { } } -func StoreAtOffset(dst, a []int32) { - for i := 0; i < len(dst)-1; i++ { - dst[i+1] = a[i] - } -} - func OffsetFromIndex(dst, a []int32) { for i := 0; i < len(dst); i++ { dst[i] = a[i+i] @@ -79,5 +73,4 @@ func OffsetFromIndex(dst, a []int32) { {"file":"$WORK/ops.go","line":22,"func":"Strided","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":28,"func":"OffsetFromSlice","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} {"file":"$WORK/ops.go","line":34,"func":"OffsetAssigned","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} -{"file":"$WORK/ops.go","line":41,"func":"StoreAtOffset","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} -{"file":"$WORK/ops.go","line":47,"func":"OffsetFromIndex","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} +{"file":"$WORK/ops.go","line":41,"func":"OffsetFromIndex","vectorized":false,"reason":"operand is not a same-index slice, range value, or plain scalar","stage":"lower"} diff --git a/testdata/offset_store.txt b/testdata/offset_store.txt new file mode 100644 index 0000000..f5996ef --- /dev/null +++ b/testdata/offset_store.txt @@ -0,0 +1,671 @@ +# A store may be at a loop-invariant offset from the index (b[i*ldb+j], +# line[off+x], dst[i-k]), the way a load may. Every store to one slice is at the +# same offset, so each iteration writes its own element, a load of that element is +# ordered after the stores before it, and a loop that counts down runs forward as +# when it stores at the index. A read of the stored slice at a constant positive +# distance past the stored element is loaded before any store of the chunk, as for +# a store at the index. The store is capped at the limit plus the offset, so the +# vector loop writes no element the scalar loop does not, and the length check +# covers the last element stored. + +# The scalar program is the reference behavior. +exec go run ./testmain +cmp stdout testmain/output + +cp ops.go ops.go.orig +loopvec -split . +! stdout . +cmpenv stderr stderr.want +cmp ops_simd.go ops_simd.go.want +cmp ops.go ops.go.guarded + +# The simd build must behave identically. +env GOEXPERIMENT=simd +exec go run ./testmain +cmp stdout testmain/output + +-- go.mod -- +module example.com/test + +go 1.27.1 +-- ops.go -- +package test + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + for x := range w { + line[off+x] = v + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + for i := 0; i < n; i++ { + a[i+1] = a[i+3] * 2 + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + for i := range n { + if a[off+i] < 0 { + a[off+i] = 0 + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + for i := range a { + s += a[i] + d[off+i] = a[i] + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } +} +-- ops_simd.go.want -- +//go:build goexperiment.simd + +package test + +import ( + "simd" + "unsafe" +) + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + if _loopvecOverlap(b, a, 0+(min(0, i*ldb)), n+(max(0, i*ldb)), 0+(min(0, i*lda)), n+(max(0, i*lda))) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } + } else { + if n > 0 { + _ = b[n+(i*ldb)-1] + _ = a[n+(i*lda)-1] + copy(b[i*ldb:n+(i*ldb)], a[i*lda:]) + } + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + _vcAFloat32s1 := simd.BroadcastFloat32s(alpha) + if _loopvecOverlap(c, x, 0+(min(0, row)), n+(max(0, row)), 0, n) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } + } else { + if n > 0 { + _ = c[n+row-1] + _ = x[n-1] + { + _w1 := c[row:][:n:n] + _count := len(_w1) + _w2 := x[:_count:_count] + _i, _lanes := 0, simd.BroadcastFloat32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadFloat32s(_c2) + _v2 := simd.LoadFloat32s(_c1) + _v1.MulAdd(_vcAFloat32s1, _v2).Store(_c1) + _v3 := simd.LoadFloat32s(_c2[_lanes:]) + _v4 := simd.LoadFloat32s(_c1[_lanes:]) + _v3.MulAdd(_vcAFloat32s1, _v4).Store(_c1[_lanes:]) + _v5 := simd.LoadFloat32s(_c2[2*_lanes:]) + _v6 := simd.LoadFloat32s(_c1[2*_lanes:]) + _v5.MulAdd(_vcAFloat32s1, _v6).Store(_c1[2*_lanes:]) + _v7 := simd.LoadFloat32s(_c2[3*_lanes:]) + _v8 := simd.LoadFloat32s(_c1[3*_lanes:]) + _v7.MulAdd(_vcAFloat32s1, _v8).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v9 := simd.LoadFloat32s(_w2[_i:]) + _v10 := simd.LoadFloat32s(_w1[_i:]) + _v9.MulAdd(_vcAFloat32s1, _v10).Store(_w1[_i:]) + } + for _i < _count { + _v11, _n := simd.LoadFloat32sPart(_w2[_i:]) + _v12, _ := simd.LoadFloat32sPart(_w1[_i:]) + _v11.MulAdd(_vcAFloat32s1, _v12).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + _vcUint16s2 := simd.BroadcastUint16s(v) + if w > 0 { + _ = line[w+off-1] + { + _w1 := line[off:][:w:w] + _count := len(_w1) + _i, _lanes := 0, simd.BroadcastUint16s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1 := _w1[_i:][:4*_lanes] + _vcUint16s2.Store(_c1) + _vcUint16s2.Store(_c1[_lanes:]) + _vcUint16s2.Store(_c1[2*_lanes:]) + _vcUint16s2.Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _vcUint16s2.Store(_w1[_i:]) + } + for _i < _count { + _n := _vcUint16s2.StorePart(_w1[_i:]) + _i += _n + } + } + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + _vcInt32s3 := simd.BroadcastInt32s(2) + if _loopvecOverlap(dst, a, k+(min(0, -(k))), len(a)+(max(0, -(k))), k, len(a)) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } + } else { + if len(a) > k { + _ = dst[len(a)-(k)-1] + _ = a[len(a)-1] + { + _w1 := dst[k-(k):][:(len(a) - k):(len(a) - k)] + _count := len(_w1) + _w2 := a[k:][:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v1.Mul(_vcInt32s3).Store(_c1) + _v2 := simd.LoadInt32s(_c2[_lanes:]) + _v2.Mul(_vcInt32s3).Store(_c1[_lanes:]) + _v3 := simd.LoadInt32s(_c2[2*_lanes:]) + _v3.Mul(_vcInt32s3).Store(_c1[2*_lanes:]) + _v4 := simd.LoadInt32s(_c2[3*_lanes:]) + _v4.Mul(_vcInt32s3).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v5 := simd.LoadInt32s(_w2[_i:]) + _v5.Mul(_vcInt32s3).Store(_w1[_i:]) + } + for _i < _count { + _v6, _n := simd.LoadInt32sPart(_w2[_i:]) + _v6.Mul(_vcInt32s3).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + _vcInt32s4 := simd.BroadcastInt32s(2) + if n > 0 { + _ = a[n] + _ = a[n+2] + { + _w1 := a[1:][:n:n] + _count := len(_w1) + _w2 := a[3:][:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v1.Mul(_vcInt32s4).Store(_c1) + _v2 := simd.LoadInt32s(_c2[_lanes:]) + _v2.Mul(_vcInt32s4).Store(_c1[_lanes:]) + _v3 := simd.LoadInt32s(_c2[2*_lanes:]) + _v3.Mul(_vcInt32s4).Store(_c1[2*_lanes:]) + _v4 := simd.LoadInt32s(_c2[3*_lanes:]) + _v4.Mul(_vcInt32s4).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v5 := simd.LoadInt32s(_w2[_i:]) + _v5.Mul(_vcInt32s4).Store(_w1[_i:]) + } + for _i < _count { + _v6, _n := simd.LoadInt32sPart(_w2[_i:]) + _v6.Mul(_vcInt32s4).StorePart(_w1[_i:]) + _i += _n + } + } + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + _vcAInt16s5 := simd.BroadcastInt16s(0) + if n > 0 { + _ = a[n+off-1] + { + _w1 := a[off:][:n:n] + _count := len(_w1) + _i, _lanes := 0, simd.BroadcastInt16s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1 := _w1[_i:][:4*_lanes] + _v1 := simd.LoadInt16s(_c1) + _vcAInt16s5.IfElse(_v1.Less(_vcAInt16s5), _v1).Store(_c1) + _v2 := simd.LoadInt16s(_c1[_lanes:]) + _vcAInt16s5.IfElse(_v2.Less(_vcAInt16s5), _v2).Store(_c1[_lanes:]) + _v3 := simd.LoadInt16s(_c1[2*_lanes:]) + _vcAInt16s5.IfElse(_v3.Less(_vcAInt16s5), _v3).Store(_c1[2*_lanes:]) + _v4 := simd.LoadInt16s(_c1[3*_lanes:]) + _vcAInt16s5.IfElse(_v4.Less(_vcAInt16s5), _v4).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v5 := simd.LoadInt16s(_w1[_i:]) + _vcAInt16s5.IfElse(_v5.Less(_vcAInt16s5), _v5).Store(_w1[_i:]) + } + for _i < _count { + _v6, _n := simd.LoadInt16sPart(_w1[_i:]) + _vcAInt16s5.IfElse(_v6.Less(_vcAInt16s5), _v6).StorePart(_w1[_i:]) + _i += _n + } + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + if _loopvecOverlap(a, d, 0, len(a), 0+(min(0, off)), len(a)+(max(0, off))) { + for i := range a { + s += a[i] + d[off+i] = a[i] + } + } else { + if len(a) > 0 { + _ = a[len(a)-1] + _ = d[len(a)+off-1] + { + _i := 0 + if len(a) >= 64 { + _a1_0 := simd.BroadcastInt32s(0) + _a1_1 := simd.BroadcastInt32s(0) + _a1_2 := simd.BroadcastInt32s(0) + _a1_3 := simd.BroadcastInt32s(0) + _lanes := _a1_0.Len() + for ; _i <= len(a)-4*_lanes; _i += 4 * _lanes { + _v1 := simd.LoadInt32s(a[_i:len(a)]) + _a1_0 = _a1_0.Add(_v1) + _v1.Store(d[_i+off : len(a)+off]) + _v2 := simd.LoadInt32s(a[_i+_lanes : len(a)]) + _a1_1 = _a1_1.Add(_v2) + _v2.Store(d[_i+_lanes+off : len(a)+off]) + _v3 := simd.LoadInt32s(a[_i+2*_lanes : len(a)]) + _a1_2 = _a1_2.Add(_v3) + _v3.Store(d[_i+2*_lanes+off : len(a)+off]) + _v4 := simd.LoadInt32s(a[_i+3*_lanes : len(a)]) + _a1_3 = _a1_3.Add(_v4) + _v4.Store(d[_i+3*_lanes+off : len(a)+off]) + } + for ; _i <= len(a)-_lanes; _i += _lanes { + _v5 := simd.LoadInt32s(a[_i:len(a)]) + _a1_0 = _a1_0.Add(_v5) + _v5.Store(d[_i+off : len(a)+off]) + } + _a1_0 = _a1_0.Add(_a1_1) + _a1_0 = _a1_0.Add(_a1_2) + _a1_0 = _a1_0.Add(_a1_3) + var _buf1 [16]int32 + _a1_0.Store(_buf1[:]) + for _, _x := range _buf1[:_lanes] { + s = s + _x + } + } + for i := _i; i < len(a); i++ { + s += a[i] + d[off+i] = a[i] + } + } + } + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + _vcInt32s7_1 := simd.BroadcastInt32s(1) + if _loopvecOverlap(a, b, 0+(min(0, off)), len(b)+(max(0, off)), 0, len(b)) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } + } else { + if len(b) > 0 { + _ = a[len(b)+off-1] + _ = b[len(b)-1] + { + _w1 := a[off:][:len(b):len(b)] + _count := len(_w1) + _w2 := b[:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v1.Add(_vcInt32s7_1).Store(_c1) + _v2 := simd.LoadInt32s(_c1) + _v2.Mul(_v1).Store(_c1) + _v3 := simd.LoadInt32s(_c2[_lanes:]) + _v3.Add(_vcInt32s7_1).Store(_c1[_lanes:]) + _v4 := simd.LoadInt32s(_c1[_lanes:]) + _v4.Mul(_v3).Store(_c1[_lanes:]) + _v5 := simd.LoadInt32s(_c2[2*_lanes:]) + _v5.Add(_vcInt32s7_1).Store(_c1[2*_lanes:]) + _v6 := simd.LoadInt32s(_c1[2*_lanes:]) + _v6.Mul(_v5).Store(_c1[2*_lanes:]) + _v7 := simd.LoadInt32s(_c2[3*_lanes:]) + _v7.Add(_vcInt32s7_1).Store(_c1[3*_lanes:]) + _v8 := simd.LoadInt32s(_c1[3*_lanes:]) + _v8.Mul(_v7).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v9 := simd.LoadInt32s(_w2[_i:]) + _v9.Add(_vcInt32s7_1).Store(_w1[_i:]) + _v10 := simd.LoadInt32s(_w1[_i:]) + _v10.Mul(_v9).Store(_w1[_i:]) + } + for _i < _count { + _v11, _n := simd.LoadInt32sPart(_w2[_i:]) + _v11.Add(_vcInt32s7_1).StorePart(_w1[_i:]) + _v12, _ := simd.LoadInt32sPart(_w1[_i:]) + _v12.Mul(_v11).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + if _loopvecOverlap(dst, a, 0+(min(0, off)), len(a)+(max(0, off)), 0, len(a)) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } + } else { + if len(a) > 0 { + _ = dst[len(a)+off-1] + _ = a[len(a)-1] + { + _w1 := dst[off:][:len(a):len(a)] + _count := len(_w1) + _w2 := a[:_count:_count] + _i, _lanes := 0, simd.BroadcastInt32s(0).Len() + for ; _i <= _count-4*_lanes; _i += 4 * _lanes { + _c1, _c2 := _w1[_i:][:4*_lanes], _w2[_i:][:4*_lanes] + _v1 := simd.LoadInt32s(_c2) + _v2 := simd.LoadInt32s(_c1) + _v1.Add(_v2).Store(_c1) + _v3 := simd.LoadInt32s(_c2[_lanes:]) + _v4 := simd.LoadInt32s(_c1[_lanes:]) + _v3.Add(_v4).Store(_c1[_lanes:]) + _v5 := simd.LoadInt32s(_c2[2*_lanes:]) + _v6 := simd.LoadInt32s(_c1[2*_lanes:]) + _v5.Add(_v6).Store(_c1[2*_lanes:]) + _v7 := simd.LoadInt32s(_c2[3*_lanes:]) + _v8 := simd.LoadInt32s(_c1[3*_lanes:]) + _v7.Add(_v8).Store(_c1[3*_lanes:]) + } + for ; _i <= _count-_lanes; _i += _lanes { + _v9 := simd.LoadInt32s(_w2[_i:]) + _v10 := simd.LoadInt32s(_w1[_i:]) + _v9.Add(_v10).Store(_w1[_i:]) + } + for _i < _count { + _v11, _n := simd.LoadInt32sPart(_w2[_i:]) + _v12, _ := simd.LoadInt32sPart(_w1[_i:]) + _v11.Add(_v12).StorePart(_w1[_i:]) + _i += _n + } + } + } + } +} + +func _loopvecOverlap[T any](a, b []T, aLo, aHi, bLo, bHi int) bool { + if aLo >= aHi || bLo >= bHi { + return false + } + size := unsafe.Sizeof(a[0]) + aBase := uintptr(unsafe.Pointer(unsafe.SliceData(a))) + bBase := uintptr(unsafe.Pointer(unsafe.SliceData(b))) + return aBase+uintptr(aLo)*size < bBase+uintptr(bHi)*size && bBase+uintptr(bLo)*size < aBase+uintptr(aHi)*size +} +-- ops.go.guarded -- +//go:build !goexperiment.simd + +package test + +// Copies row i of an lda-strided matrix into row i of an ldb-strided one: a copy. +func CopyRow(b, a []float64, i, n, lda, ldb int) { + for j := 0; j < n; j++ { + b[i*ldb+j] = a[i*lda+j] + } +} + +// Adds alpha*x into a row of c. +func AxpyRow(c, x []float32, alpha float32, row, n int) { + for j := 0; j < n; j++ { + c[row+j] += alpha * x[j] + } +} + +// Fills w elements starting at off. +func FillAt(line []uint16, off, w int, v uint16) { + for x := range w { + line[off+x] = v + } +} + +// Stores behind the index. +func Behind(dst, a []int32, k int) { + for i := k; i < len(a); i++ { + dst[i-k] = a[i] * 2 + } +} + +// Reads two elements ahead of the one it stores, in the slice it stores. +func Ahead(a []int32, n int) { + for i := 0; i < n; i++ { + a[i+1] = a[i+3] * 2 + } +} + +// Clamps at an offset: the condition reads the element it stores. +func ClampAt(a []int16, off, n int) { + for i := range n { + if a[off+i] < 0 { + a[off+i] = 0 + } + } +} + +// Sums a while copying it to an offset. +func SumCopy(d, a []int32, off int) int32 { + var s int32 + for i := range a { + s += a[i] + d[off+i] = a[i] + } + return s +} + +// Stores the same element twice; the second store reads the first. +func Twice(a, b []int32, off int) { + for i := 0; i < len(b); i++ { + a[off+i] = b[i] + 1 + a[off+i] *= b[i] + } +} + +// Counts down, storing at an offset and reading only the element it stores. +func Reverse(dst, a []int32, off int) { + for i := len(a) - 1; i >= 0; i-- { + dst[off+i] = a[i] + dst[off+i] + } +} +-- stderr.want -- +loopvec: rewrote 9 loop(s) in $WORK/ops.go +-- testmain/main.go -- +package main + +import ( + "fmt" + "slices" + + "example.com/test" +) + +var sizes = []int{0, 1, 2, 3, 5, 8, 9, 16, 17, 33, 64, 65, 100, 131} + +func ramp(n int, scale int32) []int32 { + s := make([]int32, n) + for i := range s { + s[i] = int32(i*i+1)*scale - 40 + } + return s +} + +func sum[T int16 | uint16 | int32 | float32 | float64](s []T) (t float64) { + for i, v := range s { + // The conversion rounds the product, so no target fuses it into the sum. + t += float64(float64(v) * float64(i+1)) + } + return t +} + +func try(f func()) (panicked bool) { + defer func() { panicked = recover() != nil }() + f() + return false +} + +func main() { + for _, n := range sizes { + const lda, ldb = 140, 150 + a := make([]float64, 4*lda) + for i := range a { + a[i] = float64(i) / 3 + } + b := make([]float64, 4*ldb) + for i := range 4 { + test.CopyRow(b, a, i, n, lda, ldb) + } + + c := make([]float32, n+7) + x := make([]float32, n) + for i := range x { + x[i] = float32(i) / 4 + } + test.AxpyRow(c, x, 3, 7, n) + test.AxpyRow(c, x, -1, 0, n) + + line := make([]uint16, n+5) + test.FillAt(line, 5, n, 9) + + d := make([]int32, n) + test.Behind(d, ramp(n+3, 1), 3) + + h := ramp(n+3, 2) + test.Ahead(h, n) + + s16 := make([]int16, n+2) + for i := range s16 { + s16[i] = int16(i*5%13) - 6 + } + test.ClampAt(s16, 2, n) + + sd := make([]int32, n+9) + total := test.SumCopy(sd, ramp(n, 3), 9) + + tw := make([]int32, n+11) + test.Twice(tw, ramp(n, 1), 11) + + rv := ramp(n+4, 5) + test.Reverse(rv, ramp(n, 1), 4) + + fmt.Println(n, sum(b), sum(c), sum(line), sum(d), sum(h), sum(s16), total, sum(sd), sum(tw), sum(rv)) + } + + // Slices too short for the loop panic before the loop writes anything wrong; + // each call gets its own copy, so what a panicking loop wrote is not observed. + x := make([]float32, 37) + c := make([]float32, 40) + fmt.Println( + try(func() { test.CopyRow(make([]float64, 20), make([]float64, 30), 1, 11, 10, 10) }), + try(func() { test.AxpyRow(slices.Clone(c), x, 3, 4, 37) }), + try(func() { test.AxpyRow(c, x, 1, 0, -2) }), + try(func() { test.FillAt(make([]uint16, 70), 69, 2, 7) }), + try(func() { test.Behind(make([]int32, 40), ramp(43, 1), 2) }), + try(func() { test.SumCopy(make([]int32, 10), ramp(9, 1), 2) }), + ) + + // Overlapping views take the original loop, so the result is the scalar one. + p := make([]float32, 40) + for i := range p { + p[i] = float32(i%7) - 2 + } + test.AxpyRow(p[1:], p, 1, 0, 30) + q := ramp(40, 1) + fmt.Println(sum(p), test.SumCopy(q, q[:39], 1), sum(q)) +} +-- testmain/output -- +0 0 0 0 0 -200 -8 0 0 0 -100 +1 98280 0 54 -60 -288 4 -37 -370 17784 -70 +2 197142.6666666667 6.25 117 -152 -264 4 -71 -744 36062 242 +3 296590.6666666667 19.75 189 -236 -16 9 -96 -1044 52722 1012 +5 497253.3333333334 72.5 360 -160 1720 45 -95 -1020 73868 4810 +8 802704 231 684 1704 10184 69 124 2556 79160 20412 +9 905720 309 810 3180 15472 124 279 5346 92160 29642 +16 1.6438933333333335e+06 1310 1944 39440 118232 232 3128 70040 2.646768e+06 198300 +17 1.7518160000000002e+06 1530 2142 50388 146816 346 3859 89046 3.971336e+06 244122 +33 3.5654373333333326e+06 8756 6534 690404 1.649184e+06 1022 33099 1.147806e+06 2.524566e+08 2.583162e+06 +64 7.575680000000004e+06 54264 21600 9.22272e+06 2.0135352e+07 3512 253664 1.4654816e+07 1.3013051328e+10 3.0776732e+07 +65 7.716626666666672e+06 56680 22230 9.80122e+06 2.136816e+07 3512 265915 1.556139e+07 1.4264262584e+10 3.264953e+07 +100 1.3143399999999993e+07 192637.5 49950 5.33583e+07 1.127652e+08 8692 981350 8.313785e+07 1.8293135886e+11 1.710221e+08 +131 1.881247333333331e+07 419363.75 83709 1.5496514e+08 3.23149552e+08 14389 2217568 2.393763e+08 9.07732130162e+11 4.8864562e+08 +true true false true true true +8775 -1521 -31980 diff --git a/testdata/offset_store_no_rewrite.txt b/testdata/offset_store_no_rewrite.txt new file mode 100644 index 0000000..f5def61 --- /dev/null +++ b/testdata/offset_store_no_rewrite.txt @@ -0,0 +1,75 @@ +# A store at an offset is left alone when two iterations could touch the same +# element in an order the vector loop does not keep: a slice stored at two +# offsets, a read of the stored slice behind the stored element (a recurrence), a +# read at a distance that is not a constant, a read at another offset in a loop +# that counts down, and an offset the body changes. A stride and a nested index +# are not an offset. + +cp ops.go ops.go.orig +loopvec -split . +! stdout . +! stderr . +! exists ops_simd.go +cmp ops.go ops.go.orig + +loopvec -json . +cmpenv stdout stdout.want + +-- go.mod -- +module example.com/test + +go 1.27.1 +-- ops.go -- +package test + +func TwoOffsets(a, b, c []int32) { + for i := 0; i < len(b); i++ { + a[i] = b[i] + a[i+1] = c[i] + } +} + +func Recurrence(a, b []int32) { + for i := 0; i < len(b); i++ { + a[i+1] = a[i] + b[i] + } +} + +func DistanceUnknown(a []int32, k, n int) { + for i := 0; i < n; i++ { + a[i+k] = a[i] * 2 + } +} + +func CountingDownAhead(a []int32) { + for i := len(a) - 4; i >= 0; i-- { + a[i+1] = a[i+3] * 2 + } +} + +func OffsetAssigned(dst, a []int32, off int) { + for i := range a { + dst[off+i] = a[i] + off++ + } +} + +func Stride(dst, a []int32) { + for i := range a { + dst[2*i] = a[i] + } +} + +func Nested(dst [][]int32, a []int32) { + for i := range a { + dst[0][i] = a[i] + } +} +-- stdout.want -- +{"file":"$WORK/ops.go","line":4,"func":"TwoOffsets","vectorized":false,"reason":"a slice is stored at more than one offset","stage":"lower"} +{"file":"$WORK/ops.go","line":11,"func":"Recurrence","vectorized":false,"reason":"an iteration reads an element an earlier iteration wrote","stage":"lower"} +{"file":"$WORK/ops.go","line":17,"func":"DistanceUnknown","vectorized":false,"reason":"a slice is read at an offset and written in the same loop","stage":"lower"} +{"file":"$WORK/ops.go","line":23,"func":"CountingDownAhead","vectorized":false,"reason":"a slice is read at an offset and written in the same loop","stage":"lower"} +{"file":"$WORK/ops.go","line":29,"func":"OffsetAssigned","vectorized":false,"reason":"loop body is not a sequence of indexed assignments","stage":"lower"} +{"file":"$WORK/ops.go","line":36,"func":"Stride","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} +{"file":"$WORK/ops.go","line":42,"func":"Nested","vectorized":false,"reason":"destination is not a simple, same-index slice element","stage":"lower"} diff --git a/tsvc/README.md b/tsvc/README.md index af5d5f0..3bf05fc 100644 --- a/tsvc/README.md +++ b/tsvc/README.md @@ -107,8 +107,8 @@ by `n*2^-24`, for the kernels that vectorize only with `-fp-reassoc`). | s421 | storage classes and equivalencing | equivalence- no overlap | bits | vectorize | yes | | s1421 | storage classes and equivalencing | equivalence- no overlap | bits | vectorize | yes | | s422 | storage classes and equivalencing | common and equivalence statement; anti-dependence, threshold of 4 | bits | vectorize | yes | -| s423 | storage classes and equivalencing | common and equivalenced variables - with anti-dependence; do this again here | bits | vectorize | no | -| s424 | storage classes and equivalencing | common and equivalenced variables - overlap; vectorizeable in strips of 64 or less; do this again here | bits | vectorize | no | +| s423 | storage classes and equivalencing | common and equivalenced variables - with anti-dependence; do this again here | bits | vectorize | yes | +| s424 | storage classes and equivalencing | common and equivalenced variables - overlap; vectorizeable in strips of 64 or less; do this again here | bits | vectorize | yes | | s431 | parameters | parameter statement | bits | vectorize | no | | s441 | non-logical if's | arithmetic if | fused | vectorize | no | | s443 | non-logical if's | arithmetic if | fused | vectorize | no | @@ -160,7 +160,7 @@ loop with a real dependence between iterations is correct to leave alone. the expectation copied from the kernel's directive so the file reads on its own: ``` -# loopvec TSVC coverage: vectorized 22, gaps 70, declined 1, skipped 0 (of 93) +# loopvec TSVC coverage: vectorized 24, gaps 68, declined 1, skipped 0 (of 93) s000 vectorize yes s111 vectorize no loop start is not 0 (or len(s)-1 when counting down) ... diff --git a/tsvc/testdata/coverage.txt b/tsvc/testdata/coverage.txt index 420b54c..4f133f4 100644 --- a/tsvc/testdata/coverage.txt +++ b/tsvc/testdata/coverage.txt @@ -1,8 +1,8 @@ -# loopvec TSVC coverage: vectorized 22, gaps 70, declined 1, skipped 0 (of 93) +# loopvec TSVC coverage: vectorized 24, gaps 68, declined 1, skipped 0 (of 93) s000 vectorize yes s111 vectorize no loop step is not i++ (or i-- when counting down) s1111 vectorize no destination is not a simple, same-index slice element -s112 vectorize no destination is not a simple, same-index slice element +s112 vectorize no a slice is read at an offset and written in the same loop s1112 vectorize yes s113 vectorize no operand is not a same-index slice, range value, or plain scalar s1113 vectorize no operand is not a same-index slice, range value, or plain scalar @@ -24,8 +24,8 @@ s141 vectorize no destination is not a simple, same-index slice element s162 vectorize no a slice is read at an offset and written in the same loop s171 vectorize no destination is not a simple, same-index slice element s172 vectorize no loop step is not i++ (or i-- when counting down) -s173 vectorize no destination is not a simple, same-index slice element -s174 vectorize no destination is not a simple, same-index slice element +s173 vectorize no a slice is read at an offset and written in the same loop +s174 vectorize no a slice is read at an offset and written in the same loop s175 vectorize no loop step is not i++ (or i-- when counting down) s176 vectorize no operand is not a same-index slice, range value, or plain scalar s211 vectorize no an iteration reads an element an earlier iteration wrote @@ -36,10 +36,10 @@ s2233 vectorize no destination is not a simple, same-index slice element s235 vectorize no destination is not a simple, same-index slice element s241 vectorize yes s243 vectorize yes -s2244 vectorize no destination is not a simple, same-index slice element +s2244 vectorize no a slice is stored at more than one offset s251 vectorize yes s1251 vectorize yes -s3251 vectorize no destination is not a simple, same-index slice element +s3251 vectorize no an iteration reads an element an earlier iteration wrote s252 vectorize no operand is not a same-index slice, range value, or plain scalar s254 vectorize no operand is not a same-index slice, range value, or plain scalar s255 vectorize no operand is not a same-index slice, range value, or plain scalar @@ -73,8 +73,8 @@ s352 vectorize no loop step is not i++ (or i-- when counting down) s421 vectorize yes s1421 vectorize yes s422 vectorize yes -s423 vectorize no destination is not a simple, same-index slice element -s424 vectorize no destination is not a simple, same-index slice element +s423 vectorize yes +s424 vectorize yes s431 vectorize no a slice is read at an offset and written in the same loop s441 vectorize no a slice is read only under a condition s443 vectorize no a slice is read only under a condition